[Experimental] Add a new SfM pipeline based on stellar reconstruction

[pmoulon]

History and background:

• Incremental SfM v2 #1241 introduced a method to recover global motions of a Stellar / n-uplets pod (one central image + n satellites) as a convenient way to bootstrap an Incremental SfM pipeline from N images instead of 2, we are introducing here a SfM pipeline using n-uplets as input of a Global SfM pipeline.

What is this Stellar SfM pipeline?

• We are now introducing a SfM pipeline using n-uplets over the view-graph to a GlobalSfM pipeline.

We see this pipeline as a natural evolution or OpenMVG research progress for SfM due to the following fact:

• We realized that using triplets for GlobalSfM was helping support forward motions and finding much more relative translations to feed for translation averaging.
• We now think that using n-uplet is a natural extension to enforce more consistency and thigh stable results to compare relative motions in epipolar view graph.

Pros:

• Using n-uplets or stellar pods allows us to refine local motions and make relative motions consistency tighter; resulting in improved robust behavior on Rotation and Translation averaging results.
• Adding view graph simplification to this pipeline is extremely easy and allows to leverage often only 50% of the edges to perform very nice reconstructions. We have used here some similar concept as skeletal graph but adapted to n-uplets / stellar configuration by selection X Maximum Spanning tree (forcing local stars configurations).

Cons:

• In an ideal world we would have accurate 2-view rotation and translation at the first stage... but as we know since these steps rely on approximate intrinsics parameters (approximate focal and unknown distortion factors), those relative motions are not perfect and often contaminated by invalid relative motion (due to repetitive patterns or local structure match).

You can find below a short summary comparison of the various reconstruction algorithms:

How to test it?

$ openMVG_main_SfM -i <sfm_data.json> -m <match_path> -o <output_reconstruction_folder> --sfm_engine STELLAR

How can you provide us with feedback?

• Please report on this PR your reconstruction and your observations (timing, quality of the reconstruction) ...
• This is an experimental pipeline, so don't expect it to be more accurate, and faster on all your datasets, but with your feedback, and contribution we are looking forward make it really robust/solid!

Thank you for @rjanvier for his contribution to this development and for helping stabilize the stellar reconstruction algorithms.

[csparker247]

Alright, I've had a chance to run comparison recons on the same dataset. As backstory, my scanning setup is essentially the one I described way back in 2020: five cameras which are fixed relative to each other, but which are traveling a grid path on the XY axis above a mostly planar object. The dataset is 780 images (5MP each). My motion is motor controlled, so I have estimates for the extrinsics for each image which haven't ever been good enough to triangulate on their own. Instead, I initialize sfm_data.json with those extrinsics as a "hint" and run the standard SfM pipeline. One misconfiguration on my part for this dataset is that the imported extrinsic has an identity rotation and the center is in the correct region of the world coordinate frame, but not where the camera actually was. Historically, global reconstruction has produced better results than either incremental version, so that's what I use exclusively. I have mostly been using OpenMVG v1.6 and just recently updated to v2.0, so I compare global sfm in each of those versions against stellar.

Reconstruction commands

• Global (v1.6)

  openMVG_main_GlobalSfM -i sfm_data.json -m matches_dir -o recon_dir -M matches_dir/matches.f.bin

• Global+Robust (v1.6):
  • In v1.6, I always gotten better results if I followed SfM with a robust triangulation using ComputeStructureFromKnownPoses. I'm still testing whether this is helpful or just a waste of time in v2.0.

  openMVG_main_ComputeStructureFromKnownPoses -i sfm_data.bin -m matches_dir -o sfm_data_structured.bin -f matches_dir/matches.f.bin

• Global (v2.0)

  openMVG_main_SfM -i sfm_data.json -s GLOBAL -m matches_dir -o recon_dir -M matches_filtered.bin

• Stellar

  openMVG_main_SfM -i sfm_data.json -s STELLAR -m matches_dir -o recon_dir -M matches_filtered.bin

Subjective Results

Visually, the reconstructed camera positions are fairly similar. Since I don't have ground truth for this dataset, it's hard to know how close they are to expected, but they seem reasonable relative to the detected scene. A few notes:

• Stellar shows pretty significant "bowing" in the camera positions and scene features.
  
  My gut instinct tells me that this might be caused by differing intrinsic estimations. I have two intrinsics, and the estimates between each method do calculate differing focal lengths, but I'm not experienced to know if they're significantly different in this context. Since Global v2.0 seems visually correct, here are the values from that and Stellar:

	Intrinsic 1 (Global v2.0)	Intrinsic 2 (Global v2.0)	Intrinsic 1 (Stellar)	Intrinsic 2 (Stellar)
Focal Length	4613.07	3717.48	4312.17	3440.81
Principal Pt.	[1235.98, 1001.79]	[1221.34, 1006.40]	[1231.27, 1277.60]	[1233.89, 1006.36]
K3	[-0.23101, 0.24692, -0.38516]	[-0.08339, 0.12214, -0.45791]	[-0.19790, 0.14862, -0.22706]	[-0.08891, 0.10494, -0.29192]

• Global+Robust has more outlier features than just Global. Stellar has many, many more feature outliers than any other method. Additionally, there are features floating way out in space, behind the cameras? Here's the stellar bounding box if I zoom out:

Empirical Results

To try to put some numbers to this, I've aggregated the results from SfMReconstruction_Report.html (or the equivalent) for each reconstruction. Time (s) is how long the SfM step took. I don't have this for the v1.6 recons. Recon scale is the scale of the reconstruction calculated from known fiducials in the scene after running things through OpenMVS. Residuals [min|median|max] are the given operation performed over the corresponding column in the report. For example, Residuals min provides the min of all views in the report's Residuals min column, and Residuals median reports the median of all views in the report's Residuals median column. [Mean|Median] Residuals * is similar, except that the mean or median is taken over the given column. Confusing, no?

	Time (s)	Recon Scale	Residuals min	Mean residual min	Median residual min	Residuals median	Mean residual median	Median residual median	Mean residuals mean	Residuals max	Mean residuals max	Median residuals max
Global (v1.6)	-	-	1.76E-07	1.18E-04	6.45E-05	0.2482700	0.2493156	0.2482700	0.3648423	6.8284500	3.8677809	3.7942550
Global+Robust (v1.6)	-	0.0523380	3.16E-08	2.36E-05	1.19E-05	0.2961755	0.2934834	0.2961755	0.4378704	3.9969700	3.7490539	3.7822950
Global (v2.0)	738	0.0734842	4.65E-08	1.25E-04	6.68E-05	0.2500920	0.2503197	0.2500920	0.3666414	5.9926200	3.7729333	3.7325250
Stellar	1050	1.1395956	1.54E-08	6.15E-05	2.60E-05	0.2172065	0.2194761	0.2172065	0.3155448	153.4100000	6.2293916	4.3827700

I think the takeaways from this table are:

• Stellar took about 40% longer to compute in this test, but the computation times are not unreasonable for such a large scene.
• Broadly, stellar's residual error seems to be better than or on par with the other methods. One exception is with the maximum residual error. There are some pretty significant feature outliers as shown in the subjective results.
• Stellar produces a recon that's closer to the expected scale than the other methods. I assume this is only because I'm importing some estimate for the camera positions. I vaguely remember seeing a similar thing with IncrementalV2.

[pmoulon]

Thank you @csparker247 for this details test and report. Appreciated the level of details!

Just a quick question:

• You mention you have a camera system based on 5 cameras, but you are then listing 2 intrinsics.
  A. Are you configuring the same intrinsics for different cameras (based on the notion that they are sharing the same sensor and lens)?
  B. Or are you using 2 intrinsics, because you change the focal length between two captures?

Nit: I would just set 1 intrinsic per camera (since each camera can be a little different)

Bowing:

• Using fixed intrinsic calibrations could help solve the problem of bowing (here we see intrinsics and extrinsics compensating for each other) => So you would estimate only the extrinsics, or provide a better guess for the minimization.
  • You could also use the Camera Model -c 1 to use a lighter distortion model and see if it still fit nicely your camera model
  • This is also why you see some improvement using global + robust, since robust is using the refined intrinsics to recover better 2-view matches

Floating points:

• Maybe I forgot to add some point filtering methods I'm using in the other pipelines (as showing in the max error)

Scale:

• Stellar SfM does not assume to use your existing poses, so being closer to the real scale is just so luck

I'm glad that you were able to test flawlessly the new pipeline and to notice that the scene seems to have more 3D points.

• Regarding speed (Global vs Stellar) - depending of some graph, stellar or global could be faster, but Stellar will be in general much more robust to outlier data and so fail much less often than the Global pipeline.

[csparker247]

Thanks @pmoulon!

You mention you have a camera system based on 5 cameras, but you are then listing 2 intrinsics.
A. Are you configuring the same intrinsics for different cameras (based on the notion that they are sharing the same sensor and lens)?
B. Or are you using 2 intrinsics, because you change the focal length between two captures?

Nit: I would just set 1 intrinsic per camera (since each camera can be a little different)

All five cameras share the same body and 4 have the same type of fixed focal length lens. My assumption was that the intrinsic variations between lenses of the same make and model would not make a significant difference in practice. I have admittedly been very lazy about precalibrating my cameras recently, though, so I suppose I should measure this and make sure my I'm still initializing sfm_data.json correctly.

Bowing:

Using fixed intrinsic calibrations could help solve the problem of bowing (here we see intrinsics and extrinsics compensating for each other) => So you would estimate only the extrinsics, or provide a better guess for the minimization.

This tradeoff makes sense. So I should stop being lazy and calibrate my lenses ahead of time 😄. I'll try that and re-run stellar. I will say that I ran a stellar + robust job yesterday and it doesn't appear to improve the bowing. However...

Floating points:

Maybe I forgot to add some point filtering methods I'm using in the other pipelines (as showing in the max error)

Adding a robust step did remove the worst of the outlier features, but quite a few still falling behind the imaging plane. Obviously not super important to the discussion here, but thought I should at least mention it:

Regarding speed (Global vs Stellar) - depending of some graph, stellar or global could be faster, but Stellar will be in general much more robust to outlier data and so fail much less often than the Global pipeline.

Good to know! I'll keep testing this features as it develops and let you know how it goes!

[ghost]

Command line:

• Stellar

openMVG_main_SfM -i $matchesDir\sfm_data.json -m $matchesDir -o $reconstructionDir -s STELLAR -g <#>

• Global

openMVG_main_SfM -i $matchesDir\sfm_data.json -m $matchesDir -o $reconstructionDir -s GLOBAL -M $matchesDir\matches.e.bin

• IncrementalV2

openMVG_main_SfM -i $matchesDir\sfm_data.json -m $matchesDir -o $reconstructionDir -s INCREMENTALV2

• Global+IncrementalV2

openMVG_main_SfM -i $matchesDir\sfm_data.json -m $matchesDir -o $reconstructionDir -s GLOBAL -M $matchesDir\matches.e.bin
openMVG_main_SfM -i $reconstructionDir\sfm_data.bin -m $matchesDir -o $reconstructionDir -s INCREMENTALV2 -S EXISTING_POSE

• Stellar+IncrementalV2

openMVG_main_SfM -i $matchesDir\sfm_data.json -m $matchesDir -o $reconstructionDir -s STELLAR -g 2
openMVG_main_SfM -i $reconstructionDir\sfm_data.bin -m $matchesDir -o $reconstructionDir -s INCREMENTALV2 -S EXISTING_POSE

WAH! 🐙	Komainu	Komainu w/ ball	Snails	Ammonite
Stellar -g 2	00h 11m 18s 1037 / 1040	N/A	00h 01m 48s 227 / 227	N/A
Stellar -g 3	00h 11m 12s 1040 / 1040	00h 42m 38s 1154 / 1156	00h 01m 50s 227 / 227	N/A
Stellar -g 5	00h 13m 15s 1040 / 1040	N/A	00h 01m 38s 227 / 227	N/A
Global	00h 20m 20s 1039 / 1040	00h 50m 58s 1153 / 1156	00h 01m 11s 223 / 227	N/A
IncrementalV2	00h 19m 36s 1040 / 1040	03h 58m 28s 1154 / 1156	00h 07m 05s 227 / 227	N/A
Global+IncrementalV2	00h 22m 53s 1040 / 1040	01h 08m 52s 1154 / 1156	00h 02m 25s 227 / 227	N/A
Stellar+IncrementalV2	00h 16m 54s 1040 /1040	N/A	00h 02m 41s 227 / 227	N/A

[pmoulon]

@4CJ7T Thank you for testing and comparing the different pipeline, this is interesting to see that Stellar -g 5 is providing a excellent trade off between completeness and timing

Cannot wait to see if you will test on more scenes ;-)