Issues with Pose Priors

[csparker247]

I've been trying to import pose priors from an automatic acquisition rig in order to make my results more reliable across different objects, but I've been having issues with the reconstruction workflow. The scanner is a custom rig with five cameras mounted to a motorized XYZ stage. The system is intended to capture flat-ish surfaces (like topographical models) so all cameras point down at the ground plane. The cameras are arranged in a cross pattern where the central camera faces straight down, and the wing cameras point down at a 45 deg angle:

Top-Down View
   N
W  C  E
   S 

Side View (X-axis)
 W    C    E  
 \    |    /

----ground----

The system runs step-and-shoot acquisition along a customizable path (generally a grid pattern), producing 5 images at discrete acquisition points along the path. The acquisition points are set so that every point in the interior scan area is seen by all five cameras. The final dataset includes the images for all five cameras, the radial (k3) distortion parameters for the cameras, the stage position at each acquisition point, and the relative offset of the cameras from the stage position. I have custom code which converts this metadata into an sfm_data.json file. The work I did on #1702 was so that I could verify that my camera poses were being converted correctly.

I know there's likely to be some error in my camera poses due to both measurement and mechanical errors in the XYZ stage, so I need to do some sort of automatic adjustment on the scene to minimize reprojection error. Following advice that I've seen in other discussions, my workflow has been as follows:

1. Compute features
2. Use frustum filtering and initial poses to create pair list
3. Compute matches with pair list
4. Reconstruct (pick one):
   • (a) ComputeStructureFromKnownPoses + bundle adjustment
   • (b) IncrementalSfM2 with -f NONE -S EXISTING_POSE -P

When I run 4a, the filtering step prior to BA removes all of my poses and I get an empty scene. If I disable BA, I get a scene where all of the feature points are dispersed incoherently below the cameras. Not surprising since I assume my camera positions are off:

When I run 4b, my cameras are adjusted way out of position and my feature points do not seem any more coherent than they did with 4a:

So now I'm left not really sure where my problem lies. I see a number of questions that are open:

1. Is this the correct workflow for pose priors in OpenMVG?
2. Have I missed something in my assumptions about how this all works?
3. Is the error in my pose priors larger than I expect?
4. What if I could set limits on the bundle adjustment for position to keep cameras from flying off into space?

I've been looking at this long enough that I would appreciate any help or guidance that anyone has to give. I would also like to contribute back to the project in the form of documentation surrounding initialization with pose priors since that seems to be somewhat lacking. What would be the best way to do that?

[pmoulon]

Hello @csparker247
Thank you for the deep description.

Your pose workflow is correct, it is the pipeline we follow for importing external dataset (see here)

Here what I would test:

• using less image (for example only the Center one for the moment)
• checking that you have good feature matches between your images (using the frustum filtering is perhaps useless since all your cones are most likely to intersect). You can visualize the matches once geometric filtering is done by using ExportMatches
• ignore your k3 distortion factor for the moment

In order to check if you pose prior error is larger than what you expect I would do the following:

• run existing SfM pipeline on your image (see if you have the right pattern (cross))
• compute a rigid transform between your camera positions and the one found by sfm and see the STD

Let me know if this guidance is useful.
We can also take a look to your data together more closely if you want

[csparker247]

Thanks very much! This gives me a good number of tests to work on. I did just get more accurate camera mount measurements from the engineer, and those were off by a number of units, so I know that was at least contributing to my poor reconstructions. I'll be back in touch once I've had a chance to test these things and see what I've found.

[csparker247]

I've gotten some of the above suggestions tested and just wanted to follow up with what I've found. First, the improved camera mount measurements did not result in any sort of magical improvement in scene reconstruction. Not surprising, but at least that's factor that can be removed from consideration. I feel as though I can reasonably assume my measurement error in pose priors is within 1 unit now. That still doesn't account for mechanical positioning error, which is unknown but should be reasonably small (it is a CNC machine head after all).

I have run various pipelines using just the center images, comparing the results of the default pipeline with the pose priors version. I did not use my distortion parameters in these tests. With the particular test dataset that I used, I was not able to get a good SfM reconstruction even when ignoring the priors, so I wasn't able to compare the transform between that reconstruction and my expected scene. This test dataset is a subregion of a larger dataset which has worked previously, so I'm currently working on getting a test dataset that does reconstruct.

That said, these tests weren't wholly unenlightening. I can confirm that filtering the frustums (even when setting the near and far distances to reasonable values) are barely helping compared to no filtering. The final geometric matches were identical and filtering the putative matches only removed about 12 pairs. So not much of a gain there.

Next, spot checking the matches showed okay results: generally strong bundles of matches with occasional outliers. I did discover one major problem, though. My test data is of an X-Rite color checker, which has the X-Rite logo printed on it in multiple locations. The multiple instances of this logo got strongly matched despite being in different locations:

I'm guessing that my inability to get a good SfM reconstruction with my subset data is because of these incorrect matches. There's nothing in my subset dataset to indicate that these are actually outliers on the scene. However, when I reconstruct the full scan area, the matches over the rest of the scene show that these are in fact outliers. Is that reasonable?

Finally, I noticed a dramatic difference in scene initialization for SfM2 with pose priors in comparison to without pose priors:

Without priors, I get a semi-coherent scene initialization that seems like a reasonable start. My center camera has strong radial distortion, so the curve in the initialization doesn't surprise me. With priors, however, my initialized scene is not at all coherent. I can imagine that this could be caused by:

1. Initialization with pose priors and an uncalibrated lens puts cameras in the wrong position w.r.t. detected features. Or something related to my radial distortion? I feel like it should still be somewhat coherent though.
2. The previously discovered matching error is enough to throw off scene initialization
3. A combination of 1 & 2

Interestingly, GlobalSfM does even better at scene init than SfM2:

That appears to use all of my cameras at least! In all circumstances (Global, SfM2, SfM2+priors) the final reconstruction is still bad.

So that's an update at least. I'm going to continue with collecting a better test dataset that actually reconstructs without poses and seeing what that shows me. I can share data at any point (with requisite delay for my terrible DSL connection), so if it would be useful for me to make something available, please let me know. Thanks!

[pmoulon]

Thank you for your detailed feedback.
When images have a lot of overlap GlobalSfM is often working better since there is a lot of data to average. The curve effect could be due to the fact that we don't estimate the right distortion. Providing the right distortion would help.

It seems that the scene is not easy to match (a lot of redundant information in the checkboard). Did you try with a texture object (like a picture or a newspaper?)

It seems that we are not very robust to this logo repetition... in the incremental case since it must provide some wrong. camera initialization since the start.

Regarding the prior (it could be nice to share your data and let us take a look)

[csparker247]

Alright, another update. Things are slowly starting to make more sense.

I made a test dataset of the most simple case: the five cameras pointed at the same spot on the ground plane. The spot chosen has a prominent orientable feature and no repetitive background features (like the grid pattern). You can download it from here. This includes the images and two SfM files: one created with openMVG_main_SfMInit_ImageListing and one I generated with my code which includes priors and lens calibrations.

Using this dataset, I've been able to run some tests. All tests were run with SIFT features on HIGH. If I run:

openMVG_main_IncrementalSfM2 -i sfm_data.json -m matches/ -o recon/ -M matches/matches.f.bin -S MAX_PAIR

then the resulting reconstruction looks correct and I get camera poses that are roughly what I expected relative to each other. What is not expected is that my camera orientations are all rotated 180 deg around the optical axis. The top face for all of these frustums should actually be the face directly opposite the one that's colored red:

So that's odd. I've been having trouble getting CloudCompare or Meshlab to do registration on my laptop, but I'll hopefully get that ironed out soon and we can see just how off the SfM poses are from my pose priors.

I also tested running the following commands with a pose-prior sfm:

# SfM has priors, but don't init with them
openMVG_main_IncrementalSfM2 -i sfm_priors.json -m matches/ -o recon/ -M matches/matches.f.bin -S MAX_PAIR

# SfM has priors and do init with them
openMVG_main_IncrementalSfM2 -i sfm_priors.json -m matches/ -o recon/ -M matches/matches.f.bin -P -S EXISTING_POSE

# Direct triangulation
openMVG_main_ComputeStructureFromKnownPoses -i sfm_priors.json -m matches/ -o recon/sfm_data_structured.bin -f matches/matches.f.bin -d

The first produces a decent reconstruction and the cameras are in the correct cross pattern, but the center camera is closer to the ground plane than it should be. The second puts the cameras in incorrect positions and fails to give a decent reconstruction. For the final command, the feature points are more coherent than they were on my previous dataset, but they still are more of a clump than a surface, and any further reconstruction with MVS fails:

As before, adding bundle adjustment to this command results in an empty scene.

So, all of that reported, here's some follow-up things:

It seems that the scene is not easy to match (a lot of redundant information in the checkboard). Did you try with a texture object (like a picture or a newspaper?)

It seems that we are not very robust to this logo repetition... in the incremental case since it must provide some wrong. camera initialization since the start.

Yes, it's become very clear that my repetitive backdrop is causing problems. I'm going to implement some sort of random dot pattern that I can put on my backdrop to try to reduce bad matches. I don't have a good answer for the color checker or scale marker (i.e. ruler). I will likely need to include one or both. I suppose I can always make a "ruler" from two coded dots of known distance.

Regarding the prior (it could be nice to share your data and let us take a look)

There's the small test dataset I mentioned above. Additionally, the full image dataset can be download from here and corresponding sfm files can be downloaded here. There's a lot happening in this dataset, so feel free to ask me questions about any of it. The most important thing to note is that the cameras are not pointed at the same spot on the ground plane. For example, the south camera frustum crosses the center camera frustum and captures the area just to the north of the center camera's FOV. This has been very confusing to work with and makes frustum filtering pretty much useless. I'm going to be changing it on the hardware soon.

[csparker247]

Oh, I also wanted to ask about rigidly coupled cameras (#580). This is almost exactly the use case I have. I'm betting that since the branch is multiple years old, it probably would need some significant work to get running on top of a newer develop. Any thoughts on the likelihood that this feature would be of benefit to my reconstructions?

[pmoulon]

Hello @csparker247
Sorry for the late answer.

Happy to see that you were able to make some progress on understanding what is happening with a single rig pose.
Since we don't fix the rotation of a camera, it is possible that the BA made a 180 rotation of your camera (or simply than between your camera coordinate system on the default one of OpenMVG one axis is flipped)

If you want to compare two SfM camera poses coordinates you can do it with this function https://github.com/openMVG/openMVG/blob/develop/src/software/SfM/main_evalQuality.cpp#L160 (just load your two sfm_data scene and add the parameters)

• Let me know if the random dot pattern or new journal paper could help

You are right the branch #580 was a prototype feature branch and I think it would take quite a lot of work to merge develop in it.
A similar concept could be reused

[csparker247]

Thanks @pmoulon. I think my pose import is incorrect. I've rotated all of my cameras to match the "upside down" frustum, and Incremental2 with imported poses is now working on a small test case.

I think I might have been confused because I was coloring the wrong face in the export frustums code. Can you test the frustum color export with whatever known poses you might have? Even the identity pose. I think I might be coloring the incorrect face as the top of the frustum...

EDIT: Well that was disappointing. False alarm on the reconstructions working with priors. I was setting my script's flag to use priors, but then was overriding the sfm initializer with MAX_PAIR, so poses weren't actually getting used. Regardless, I'm still not confident in the frustum coloring, so that question remains.

[csparker247]

Success! I've finally been able to import poses in such a way that Incremental2 will at least accept them and work with them. My biggest error was that all of my pose rotations were originally in a right-handed system, but I was inadvertently converting to OpenMVG's left-handed system in two different places. It took me a long time to find this, though, because I assumed that the frustum coloring code I contributed was correct. I'm now about 99% sure that the colorized face in the frustum export is actually the bottom of the frustum and not the top. If anyone else can help confirm this, please let me know, and I'll submit a patch.

Now that the pose import is fixed, the issues with my pose priors are more clear. In particular, there's enough disparity between my imported positions and the actual positions that I'm getting close but distinct planes of features points when I try to directly triangulate them:

This makes sense to me. My provided z-positions are probably off slightly for each camera, so each camera gives me a plane of features. My intuition is that I should be able to run bundle adjustment in ComputeStructureFromKnownPoses to perhaps fix this. However, as before, when I enable bundle adjustment with direct triangulation, all of my points are filtered out and I get an empty scene. Do you have any ideas what principle of the BA filtering I might be violating?

I'm sure that I'm going to have more questions soon, but it seems as though the main issues I was experiencing are now solved. I'm now also curious if I can give back by way of documenting these processes. I've also got my Python import code that I could post in an open repo if you think it might be helpful to others.

[pmoulon]

Hello @csparker247
In order to test that the right faces are colored we can use this dataset https://github.com/openMVG/ImageDataset_SceauxCastle

I don't understand yet why Incremental2 & the Bundle Adjustment reject your points, we would have to take a look to the residuals values and check what is happening.

In the past, we were able to import multiple datasets successfully, but yes we could add a wiki page about the process:
https://github.com/openMVG/openMVG/tree/develop/src/software/SfM/import
and a minimal python example.

[csparker247]

In order to test that the right faces are colored we can use this dataset https://github.com/openMVG/ImageDataset_SceauxCastle

Tested, and it's definitely a bug. I've submitted a patch here: #1773

I don't understand yet why Incremental2 & the Bundle Adjustment reject your points, we would have to take a look to the residuals values and check what is happening.

I now have a good dataset that I feel confident is a good test set. I've run it through a full OpenMVG + OpenMVS pipeline with various settings (global, sfm2, sfm2+priors), and I've kept all of the intermediate files. So I can provide all sorts of things for you to look at.

I also have a new, small test dataset that you can run yourself. The zip has the images and an SfM file with the poses already imported. There's also a text file with the line you'd need to add to the sensor width database if you need to import the images yourself. This dataset looks excellent with GlobalSfM, has problems with Incremental2 (with and without priors), and does not reconstruct at all with direct triangulation + bundle adjustment.

In the past, we were able to import multiple datasets successfully, but yes we could add a wiki page about the process: https://github.com/openMVG/openMVG/tree/develop/src/software/SfM/import and a minimal python example.

I can definitely work on that. Thanks again for all of your help!

[csparker247]

Just to keep things updated here for posterity and to help others, I've used the Charuco boards implementation in OpenCV to improve my pose estimates relative to my motorized scanning head. Depending on the camera, my previous pose estimates had minor to major deviations from what the Charuco boards gave me. The good news is that the improved pose estimates further improved the results returned by Incremental SfM2 and direct triangulation. However, neither yet match the (admittedly subjective) reconstruction quality of running Global SfM on my full dataset.

I had a couple of goals for importing my pose priors: consistent reconstruction quality across samples, exporting scenes with the appropriate real-world scale information, and improved reconstruction time. My performance bottleneck is now no longer in OpenMVG, but in the further MVS reconstruction steps, so that goal is achieved. With the success of using the Charuco boards, I can import scene scale via another method, so that is also acceptably accomplished as well.

The only thing that really remains is consistent reconstruction results. While it's probably not ideal to have to run SfM for every single dataset (increasing the chances of introducing inconsistency in results), it's certainly an acceptable fallback if my global results remain as good as they have been. Also, and I haven't tested this, maybe my mechanical error is minimal, and I can use a "calibration" global reconstruction result to preinitialize my scene for every scanned object. Then I could skip the recomputation of SfM for every dataset and just run BA. Just some random thoughts.

Anyway, I think I'm winding down this inquiry for now. I still want to write some sort of guide to help others import their own datasets. To that end, I've released my SfM utilities package on PyPI and GitHub. I will definitely write a guide for using that package, but will also try to do so for the OpenMVG interface as well.

Thank you for all of your help, @pmoulon!