Problems when using multiple sets of images with multiple patterns

[ElectronicElephant]

Hello,

I successfully ran multiple cameras with a single set of images. i.e.,

multical calibrate --cameras h0 h1 h2 h3 r05 r08 r22 r47

- cam1
     - im0.png
- cam2
     - im0.png
- cam3
     - im0.png
- cam4
     - im0.png

, where the five boards are generated from cube_10x10.yaml config. The cost converges to 5.0243e+02, the RMS is below 1, and the visualization looks good.

However, after I changed how the boards are placed and took multiple sets of images, things get worse. (The cameras are static.)

multical calibrate --cameras h0 h1 h2 h3 r05 r08 r22 r47

- cam1
     - im0.png
     - im1.png
- cam2
     - im0.png
     - im1.png
- cam3
     - im0.png
     - im1.png
- cam4
     - im0.png
     - im1.png

The cost is around 8e+7, and the RMS is larger than 100. The visualization does not look good.

However, I'm pretty sure that the detections are good. For example,

So it looks like that there are some problems in the optimization process, when there are multiple sets of images. How can I diagnose what's going wrong?

Best wishes,

[oliver-batchelor]

Hi!

Thanks for having a look! I'm not quite sure what's going wrong, however... there are a couple of things to check. Can you check the logs and see what it says around the initialization? If this goes doesn't initialize robustly then the optimization likely won't converge.

You can also check this in the UI - up the top there's a dropdown which says Stage: Calibration - there will be another entry "Initialization" - have a look what the projected patterns after initialization look like.

Usually when I've had this problem previously there is some mismatch between my board config and how I've printed the patterns, but since you've used my config file this seems less likely.

I would say your line of reasoning is probably good - but this is the normal use case for the software (i.e. this is how I'm usually using it, multiple boards, multiple sets of images and multiple cameras)

Cheers,
Oliver

[ElectronicElephant]

Hello @oliver-batchelor !

Thanks for your detailed reply, and this awesome project!

I have checked the initialization stage and found that it looks not good.

And here is the log.
log.txt

Also, I have checked all detections, and found there is no false positive, although the true positive rate is low. For example,

Does it mean that I need to improve the image quality? (Probably I should go for a better camera.)

Finally, I'm glad to share the images I used. I will appreciate it if you could take your time to help me check them.

https://drive.google.com/file/d/1qfEU-3ftzjOda6p4CzvquxMzJUpY64LW/view?usp=sharing

The command I used is

multical calibrate --cameras h0 h1 h2 h3 r05 r08 r22 r47

Bests,

[ElectronicElephant]

After a few days, I came to realize that during the calibration process, the relative position of boards should not be changed. This explains why, in my situation, it works when there is only one set of photos, but not when there are multiple sets of photos.

Great thanks for this awesome repo!

[oliver-batchelor]

Hmm. During the calibration process the relative position of the boards IS refined, though this can be turned on and off (see --fix_board_poses option).

…

On Sat, Feb 26, 2022 at 12:23 AM T.T. Tang ***@***.***> wrote: After a few days, I came to realize that during the calibration process, the relative position of boards should *not* be changed. This explains why, in my situation, it works when there is only one set of photos, but not when there are multiple sets of photos. Great thanks for this awesome repo! — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAITRZISSE2ADFNGV3PNEVLU45REPANCNFSM5OAON67A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kl2005ad]

Hmm. During the calibration process the relative position of the boards IS refined, though this can be turned on and off (see --fix_board_poses option).
…
On Sat, Feb 26, 2022 at 12:23 AM T.T. Tang @.> wrote: After a few days, I came to realize that during the calibration process, the relative position of boards should not be changed. This explains why, in my situation, it works when there is only one set of photos, but not when there are multiple sets of photos. Great thanks for this awesome repo! — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAITRZISSE2ADFNGV3PNEVLU45REPANCNFSM5OAON67A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.>

@oliver-batchelor Do you mean that setting "--fix_board_poses False" should allow multiple boards change relative poses when taking multiple sets of images? I tried and found that it does not work no matter I turn it on or off.
My case is multiple static cameras and I assume by default it should allow the boards to change poses between different sets of images, because otherwise only one moment of images can be used...
If it is like this by design, can you suggest an easy way to modify the code so that it can work on multi-camera image sets of multiple boards that are moving independently across time?

[oliver-batchelor]

I don't think with multiple boards there's ever a good reason to set "--fix_board_poses True" unless you have a way of precisely initialising them (e.g. by a previous known good calibration), though this isn't possible from the command line app.

…

On Tue, Nov 1, 2022 at 12:03 PM kl2005ad ***@***.***> wrote: Hmm. During the calibration process the relative position of the boards IS refined, though this can be turned on and off (see --fix_board_poses option). … <#m_2603391919270830726_m_695677814863510566_> On Sat, Feb 26, 2022 at 12:23 AM T.T. Tang *@*.*> wrote: After a few days, I came to realize that during the calibration process, the relative position of boards should not be changed. This explains why, in my situation, it works when there is only one set of photos, but not when there are multiple sets of photos. Great thanks for this awesome repo! — Reply to this email directly, view it on GitHub <#22 (comment) <#22 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAITRZISSE2ADFNGV3PNEVLU45REPANCNFSM5OAON67A <https://github.com/notifications/unsubscribe-auth/AAITRZISSE2ADFNGV3PNEVLU45REPANCNFSM5OAON67A> . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: @.*> @oliver-batchelor <https://github.com/oliver-batchelor> Do you mean that setting "--fix_board_poses False" should allow multiple boards change relative poses when taking multiple sets of images? I tried and found that it does not work no matter I turn it on or off. I assume by default it should allow the boards to change poses between different sets of images, because otherwise only one moment of images can be used... — Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAITRZP7DWZEQETNYA7Z6K3WGBF3HANCNFSM5OAON67A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kl2005ad]

I agree with you. But my point is that multical now does not support using multiple moving boards to do multi-camera calibration. I just wonder how to change the code to make it possible.