PoC Mavic 3M RGB unwarping

[pierotofy]

Opening early to encourage testing.

Since the Mavic 3M's RGB sensor differs significantly from the multispectral sensor, image alignment fails. We actually currently drop the RGB band to allow the program to at least process the rest of the bands.

This PR unwarps the RGB images before processing using the EXIF dewarp data field which is provided by DJI and rescale the images to match the focal lengths between the two sensors.

The images this way are pretty close, although on very small datasets (5 images) this still yields some alignment issues.

I will test with larger datasets in the upcoming days.

[danieldobosi]

good day, what can I do to help closing this issue?

[pfxuan]

Previously, I tested this PR with a series of DJI M3M datasets (50 ~ 400 acres). Everything works perfectly so far. However, one aspect I'm uncertain about is whether we need to execute the downstream calibration/unwarping from OpenSfM after applying unwarping on the source images.

[pierotofy]

Thanks for testing @pfxuan ! The unwarping from OpenSfM is needed because the reference images are unwarped also (and they need to align).

This PR needs more testing, in my (small) experiments this did not completely solve the alignment issue and since this PR introduces some complex logic, I'm hesitant to merge it unless it really shows some good improvements over baseline.

[pfxuan]

Thanks for explaining the OpenSfM unwarping part! From my previous tests, I found the following setup (Incremental + 2.5D mesh) is working particularly good in a high density of vegetation area. It can produce a better orthos than pix4d and metashape does. However, I was not able to produce a similar result using Triangulation and Plannar SfM-algorithm.

In addition, I found the names of DJI GPS tags are inconsistent across different DJI models and probably firmwares. e.g. in M3M, EXIF meta shows @drone-dji:GpsLongitude rather than @drone-dji:Longitude. I did some tests with the updated code in photo.py. But it seems like there is no improvement for Triangulation SfM-algorithm.

/code/run.sh
                --cog \
                --orthophoto-resolution 1.0 \
                --max-concurrency 20 \
                --use-hybrid-bundle-adjustment \
                --fast-orthophoto \
                --sfm-algorithm incremental \
                --feature-type dspsift \
                --feature-quality high \
                --min-num-features 45000 \
                --gps-accuracy 1 \
                --build-overviews \
                --auto-boundary

[Saijin-Naib]

In addition, I found the names of DJI GPS tags are inconsistent across different DJI models and probably firmwares

This is a never-ending nightmare that impacts practically every tag, including altitude, offsets, etc. Sorry this is impacting your work on this -_-

[smathermather]

Testing this this weekend with a 1805 image dataset, RGB and multispec 4-band from @IndiaTJohnson.

[smathermather]

Running into challenges in getting this dataset to process, but it's also my first multispectral dataset, so teething pains. @pierotofy -- do you have the test dataset you ran so I can do some comparisons re: tags and file naming conventions?

[pierotofy]

I actually can't remember which dataset I used, it was one from the forum, maybe one of these? https://onedrive.live.com/?authkey=%21ADbiQDpbCwdZLG8&id=8565DBF032EAB4CA%21450270&cid=8565DBF032EAB4CA https://community.opendronemap.org/t/misalignment-between-bands-in-mavic-3m-multispectral-images/15996/20

[smathermather]

Ran FujisawaMultispectral1, a 582 image dataset. It doesn't run at all on latest opendronemap/odm, and does complete on smartband. However the alignment is poor:

Happy to share logs if useful.

[pierotofy]

No need; I had misalignment issues in my tests also. Mm.

[smathermather]

You know this space much better than I do, so probably no new content/context, but is there something @wisescootering's approach that will help with gradient matching: https://github.com/wisescootering/infrareddrone?tab=readme-ov-file#-scientific-references?

[pierotofy]

Interesting approach; no license in the repo, but could be interesting to try aligning the images that we're having troubles with using the approach from the link and see if they work.

[balthazarneveu]

Hi @pierotofy @smathermather please tell me what licence you want me to put on IRdrone.
This is a difficult topic you're tackling here. I'd suggest you give a quick try to my python code (multiscale brute force alignment on a handcrafted feature map) to see if it fixes some of your problems.

There's a sample notebook in my repo, it is not finished and auto alignment is not plugged but it shows how to compute the feature maps to align

wisescootering/infrareddrone#34

[balthazarneveu]

Also just a very naïve question : isn't there a piece of DJI software which align the images.
There may be information in the EXIF which may help a lot pre-aligning

on the mavic air, gimbal orientation and drone orientation solved 90% of the coarse pre alignment. So in IRdrone, I pre-warp (jointly distortion + homography) using the DJI angles in exifs so the fisheye infrared image is roughly aligned.

[smathermather]

Hi @pierotofy @smathermather please tell me what licence you want me to put on IRdrone.

GNU Affero General Public License is the most restrictive compatible license (same one we use in most of the repos at github.com/opendronemap) and thus provides the most protection for you. If that's not one you like, regular GPLv3 is fine as are many permissive licenses.

There's a sample notebook in my repo, it is not finished and auto alignment is not plugged but it shows how to compute the feature maps to align

wisescootering/infrareddrone#34

Ah, brilliant! I'll take a look.

Also just a very naïve question : isn't there a piece of DJI software which align the images.
There may be information in the EXIF which may help a lot pre-aligning

Not sure. But, I'm meeting with India on Friday to do a deep dive. I haven't even seen the drone yet, but in pictures, it looks like the whole array of cameras is on a single gimbal, so the relationship between the different sensors should be fixed 🤞

[balthazarneveu]

✌️ done + you have my formal agreement that you can do whatever you want with this code 🤗

If the rig is rigid, it opens the realm of calibration which will simplify a lot of things (very tiny baseline may require extra alignment but things are much easier then). Really good news, let me know if you need anything

[smathermather]

✌️ done + you have my formal agreement that you can do whatever you want with this code 🤗

Excellent. Pull request with repo license added. Let me know if you want me to revise that with something more permissive.

I'll update as we get into it tomorrow. I'm looking forward to playing with the notebook.

[smathermather]

Found DJI's Mavic 3M Image Progressing Guide
Mavic_3M_Image_Processing_Guide_EN.pdf

Haven't dug into DJI post-processing maybe via the API yet.

[smathermather]

Found DJI's Mavic 3M Image Progressing Guide
Mavic_3M_Image_Processing_Guide_EN.pdf

Looking more deeply at the pull request, my hunch is the Piero is already reading these parameters out of the XMP tags and applying them.

[smathermather]

Ok, digging deeper and still well out of my depth, I think the missing step is maybe using the Dewarp HMatrix, which is also in exif:

[balthazarneveu]

From what I see in yours screenshot , looks like the calibration homography (close to identity = just a bit of misalignment) is the one you'll need to apply in the warp. This looks good if DJI provides this.. 95% shall be fixed ... The rest will be tiny parallax and unperfect Calibration ✌️✌️✌️🚀🚀🚀

[smathermather]

From what little I can discern for the process, I assume we'd apply a cv2.warpPerspective after cv2.initUndistortRectifyMap, or would we somehow bundle those transformations together?

[lewispb]

I've been playing with getting a good alignment locally and came up with this, based on the DJI image processing guide. I'm a newbie when it comes to CV and Python though so there's likely issues.

https://gist.github.com/lewispb/6c7dd6022ba3266cfd4f1310d4d51f5f

One thing I haven't attempted is using the sun sensor data for calibration. Seems to only be present in the MS images XMP, not RGB.

I've been testing this branch locally but was intermittently running into several stack overflows at the final orthophoto stage. I can share a dataset if it's helpful, or help with testing.

[smathermather]

I've been playing with getting a good alignment locally and came up with this, based on the DJI image processing guide. I'm a newbie when it comes to CV and Python though so there's likely issues.

https://gist.github.com/lewispb/6c7dd6022ba3266cfd4f1310d4d51f5f

This is great! Reading through (as a non-expert) this looks pretty solid. Would you be willing to do a pull request to integrate? As far as I can tell, just integrating the parse_homography_matrix and then use that padded warp_matrix in the align_image function prior to writing out would do the trick. But I'm no expert either, so happy to be corrected on approach.

Testing data would be great, and I can do some testing too. Are your overflows memory overflows, or a race condition, or something else? I can run a few different docker images with different memory allocated up to 1.5TB, if we need to do any testing on memory needs.

[smathermather]

I'm getting segmentation faults when I run your code snippet headless. I'm not sure if that's an opencv module that requires a gui or an issue with my configuration, but I thought I'd mention it.

Crunching though some test data now.

[balthazarneveu]

@lewispb you seem to be on the right track.
One point whereI hope I can help:

you can do both warping operations at once:

https://github.com/wisescootering/infrareddrone/blob/master/registration/warp_flow.py#L95-L138
cv2.initUndistortRectifyMap & cv2.remap allows warping with a generic flow ... so perform homography + distortion at once (which avoid 2 geometric interpolations).

[smathermather]

@lewispb -- do you have some test data to share? Perhaps unsurprisingly, my test data doesn't match your tags. It'd be good to cross walk them.

[smathermather]

Disregard: I have a matching dataset. And it works great.

[smathermather]

This looks quite good:

I had to disable multiprocessing, and we might have datasets that don't have the appropriate tags, in which case, we may have to fall back to DJI's published expected values.

@lewispb -- are you ok with licensing this GNU Affero v3 ala the repo: https://github.com/OpenDroneMap/ODM/blob/master/LICENSE?

Alternatively a pull request integrating it would be even better. :) Thanks for sharing this!

[lewispb]

@lewispb -- are you ok with licensing this GNU Affero v3 ala the repo.

Of course, that's fine.

I'm not sure I if I'd be able to get a PR ready - I already attempted an integration with the work started here but struggled to get past / understand seg faults in the final orthophoto stage. If I get some time this week I'll share more on those.

This is what I managed BTW - I just used the MS bands, not the RGB: