Add support for P4 camera+sun calibration

[pierotofy]

Currently the Phantom 4 multispectral sun sensor calibration does not work properly. Some modifications to the code need to be implemented to support it.

Reference: https://dl.djicdn.com/downloads/p4-multispectral/20200717/P4_Multispectral_Image_Processing_Guide_EN.pdf

Community thread: https://community.opendronemap.org/t/dji-p4m-segmented-orthophoto/12039/15

[Christoffer-Svenningsson]

I was curious whether ODM actually was reading all the meta-data from images coming from P4 correctly and noticed that there are several members in a loaded ODM_photo instance set to None or 0 which probably should not be. I am trying to add XMP-tags where i see fit in ODM_photo but I am very new to working with this kind of data and I struggle to translate what everything is.

For instance:
Is DLS (yaw pitch roll) same as Irrradiance (yaw pitch roll)?
Is DarkLevel same as BlackLevel?
What is horizontal irradiance and how does it compare to irradiance?

I am searching for information but if anyone has input it would be helpful!

Update: Just so no one else falls in this pitfall. The "Camera:RadiometricCalibration" tag in P4 should probably not be used and cant be compared to the radiometric calibration of micasense cameras, just ignore it. The first element is just a copy of "@drone-dji:SensorGainAdjustment" and can thus be greater than 1 which makes no sense at all for a radiometric calibration value.

Update2: My modifications to the code seems to be calculating indices such as NDVI correctly now. But i cant get the individual bands correct. I believe this is because I dont understand how to calculate rho_X as mentioned in the DJI P4 Multispectral Image processing guide . Please tell me if anyone has an idea of how to calculate this.

[pierotofy]

I don't think you actually need to compute $\rho_x$ directly, to my understanding the white paper boils down to implementing:

XCamera = dn_to_radiance code
pCam_X = sensor gain adjustment from XMP
XLS * pLS_X = spectral_irradiance (MicaSense calls this horizontal irradiance, also read from XMP)

I've added some changes with #1513 and I'm running some tests to see if the results look plausible.

[pierotofy]

And yes I don't think we want to use Camera:RadiometricCalibration.

[Christoffer-Svenningsson]

I don't think you actually need to compute ρx directly, to my understanding the white paper boils down to implementing:

XCamera = dn_to_radiance code pCam_X = sensor gain adjustment from XMP XLS * pLS_X = spectral_irradiance (MicaSense calls this horizontal irradiance, also read from XMP)

Yes this is what i tried to do aswell. (My code was horrible which is why i did not share it by the way). This makes the normalized indices such as NDVI look ok. But the individual bands still seems off. Maybe I am missing something here? Or do we simply ignore the individual bands and only do this to use with indices?

[pierotofy]

Can you post some screenshots?

I would also recommend to test #1513 and see how it compares to what you wrote.

[Christoffer-Svenningsson]

I will attempt to connect to the computer in which I have the images and get them. Give me a few minutes!

[pierotofy]

I also think the polynomial coefficients for vignetting for the Phantom 4 are stored inverted compared to other sensors like the Micasense.

[Christoffer-Svenningsson]

This is what qgis "default" min/max colorthreshhold gives. Note that there does not seem to be a common ceil for the values? How can i even determine this when representing the image

This is what i get if i set the same colortreshhold.

This is with global seam leveling on.

It should be mentioned that this was a sunny day and pretty much all colors should be similar to the lighter patch (more or less) as seen in the second image

Also, I will test #1513 tomorrow for sure. This is quite exciting, glad you are taking your time to add support for this messy drone!

[Christoffer-Svenningsson]

I also think the polynomial coefficients for vignetting for the Phantom 4 are stored inverted compared to other sensors like the Micasense.

Yes, from my understanding we do not multiply by 1/V(x,y) but instead just V(x,y). I changed this aswell. If I understood you correctly.

[pierotofy]

Correct.

I'm getting decent results with #1513

I think I'll go ahead and merge it, that way more people can start testing it out. We can fix other issues as they come out.

[Christoffer-Svenningsson]

Correct.

I'm getting decent results with #1513

I think I'll go ahead and merge it, that way more people can start testing it out. We can fix other issues as they come out.

This was a better example of the miscoloring for the RGB bands i was talking about, sorry i could not find a better example. Anyways, great job and thank you for fixing this again!

[Christoffer-Svenningsson]

This is what i got with #1513. It's looking kind of strange.

NDVI in grey scale

I dont know. Maybe this dataset is just bad for some reason

[pierotofy]

Maybe. Can you share the images?

[Christoffer-Svenningsson]

Maybe. Can you share the images?

You mean the raw data right? Im just waiting for an OK to share it just to be certain

[Christoffer-Svenningsson]

Hi, here is the dataset. It's my first time using dronedb. I hope i made it public so that you can view it!
The flight was automatic and with 75% side overlap and 80% front overlap.
DroneDB P4M Test dataset

[pierotofy]

There's definitely some value intensity issue going on with this dataset, but I'm not sure if this is something that we're not correcting properly in ODM, or if the sensor just wasn't calibrated properly prior to data capture, or something else.

I've checked the code and we're doing everything that DJI documented to do, so I'm not sure.

[pierotofy]

One thing I made sure to check is that mvs-texturing is not causing the artifacts, I manually overlaid the RGB images after radiometric-calibration and this is what came out of it:

It just doesn't look correct, but I'm not sure how to adjust it (especially given that other datasets seem to work).

Another image from the dataset instead looks more correct:

So perhaps some images are good and others are not, explaining the strange intensity patterns in the final orthophoto. But feel free to poke around, perhaps I've missed something.

[pierotofy]

In particular, both sensor gain and exposure time between the good/bad shots are widely different:

DJI_0771.TIF
drone-dji:SensorGain="3.750"
drone-dji:ExposureTime="3040"

DJI_0311.TIF
drone-dji:SensorGain="1.000"
drone-dji:ExposureTime="283"

I'm not sure why that is, but it's probably the cause.

[Christoffer-Svenningsson]

In particular, both sensor gain and exposure time between the good/bad shots are widely different:

DJI_0771.TIF
drone-dji:SensorGain="3.750"
drone-dji:ExposureTime="3040"

DJI_0311.TIF
drone-dji:SensorGain="1.000"
drone-dji:ExposureTime="283"

I'm not sure why that is, but it's probably the cause.

So the testdata i have been testing my changes to has been bad... Thats painful to hear. I wonder if the drone is broken. Thank you for this investigation anyway. Right now im out of ideas on what to investigate. But I will write here if anything comes up!

Edit: Maybe the fact that we are missing the rho_nir factor is playing a role here?
Edit2: I found an old discussion about this issue on the DJI forums. It seems like im not the only one having this issue https://forum.dji.com/thread-231375-1-1.html. Feels like a relief to know that the drone might not be broken atleast