Wrong colors/color skewing when using global seam leveling

[svetlins]

Hello,

I'm trying to texture a mesh with some aerial imagery (as part of a larger toolchain - OpenDroneMap). The results I'm getting are somewhat wrong and I've traced the color skewing to mvs-texturing and more specifically to the the global seam leveling option. Here is the nvm, images and ply file as well as screen shots of the results I'm comparing.

Here is the result with --skip_global_seam_leveling:

(exact command used: texrecon pmvs/nvmCams.nvm mesh.ply output/good -d gmi -o none --skip_global_seam_leveling)

And here is without:

(exact command used: texrecon pmvs/nvmCams.nvm mesh.ply output/bad -d gmi -o none)

The texrecon executable was built by me against 93a261f which was the latest master at the time of writing this issue.

One interesting tidbit/observation that may as well be wrong but it looks like this color skewing is noticeable on all aerial datasets I've tested with but is more pronounced on ones that were shot with some kind of color filter as this one is.

I'm ready to provide any other details if you need them.

And thanks a lot for your work!

[rion-saeon]

Another case image:

https://cloud.githubusercontent.com/assets/13643064/21981199/ee7c846c-dbee-11e6-8ccd-2a74cb632c5f.png

[rion-saeon]

Mesh and Orthophoto anomalies issue #454

[nmoehrle]

Thanks, for raising the issue and providing an example.
I am sorry that I haven't addressed this issue yet and that I cannot tell you when I will have time to do so...

[dakotabenjamin]

I will also provide another example of color balance issues. Maybe it is related to #59 as well?

edit: provided a better reconstruction and with global seam leveling disabled below:

[smathermather]

Hi @nmoehrle -- is there any way we can help, either by providing development resources or other issues?

[nmoehrle]

Thanks for the offer!

I have two deadlines coming up that currently require all of my attention so I cannot experiment with these datasets myself.

That being said I could not help myself and had a look at your first dataset. If you reconstruct it without any seam leveling you see how different the colors of the input image actually are. The issue here is that our global seam optimization has due to the view selection little to no constraints on how to adjust the images. There are many regions where it has only three image, with wildly different colors, that separate all other images so it simply distributes the errors.

The only Idea that I currently have is regularizing the global adjustments. In datasets with less boundaries this happens implicitly and a regularization would hurt the quality there but for these datasets it might be necessary.

The relevant section in the paper is 4.3 equation (3) the implementation is in global_seam_leveling.cpp#L243. In theory it should be possible to simply multiply with another regularization matrix that penalizes adjustments in general. If you search for a reference the wiki entry for Tikhonov regularizations is actually quite good.

The issues of the other datasets look different to me, I belief that alignment (and thus the adjustment of non corresponding colors) is more of an issue there.

I am happy to answer questions but I cannot dig into it myself right now.

Best,
Nils

[nmoehrle]

Only global seam leveling without regularization:

Only global seam leveling with regularization:

Seams like a regularization could work - could you guys do some tuning and testing?

[dakotabenjamin]

I have a few datasets run with a lens filter so yes I can do that.

[svetlins]

@nmoehrle that original dataset looks much better! I will run a few other datasets and will post results here. (I assume I need to build the regularization branch)

Thanks again for your awesome work! :)

[nmoehrle]

Yes the regularization is the multiplication with a weighted identity matrix in this line that puts a quadratic penalty on all adjustments. The hard coded weight is the first thing that came to my mind - so its arbitrary - and requires tuning or should be exposed to the command line...

[dakotabenjamin]

I will be testing on this dataset: https://github.com/OpenDroneMap/odm_data_seneca You can see the mosaic I've generated from the original mvstex results in the README.

[dakotabenjamin]

For tuning, what kind of range are we looking at? This kind of math is far beyond my understanding, so the best I can do is guess and check the numbers.

So far the results look good for me using the default 0.0001

[dakotabenjamin]

Tuning orders of magnitude, 0.0001, 0.001, 0.00001
https://clemetparks-my.sharepoint.com/personal/dmb2_clevelandmetroparks_com/_layouts/15/guestaccess.aspx?docid=156a2a039a8b14d14b1005111e488d4cc&authkey=ATO2sgKPA0N0qvVueStyiLo

Sorry for the bad file labeling.

[dakotabenjamin]

Visually, 0.0001 looks better. 0.00001 has too much darkening along the edges and 0.001 has these patches of light and dark. I would say the former does the best job accounting for some of the vignetting and cloud-cover issues from the original dataset.

[nmoehrle]

Wow, lucky guess I guess ^^

Indeed with 1e-3 we can already see how the leveling is degrading and 1e-5 is not strong enough for the borders, but I think this will depend somewhat on the dataset so I wont merge it into master right away at least not without exposing it to the command line.

It would be interesting to see the results when we only regularize the borders...

Thanks for testing!

[smathermather]

Looks promising:
OpenDroneMap/odm_data_seneca@fec053a

https://github.com/smathermather/OpenDroneMap/tree/mvs-global-illumination-patch

[rion-saeon]

Before (didn't pass --mvs_texturing-skipGlobalSeamLeveling):

After (smathermather/OpenDroneMap: mvs-global-illumination-patch Branch):

[nmoehrle]

Was there a cloud? Or is it still failing?

[rion-saeon]

Cloud.

[smathermather]

Outstanding. Dakota and I are testing a couple of other tricky datasets today. We'll let you know how it goes... .