Explanation of the depth map cutting

[sweeneychris]

Thanks for a seriously AWESOME library! It worked for me right out of the box and has given me really great results on most scenes I've run it on.

I'm interested in learning more about the depth map cutting (which is WIP I guess): https://github.com/flanggut/smvs/blob/master/lib/mesh_generator.cc#L24

It seems that the idea here is that you compute a consistency score by projecting points out from the depth map, computing the normals, then compare those normals to the values in your normal map. That all makes sense to me. However, I am getting a bit lost in the lines 127-141 https://github.com/flanggut/smvs/blob/master/lib/mesh_generator.cc#L127

These lines seem to be filtering the data somehow based on... the depth consistency? Could you clarify what is going on in those if statements and why, for example, you want to compare surface_power_j_j > 0.5 * surface_power (where does the 0.5 come from?!), etc.

Thanks and again, awesome work!

[flanggut]

Hi! First of all thanks, I'm glad you're able to use the code.

The depth map cutting is still kinda sketchy and I'm trying to figure out a more serious way to do this. The original idea, including the surface power, is already presented in Semerjian 2014. But in general there are two reasons why I want to do this:

• The first idea is that I do not want to export all generated depth values to the surface reconstruction, mainly to speed up computation times. In theory the SMVS reconstruction should already generate a high quality surfaces in a single depth map. So theoretically we just need to decide which depth map reconstructs which part of the object best and export this to the surface reconstruction (with some amount of overlap).
• The second reason is simply to remove outliers. For complex scenes with many occlusions/disocclusions outliers can happen frequently, so I also want to check if the depth I am exporting is consistent in at least one other depth map.

What happens after L127 is broadly the following:

• We have a point from the current depth map (D1) and it's projection into another depth map (D2)
• I first check if D2 has also reconstructed this part of the surface by checking if the point generated by D1 projects into D2 and coincides with the depth value that has been reconstructed there
• At this point two things can happen:
  1. The depth of D2 is smaller than expected. In this case I cannot say anything because D2 has probably seen a totally different surface which occluded the surface visible in D1
  2. The depth of D2 is larger than expected. This is where it gets interesting, because D2 has essentially seen "through" the surface that has been reconstructed in D1. So one of the two reconstructed points must be wrong. It is very hard to tell which one, so what I do is I accumulate how many other depth maps have seen the surface from a similar angle (which is what the surface_power_j_j measures (I need a better name for this)) and agree / disagree with the depth value in D1 - if most of them agree I keep the value in D1, otherwise I remove it. If the other depth map has seen the surface from a way better angle (> 2.0 * surface_power) I immediately remove the depth value in D1 because it is most likely wrong.

Does that help?

[sweeneychris]

Thanks so much for a detailed insight into the depth cutting. I figured it was something similar to what you are saying, but it is good to have additional insight and confirmation. Thanks again!

[sweeneychris]

Do you have any suggestions for outputting the depth maps from smvs as images? I have tried to read in the smvs-cut.mvei and smvs-S1.mvei files but I get weird output that looks incomplete and much lower resolution.

Input image:

SMVS-S1:

SMVS-cut:

[flanggut]

This looks kind of reasonable to me, see my mail. Posting it here for reference:
You’re reconstructing mostly specular materials and there’s not a lot of texture on the van. This is basically the worst case scenario for MVS as it is not really robust for this type of data. Other reasons could be missing neighbors or inaccuracies in the initial SfM.

The reason for why it looks so blocky is that the optimization works on larger patches and not on single pixels. You can optimize on smaller patches using the -o option.

Note that the algorithm in general is tuned for high quality multi-view reconstruction and not simply generating the most complete depth map for each image. If there is a hole it is because we would rather have no geometry than incorrect geometry. This is usually different from something you would do for regular stereo.

[flanggut]

Closing. Feel free to reopen if further questions arise.