multi_view_reconstruction with depth map

[tharinduk90]

Hi! Amazing paper, and very clean code! Thanks so much for releasing this.

I am trying to use multi_view_reconstruction with depth map for 3d reconstruction (ours_depth_mvs.yaml).

So far i can obtain results with multi_view_reconstruction without depth map (ours_rgb.yaml).

When i use the depth map with multi_view_reconstruction(ours_depth_mvs.yaml) , i did not get any output.i.e output in empty.
The depth values are in line with the camera projection matrixes(both are in mms).My dataset is below attached.

scan155.zip

i have refer below issues, when generating the data set.

#3
#16

q-1) Can u help me with this?
q-2) Does the depth map should be too perfect?

[tharinduk90]

using the https://github.com/intel-isl/Open3D/blob/master/examples/python/Basic/rgbd_image.ipynb, i visualize the depth maps.
the result for 000020.jpg with different angles is attached below.

original image

point cloud with angle 1

this is the point cloud
https://drive.google.com/file/d/13LZMxohl3a0IdkJvw5c3bUMcVne6hRY1/view?usp=sharing

the above point cloud file can be view with https://www.creators3d.com/online-viewer

[m-niemeyer]

Hi @tharinduk90 , I was now able to have a look at your data.

The depth you provide is very noisy / not accurate. The problem with your visualization above is that you only show the predicted points if you use a single image, and also you show it from the same view as shown in the image. Then, it will always look good! The question is rather how well does the point cloud look if you use all the views, and project them all into the world.

Here is a screenshot of the point cloud if you use all 22 views:

And here the respective ply file (zipped).

As you can see, the depth is so noisy that it does not lead to a consistent model. Hence, if you train with this depth information, it will rather worsen the results than improve them. If you want to improve your results, a good tool to obtain camera parameters as well as depth maps for multi-view images is Colmap.

Good luck with your research !

[tharinduk90]

@m-niemeyer , thank you very much for the clarification, can u share the script to create the point cloud ? So i can verify my depth map?

[m-niemeyer]

Hi @tharinduk90 , thanks a lot for your message.

I have now created a FAQ file describing the camera matrices and also an example script of how pixels can be projected to the object-centric world space using the matrices and depth values: https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/project_pixels_to_world_example.py

I hope this helps. Good luck!

[tharinduk90]

@m-niemeyer , thank you very much !!!!!!!!!!!!!!