cost volume constraint - closer camera view

[kwea123]

As you said in #7 ,

we think building volume in the target view may provide a better depth quality but is not an efficient way to do the free-viewpoint rendering

I also feel that the fact that the cost volume must be in some reference view has many limitations. Leave alone view extrapolations, how does your method perform when you move the camera closer to the scene (not zoom in but physically place the camera closer to the scene)?

Currently, the LLFF scenes are all captured roughly at the same distance to the scene, so interpolating at this distance seems great as your video in the readme. However I wonder how it performs in the situation I describe above. With the current cost volume, I can think of two problems:

1. Although this novel view is still view interpolation (it lies inside the frustums of all reference views), the view to the nearest reference view is very far.
2. The novel view lies inside the cost volume, a situation that is not seen at all at training time.

To my knowledge, traditional MPI methods cannot handle this kind of situation well either. NeRF performs excellently without problem since it reconstructs the whole scene in 3D without relying on reference views in test time. Following is an example where I move the camera very close to the scene (left is the nearest reference view and right is NeRF synthesized result).

Pardon me for not being able to run your code myself. I didn't find an easy way to run your code with specifying a pose...

[kwea123]

Also, I think although in MVS setting the cost volume is at one of the reference view, in general there is no such constraint, you can build the cost volume whereever you want. The implementation is also straightforward.

[apchenstu]

Yeah, it's interesting to see what happens if physically places the camera closer to the scene, I also haven't tried this situation before. I think the easiest way to test this setting is to specify the "c2ws_render" matrix in the "renderer_video.ipynb" renderer script. are there some bugs or some other reason that you can't able to run the code?

[andy970226]

Also, I think although in MVS setting the cost volume is at one of the reference view, in general there is no such constraint, you can build the cost volume whereever you want. The implementation is also straightforward.

Do you think we can build a sparse voxel octree to store the cost volume, and thus enable free view-point and real-time rendering?