Colmap fails on forward driving scene

[phongnhhn92]

Hi,
I am trying to train NERF on a sequence of a forward driving scene like this.

I have used the colmap script from LLFF to estimate the pose but it doesn't work for me.
Need to run COLMAP Features extracted Features matched Sparse map created Finished running COLMAP, see /home/phong/VKITTI2/colmap_output.txt for logs Post-colmap Cameras 5 Images # 2 ERROR: the correct camera poses for current points cannot be accessed Done with imgs2poses
here is the link to download the image. I wonder if NERF can not handle these sequences or is this from COLMAP ?

[kwea123]

It should be a problem from COLMAP (which you can check by using its GUI), I think it's better to ask https://github.com/colmap/colmap/issues

Btw, I have worked on many real car driving scenes like this, including left turn or right turn, NeRF has no problem handling these, provided COLMAP correctly reconstructs the poses.

[kwea123]

Is there a reason why you don't use the real KITTI? I think maybe COLMAP works better on real images.

[phongnhhn92]

@kwea123 I try VKITTI first to see if NERF can work on the synthetic driving dataset. However, you said that you can train NERF with KITTI images? If it is then do you use COLMAP to estimate the poses (using LLFF code) or do you use the camera poses from the KITTI dataset and do something similar to the Blender dataset?

[kwea123]

No, I tried with own dataset, not KITTI. I used COLMAP with default automatic reconstruction without any problem. Maybe you should try using bright scenes with many feature points (like road paintings) and reduce the sequence length (~50 consecutive frames) first.

[kwea123]

LLFF code does nothing more than reading the COLMAP result and does some conversion, so if it fails it means COLMAP fails, it's COLMAP's problem.

[zgojcic]

Hi Kwea, just a quick question. When fitting the self-driving scenes did you use NDC space or not? Also, did you only use the front-facing cameras, or were multiple cameras (e.g. also side looking ones) used.

[kwea123]

I use NDC space. Concerning the cameras, I use multiple forward-facing cameras, perhaps that makes the difference. Recently I found that monocular sequence doesn't work well in this kind of driving scenes.

[zgojcic]

This is interesting, I have observed that this works well for monocular sequence (but the extrinsic and intrinsic are taken over from the dataset and not estimates using COLMAP). Adding the side cameras causes weird artefacts on the images. I thought that this could come due to the NDC space because this then kind of becomes an inside out scene. However, NDC assumptions should also be violated in the case of left and right turns that you mentioned above (and this works for you apparently)...