apply nerf on interiornet

[ttsesm]

Hi @kwea123,

Thank you for this great work. I wanted to ask you whether do you think that I could use nerf to create the 3D mesh of the scenes from the Interiornet dataset (https://interiornet.org/). The Visim application that they show in their video is not available until now and moreover the provided ground truth, it doesn't seem to fit the renderings. However, they provide some renderings, including depth and albedo maps and thus I was thinking whether nerf could be useful.

[phongnhhn92]

Can you provide some images from this dataset ? If we can run colmap on this dataset then I think it might be possible.

[kwea123]

Theoretically it is possible, however the current implementation doesn't support complicated trajectories as shown in the video. You need to write your own code to convert the poses and the bounds such that the whole scene lies approximately in the cube [-1, 1]^3. For example the current llff spherical data puts the origin at the center of all poses and adjust the bounds accordingly, but this won't apply if the poses are not spheric around some point.

[ttsesm]

@kwea123 I see, yes I knew that most likely it would not be straight forward.

@phongnhhn92 you can download one scene from the following links:

1. https://drive.google.com/file/d/1FHupJ3nOhXXVibxvNc6Mg__k34_ekOp4/view
2. https://drive.google.com/file/d/1oyXVDVSExPGoPy_AmGCckIMkY0sYJr-Z/view

The first one contains the "ground truth" data and it would look like this:

where the 3 folder contain the camera information for 3 different renderings and the .xml files contain the scene information with two different lighting modes. I've tried to parse the .xml file in order to extract the 3D mesh of the scene but it seems that the output is not the same as in the rendered images if you check below.

The second link contains 3 different renderings for each lighting mode (i.e. original, random):

inside you will find the images for each rendered map, e.g. albedo, pinhole cam, fish-eye cam, depth, etc...

Now I was thinking that by loading the images from "cam" together with "depth" I should be able somehow to extract the 3D point cloud/mesh of the scene. I've tried it with open3d but the output was not good. I did not try colmap though.

Let me know if you have any other question.

[kwea123]

If you can successfully convert the poses to adapt to these complex trajectories, it would be really helpful. Please let us know if you manage to do so.

[ttsesm]

@kwea123 what do you mean with "convert the poses to adapt to these complex trajectories"

In the ground truth folder they provide the camera poses as following:

#timestamp [ns], position.x [m], position.y [m], position.z [m], quaternion.w [], quaternion.x [], quaternion.y [], quaternion.z []
0000000000031666668,0.303582847,-3.14995146,1.31473804,0.604126871,0.677215457,-0.313424408,-0.279597998

I've also tried to load the images to Colmap, but the output does not seem that consistent as you can see below:

[kwea123]

Like I draw here, suppose blue objects are the scene objects and black points is the trajectory, you need to convert the poses such that the origin is the scene center (red star) and the bounds are approximately from -1 to 1. By default I think the pose origin could be anywhere, that doesn't work for nerf if it's not at the center of the scene because of the positional encoding.

[ttsesm]

I see, hmm it doesn't seem to be straight forward. I'll check if I can do anything though.

[ttsesm]

apparently the follow up work https://nerf-w.github.io/ would be able to handle my case scenario :-)

[kwea123]

Yes, I'd also like to know the trick that they use to convert the poses. It doesn't seem that they'll open source though...