iNeRF

[rooi]

Hi,

Thanks for sharing this research and repository! NeRF has become much more accessible opening up new research and use-cases.

Are there any plans to include iNeRF functionality to derive camera poses relative to a trained model? Having this key functionality easily accessible as well would open even more use-cases.

Either way thanks,

Roy

[claudiomla]

I'm interested in that. I was watching the code and I believe that it's possible, but don't know how yet.

[Tom94]

Hi there, instant-ngp supports training of camera extrinsics (as well as an intrinsic distortion grid for that matter). You can simply tick the "Train extrinsics" checkbox in the GUI to enable it.

If you want to first pre-train a model with known extrinsics and then reconstruct the pose from additional viewpoints, you can

• train the model,
• save a snapshot,
• restart instant-ngp with a dataset corresponding to the unknown viewpoints (rather than the dataset the model was trained with),
• load the snapshot,
• uncheck "Train network" and "Train encoding",
• check "Train extrinsics"
• start training.

Since this sequence of steps is rather cumbersome, I'd recommend using the Python bindings for this sort of stuff. These will also let you skip the checkpoint dance as they allow you to replace the dataset on-the-fly.

Cheers!

[rooi]

Thanks Tom, that sounds very promising!
I just tried to experiment using your suggested approach, but I guess I'm missing on perhaps obvious step.

How do you load a dataset with unknow viewpoints? I guess it is just using the --scene in the command that points to the folder containing the data with json file, but if this is true, what does the json file contain (e.g. how do you specify unknown viewpoints?). I' ve tried a couple of thing (e.g. removing the transformation matrix, changing the numbers in the matrix and such), but I could not get training just the extrinsics to work.

Could you please give an additional pointer? Thanks!

[Tom94]

The transformation matrix needs to be an initial guess of the pose. Ideally one that is close enough to the true pose (i.e. at least shares some of its pixels) such that the optimization can be expected to work. Cheers!