Plan to implement NeRF-W

[kwea123]

Hi, I recently started implementing NeRF-W, and my experiment (on the lego dataset, section D) seems working successfully: https://github.com/kwea123/nerf_pl/blob/nerfw/test_nerfu_occ.ipynb

Currently I only tested NeRF-U with occluders, the code is ready for other experiments, I just need some more time to run all the trainings.

My branch: https://github.com/kwea123/nerf_pl/tree/nerfw

It would be great if you can include this as a reference source to NeRF-W, as it seems that they won't publish the code (and the data).

Update: NeRF-A with color perturbation also passed. https://github.com/kwea123/nerf_pl/blob/nerfw/test_nerfa_color.ipynb

Update: NeRF-W with color perturbation & occluder passed. https://github.com/kwea123/nerf_pl/blob/nerfw/test_nerfw_all.ipynb

These experiments should be enough to demonstrate that the implementation is almost correct.

[kwea123]

I thought the data wasn't provided, but actually it is. https://github.com/ubc-vision/image-matching-benchmark
I will find time to try to run on real data next!

[kwea123]

Update: I successfully reproduced visually-identical results on the phototourism dataset!
https://nbviewer.jupyter.org/github/kwea123/nerf_pl/blob/nerfw/test_phototourism.ipynb
Due to resource issue I can only train on 8 times downsampled images, so it looks blurry, but the effects of the paper can clearly be seen. I'm now training on full resolution on 8 GPUs, it's going to take a few days...

I'll submit a PR soon.

[kwea123]

The final step consists of creating poses for a flythrough video as shown in their github project page. I'd appreciate if anyone has experience on this, otherwise I need to do some research.

[yenchenlin]

Hi @kwea123 this is super exciting news! I will be happy to merge it once the high-resolution results are out.
For now, I will leave the issue open so people know that nerf-pl will support "NeRF in the wild" features soon. Is that okay for you?

[kwea123]

Yes, thanks. I will update here once the high resolution results and the flythrough video are ready!

[sungam94]

Awesome! Just in time to reference it in my thesis ;) I might be able to create poses through creation of a point cloud in colmap and using blender to create a path.

[kwea123]

Hey @sungam94 (and others if interested),
brandenburg_data.zip is the xyz points (the gate) and the training poses (maybe not required?) computed with my code, same as what is in here. Are you able to generate some flythrough poses?

[kwea123]

Hi @yenchenlin @sungam94
High resolution training (on 1/2 scale) is done, and I also did a really quick camera path design which is just moving the camera towards the gate.
https://github.com/kwea123/nerf_pl/tree/nerfw#brandenburg-gate-from-phototourism-dataset

I think my code has reproduced comparable results with nerf-w, and is ready to be used for others now. Please review my PR, thanks!