refnerf-pytorch-lightning

This is an implementation of Ref-NeRF, which is extended from refnerf-pytorch based on the original jax code released by Google.

Setup

# Clone the repo.
git clone https://github.com/gkouros/refnerf-pytorch.git
cd refnerf-pytorch

# Make a conda environment.
conda create --name refnerf-pl python=3.9
conda activate refnerf-pl
pip3 install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu116

# Prepare pip.
conda install pip
pip install --upgrade pip

# Install requirements.
pip install -r requirements.txt

# Manually install rmbrualla's `pycolmap` (don't use pip's! It's different).
git clone https://github.com/rmbrualla/pycolmap.git ./internal/pycolmap

Running

Example scripts for training, evaluating, and rendering can be found in scripts/. You'll need to change the paths to point to wherever the datasets are located. Gin configuration files for our model and some ablations can be found in configs/.

OOM errors

You may need to reduce the batch size (Config.batch_size) to avoid out-of-memory errors. If you do this but want to preserve quality, be sure to increase the number of training iterations and decrease the learning rate by whatever scale factor you decrease the batch size by.

Existing data loaders

To work from an example, you can see how this function is overloaded for the different dataloaders we have already implemented:

• Blender (*)
• LLFF (*)
• RFFR (*)
• DTU dataset
• Tanks and Temples

(*) represent the datasets that have been tested.

Hyper-Parameter Tuning

We found that in forward-facing settings, NeRF models may generate results with poor geometric properties (e.g. predicting all normals facing forward, no density in white background, etc.). Therefore, we add several geometric losses and consistency losses to constrain the model to satisfy geometric priors, references for these ideas can be found in the list of papers below.

	Ref-NeRF w/o Geometry Losses	Ref-NeRF w/ Geometry Losses
RGB	refnerf_color.mp4	refnerf_color.mp4
Accumulated Density	refnerf_acc.mp4	refnerf_acc.mp4
Normal	refnerf_normals_pred.mp4	refnerf_normals_pred.mp4
Median Distance	refnerf_distance_median.mp4	refnerf_distance_median.mp4
PSNR(↑) / SSIM(↑) / LPIPS(↓)	26.310 / 0.862 / 0.205	26.395 / 0.866 / 0.199

In our tests, improvements were not always guaranteed across different experimental settings with a given parameter setting. It still needs to be adjusted in different scenarios.

Awesome NeRF with Geometry Losses

• Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (CVPR 2022)
• Scalable Neural Indoor Scene Rendering (SIGGRAPH 2022)
• NeRFReN: Neural Radiance Fields with Reflections (CVPR 2022)

The following papers mainly discuss issues of few-shot NeRF (training NeRF with limited input images).

• Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation (CVPR 2022)
• InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering (CVPR 2022)
• GeoAug: Data Augmentation for Few-Shot NeRF with Geometry Constraints (ECCV 2022)
• GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency (ICML 2023)
• Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (CVPR 2022)
• Depth-supervised NeRF: Fewer Views and Faster Training for Free (CVPR 2022)
• RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (CVPR 2022)
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (ICCV 2021)
• DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models (CVPR 2023)
• Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs (ICCV 2023)
• SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (ICCV 2023)
• FlipNeRF: Flipped Reflection Rays for Few-shot Novel View Synthesis (ICCV 2023)
• ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields (SIGGRAPH 2023)
• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization (CVPR 2023)

Feel free to email me if you find any cool papers that should be on the list !!

References

@article{verbin2022refnerf,
    title={{Ref-NeRF}: Structured View-Dependent Appearance for
           Neural Radiance Fields},
    author={Dor Verbin and Peter Hedman and Ben Mildenhall and
            Todd Zickler and Jonathan T. Barron and Pratul P. Srinivasan},
    journal={CVPR},
    year={2022}
}

@misc{refnerf-pytorch,
      title={refnerf-pytorch: A port of Ref-NeRF from jax to pytorch},
      author={Georgios Kouros},
      year={2022},
      url={https://github.com/google-research/refnerf-pytorch},
}

@misc{multinerf2022,
      title={{MultiNeRF}: {A} {Code} {Release} for {Mip-NeRF} 360, {Ref-NeRF}, and {RawNeRF}},
      author={Ben Mildenhall and Dor Verbin and Pratul P. Srinivasan and Peter Hedman and Ricardo Martin-Brualla and Jonathan T. Barron},
      year={2022},
      url={https://github.com/google-research/multinerf},
}