Skip to content

Latest commit

 

History

History
116 lines (89 loc) · 6.89 KB

README.md

File metadata and controls

116 lines (89 loc) · 6.89 KB
Raw

refnerf-pytorch-lightning

This is an implementation of Ref-NeRF, which is extended from refnerf-pytorch based on the original jax code released by Google.

Setup

# Clone the repo.
git clone https://github.com/gkouros/refnerf-pytorch.git
cd refnerf-pytorch

# Make a conda environment.
conda create --name refnerf-pl python=3.9
conda activate refnerf-pl
pip3 install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu116

# Prepare pip.
conda install pip
pip install --upgrade pip

# Install requirements.
pip install -r requirements.txt

# Manually install rmbrualla's `pycolmap` (don't use pip's! It's different).
git clone https://github.com/rmbrualla/pycolmap.git ./internal/pycolmap

Running

Example scripts for training, evaluating, and rendering can be found in scripts/. You'll need to change the paths to point to wherever the datasets are located. Gin configuration files for our model and some ablations can be found in configs/.

OOM errors

You may need to reduce the batch size (Config.batch_size) to avoid out-of-memory errors. If you do this but want to preserve quality, be sure to increase the number of training iterations and decrease the learning rate by whatever scale factor you decrease the batch size by.

Existing data loaders

To work from an example, you can see how this function is overloaded for the different dataloaders we have already implemented:

  • Blender (*)
  • LLFF (*)
  • RFFR (*)
  • DTU dataset
  • Tanks and Temples

(*) represent the datasets that have been tested.

Hyper-Parameter Tuning

We found that in forward-facing settings, NeRF models may generate results with poor geometric properties (e.g. predicting all normals facing forward, no density in white background, etc.). Therefore, we add several geometric losses and consistency losses to constrain the model to satisfy geometric priors, references for these ideas can be found in the list of papers below.

Ref-NeRF w/o Geometry Losses Ref-NeRF w/ Geometry Losses
RGB
refnerf_color.mp4
refnerf_color.mp4
Accumulated Density
refnerf_acc.mp4
refnerf_acc.mp4
Normal
refnerf_normals_pred.mp4
refnerf_normals_pred.mp4
Median Distance
refnerf_distance_median.mp4
refnerf_distance_median.mp4
PSNR(↑) / SSIM(↑) / LPIPS(↓) 26.310 / 0.862 / 0.205 26.395 / 0.866 / 0.199

In our tests, improvements were not always guaranteed across different experimental settings with a given parameter setting. It still needs to be adjusted in different scenarios.

Awesome NeRF with Geometry Losses

The following papers mainly discuss issues of few-shot NeRF (training NeRF with limited input images).

Feel free to email me if you find any cool papers that should be on the list !!

References

@article{verbin2022refnerf,
    title={{Ref-NeRF}: Structured View-Dependent Appearance for
           Neural Radiance Fields},
    author={Dor Verbin and Peter Hedman and Ben Mildenhall and
            Todd Zickler and Jonathan T. Barron and Pratul P. Srinivasan},
    journal={CVPR},
    year={2022}
}

@misc{refnerf-pytorch,
      title={refnerf-pytorch: A port of Ref-NeRF from jax to pytorch},
      author={Georgios Kouros},
      year={2022},
      url={https://github.com/google-research/refnerf-pytorch},
}

@misc{multinerf2022,
      title={{MultiNeRF}: {A} {Code} {Release} for {Mip-NeRF} 360, {Ref-NeRF}, and {RawNeRF}},
      author={Ben Mildenhall and Dor Verbin and Pratul P. Srinivasan and Peter Hedman and Ricardo Martin-Brualla and Jonathan T. Barron},
      year={2022},
      url={https://github.com/google-research/multinerf},
}