This repository contains the code for the research project Stable NeRF. The project aims use 2D priors from Stable Diffusion and the learned 3D forward map from NeRF to perform generalizable novel view synthesis.
Contributors: Emre Arslan (earslan25), Chia-Hong Hsu (swimmincatt35), Daniel Cho (hypochoco)
Our initial proposal and current report with results can be found in the reports folder.
- Currently, directly creating from the environment.yml file is not working or is too slow. Instead, create a new environment with python=3.8 and install the packages manually.
- To install the packages manually, run the following commands:
- When using oscar, run these lines first.
module purge
module load miniconda3/23.11.0s
source /oscar/runtime/software/external/miniconda3/23.11.0/etc/profile.d/conda.sh
module load cuda/11.8.0-lpttyok- Then, run the following lines.
conda create -n stable_nerf python=3.8
conda activate stable_nerf
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
pip install submodules/raymarching- Download datasets. They can also be downloaded by running the Jupyter notebooks in the datasets folder.
curl https://cseweb.ucsd.edu//~viscomp/projects/LF/papers/ECCV20/nerf/tiny_nerf_data.npz -o datasets/nerf/tiny_nerf_data.npzInitially, we used a synthetic dataset to assess our method's performance. Given that baseline, we will attempt to extend our approach to generalize to real world views. Our method can benefit from most multiview datasets as we only need 2 views per scene.
- Implement argument parser to handle different configurations
- Integrate per-pixel features from encoded images into NeRF
- Distributed training (current Hugging Face accelerate implementation is not compatible with the code format and tiny-cuda-nn)
We would like to thank the following repositories for their code and inspiration: