Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

Official PyTorch Implementation of paper "Vision Transformer for NeRF-Based View Synthesis from a Single Input Image", WACV 2023.

Kai-En Lin^1* Lin Yen-Chen² Wei-Sheng Lai³ Tsung-Yi Lin^4** Yi-Chang Shih³ Ravi Ramamoorthi¹

¹University of California, San Diego, ²Massachusetts Institute of Technology, ³Google, ⁴NVIDIA

^* Work done while interning at Google, ^** Work done while at Google.

Project Page | Paper | Pretrained models

Requirements

Install required packages

Make sure you have up-to-date NVIDIA drivers supporting CUDA 11.1 (10.2 could work but need to change cudatoolkit package accordingly)

Run

conda env create -f environment.yml
conda activate visionnerf

Usage

1. Clone the repository git clone https://github.com/ken2576/vision-nerf.git and download dataset from PixelNeRF.

2. Download pretrained model weights.

   Here is a list of the model weights:

   • nmr_500000.pth: Our pretrained weights for the category-agnostic experiment.
   • srn_cars_500000.pth: Our pretrained weights for the category-specific experiment on ShapeNet Cars.
   • srn_chairs_500000.pth: Our pretrained weights for the category-specific experiment on ShapeNet Chairs.

3. Install requirements conda env create -f environment.yml.

4. Setup configurations in configs.

5. (Optional) Run training script with python train.py --config [config_path].

6. Run inference script with our pretrained models:

python eval.py --config [path to config file] # For ShapeNet Cars/Chairs
python eval_nmr.py --config [path to config file] # For NMR
python gen_real.py --config [path to config file] # For real car data

Prepare real data

Our pretrained model works with real car images. You can prepare the data using the same process as PixelNeRF.

Then, run gen_real.py similar to the above example.

Acknowledgement

This code is based on DPT, IBRNet and PixelNeRF.

Citation

@inproceedings {lin2023visionnerf,
    booktitle = {WACV},
    title = {Vision Transformer for NeRF-Based View Synthesis from a Single Input Image},
    author = {Lin, Kai-En and Yen-Chen, Lin and Lai, Wei-Sheng and Lin, Tsung-Yi and Shih, Yi-Chang and Ramamoorthi, Ravi},
    year = {2023},
}