Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing

Project Page | Video | Paper

Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing

[Junyi Zeng, Chong Bao^Co-Authors], Rui Chen, Zilong Dong, Guofeng Zhang, Hujun Bao, Zhaopeng Cui.

ACM Multimedia 2023

Hardware

• OS: Ubuntu 18.04
• NVIDIA GPU with CUDA>=11.1 (tested with 1 RTX3090)

Installation

We have tested the code on Python 3.8.0 and PyTorch 1.8.1, while a newer version of pytorch should also work.

The steps of installation are as follows:

• Clone this repo by git clone --recursive https://github.com/zju3dv/Mirror-NeRF
• Python>=3.8 (installation via Anaconda is recommended, use conda create -n mirror_nerf python=3.8 to create a conda environment and activate it by conda activate mirror_nerf)
• Python libraries
  • Install requirements by pip install -r requirements.txt
  • Install PyTorch 1.8.1 by pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
• [Optional] If you want to use tiny-cuda-nn for acceleration, install tiny-cuda-nn PyTorch extension by pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Data

We support synthetic datasets (datasets/blender.py) and real datasets (datasets/real_arkit.py).

For customized datasets with camera poses reconstructed by COLMAP, you can refer to datasets/real_colmap.py.

Data download

Download our captured synthetic and real datasets from here.

Download our pretrained models on the synthetic and real datasets from here.

Notes: For pretrained models with "tcnn" in the filename, select MODEL_TYPE="nerf_tcnn" in run.sh.

For pretrained models of D-NeRF, please refer to D-NeRF repository.

Running

We integrate training, testing and applications in one script run.sh,

bash run.sh {MODE} {GPU_ID}

MODE:

1 for evaluation (Novel View Synthesis),

2 for extracting mesh,

3 for placing new mirrors (application),

4 for reflecting newly placed objects (application),

5 or 52 for controlling mirror roughness (application),

6 for reflection substitution (application),

other numbers like 0 for training.

Please configure the settings in run.sh before running.

For example, choose the DATASET and MODEL_TYPE.

Notes: For acceleration, use MODEL_TYPE="nerf_tcnn".

For scenes with accurate camera poses (like synthetic scenes), MODEL_TYPE="nerf" is recommended.

For some real captures with inaccurate camera poses, MODEL_TYPE="nerf_tcnn" is recommended.

For evaluation and applications, specify the LOG (necessary), SUBSTITUTION_LOG (for reflection substitution) and OBJ_CKPT_PATH (for reflecting newly placed objects).

For more configurations, see opt.py.

The results of training will be automatically stored in the log/ directory.

The results of evaluation and application will be automatically stored in the results/ directory.

Citing

@inproceedings{zeng2023mirror-nerf,
    title={Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing},
    author={Zeng, Junyi and Bao, Chong and Chen, Rui and Dong, Zilong and Zhang, Guofeng and Bao, Hujun and Cui, Zhaopeng},
    booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
    pages={4606--4615},
    year={2023}
}

Acknowledgement

This repository is developed upon nerf_pl.

And we use parts of the code of D-NeRF for object models used in the application of reflecting newly placed objects, and parts of the code of torch-ngp for using tiny-cuda-nn if acceleration is needed.

Thanks for these great projects!