UDC-NeRF: Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis

ICCV 2023

Yuxin Wang¹, Wayne Wu², Dan Xu^1
1HKUST, ²Shanghai AI Lab

   

Demo Videos

Object Manipulation compared with Object-NeRF.

toy_scan.mp4

Installation

git clone --recursive https://github.com/W-Ted/UDC-NeRF.git

conda create -n udcnerf python=3.6.13
conda activate udcnerf
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
pip install -r requirements.txt
pip install catalyst

Dataset

Please refer to Object-NeRF's data preparation guidance to prepare the dataset.

Please follow the LaMa's guidance to config the environment and download the pre-trained checkpoint. Then please refer to the following scripts to in-paint the background.

# step 1: prepare the images and corresponding masks. 
python preprocess/scripts/prepare_lamain_xxx.py  # Please modify the data path first.
# step 2: run LaMa to in-paint the background. 
bash preprocess/scripts/run_lama.sh
# step 3: 
python preprocess/scripts/rename_lamaout.py

Editable Novel View Synthesis

Please download our pre-trained checkpoints, and put the folder in udc-nerf/pretrained_ckpts. The following scripts can be used to generate demo videos in debug/rendered_view/render_xxx_edit/.

python scripts/edit_toydesk2.sh # for toydesk2, takes more than 1.5h. 
python scripts/edit_scannet0113_multi.sh # for scannet0113_multi, takes about 30min. 

Training

In our experiments, we used two scenes in ToyDesk Dataset and four scenes in ScanNet Dataset, i.e, 0024, 0038, 0113, 0192. The following scripts are two examples, and please refer to the training scripts in scripts/ for more details.

python scripts/train_toydesk2.sh # for toydesk2
python scripts/train_scannet0113_multi.sh # for scannet0113_multi

Evaluation

The following scripts are two examples, and please refer to the evaluation scripts in scripts/ for more details.

python scripts/test_toydesk2.sh # for toydesk2
python scripts/test_scannet0113_multi.sh # for scannet0113_multi

Acknowledgements

This project is built upon Object-NeRF. The in-painted images are obtained by LaMa. Kudos to these researchers.

Citation

@inproceedings{wang2023udcnerf,
     title={Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis},
     author={Wang, Yuxin and Wu, Wayne and Xu, Dan},
     booktitle={ICCV},
     year={2023}
     }