Yuxin Wang1,
Wayne Wu2,
Dan Xu1
1HKUST,
2Shanghai AI Lab
Object Manipulation compared with Object-NeRF.
toy_scan.mp4
git clone --recursive https://github.com/W-Ted/UDC-NeRF.git
conda create -n udcnerf python=3.6.13
conda activate udcnerf
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
pip install -r requirements.txt
pip install catalyst
Please refer to Object-NeRF's data preparation guidance to prepare the dataset.
Please follow the LaMa's guidance to config the environment and download the pre-trained checkpoint. Then please refer to the following scripts to in-paint the background.
# step 1: prepare the images and corresponding masks.
python preprocess/scripts/prepare_lamain_xxx.py # Please modify the data path first.
# step 2: run LaMa to in-paint the background.
bash preprocess/scripts/run_lama.sh
# step 3:
python preprocess/scripts/rename_lamaout.py
Please download our pre-trained checkpoints, and put the folder in udc-nerf/pretrained_ckpts
. The following scripts can be used to generate demo videos in debug/rendered_view/render_xxx_edit/
.
python scripts/edit_toydesk2.sh # for toydesk2, takes more than 1.5h.
python scripts/edit_scannet0113_multi.sh # for scannet0113_multi, takes about 30min.
In our experiments, we used two scenes in ToyDesk Dataset and four scenes in ScanNet Dataset, i.e, 0024, 0038, 0113, 0192. The following scripts are two examples, and please refer to the training scripts in scripts/
for more details.
python scripts/train_toydesk2.sh # for toydesk2
python scripts/train_scannet0113_multi.sh # for scannet0113_multi
The following scripts are two examples, and please refer to the evaluation scripts in scripts/
for more details.
python scripts/test_toydesk2.sh # for toydesk2
python scripts/test_scannet0113_multi.sh # for scannet0113_multi
This project is built upon Object-NeRF. The in-painted images are obtained by LaMa. Kudos to these researchers.
@inproceedings{wang2023udcnerf,
title={Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis},
author={Wang, Yuxin and Wu, Wayne and Xu, Dan},
booktitle={ICCV},
year={2023}
}