The official implementation of PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes.
Yu-Ying Yeh, Zhengqin Li, Yannick Hold-Geoffroy, Rui Zhu, Zexiang Xu, Miloš Hašan, Kalyan Sunkavalli, Manmohan Chandraker
IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022
- This repo is tested on RTX2080Ti GPU. There are reports showing that it might not work on newer GPUs.
- Clone this repository including submodules
git clone --recursive https://github.com/ViLab-UCSD/PhotoScene.git
cd PhotoScene
-
Install OptiX 5.1.1 under
third_party/OptiX
. We assume OptiX install directory isthird_party/OptiX/NVIDIA-OptiX-SDK-5.1.1-linux64
. Note that the version should be 5.1.1 upon which our renderer is built. -
Download Total3D, MaskFormer, InvRenderNet pretrained model by running the following script.
bash scripts/download_model.sh
- Set up Substance Designer
- Install Substance Designer with Linux (.rpm) version 2021.1.2
Get free license here if you're a student or teacher. Download Substance Designer version 11.1.2 (437M) (LICENSES
->Substance Designer All builds
->Linux (.rpm) version 2021.1.2
) and run the following to install. The default installation path will be /opt/Allegorithmic/Substance_Designer
.
sudo apt-get update
sudo apt-get install alien
sudo alien -i Substance_Designer-11.1.2-4593-linux-x64-standard.rpm
- Note: It is a requirement for DiffMat
- Install Docker
Our framework consists of multiple dependencies. We strongly recommend using our provided docker image to run this repository. The provided script assumes running docker in rootless mode with a single GPU. Might need to modify /etc/nvidia-container-runtime/config.toml
for the entry under [nvidia-container-cli] to no-cgroups = true
to disable the use of cgroups by the NVIDIA container runtime. After installation, run the following to pull our image and launch the environment.
docker pull yyeh/photoscene:v1
bash scripts/run_docker_interactive.sh
- Build OptixRenderer
Please make sure OptiX 5.1.1 is installed under third_party/OptiX
before running the following script. Note that from this step all the commands must be run inside the docker environment.
bash scripts/build_renderer.sh
Now we can run the full PhotoScene framework on a scene specified by $yamlFile
.
bash run_photoscene.sh $yamlFile
For example, $yamlFile
is set default as configs/total3d/Total_246.yaml
.
Alternatively, you can run each single step sequentially:
# Initialization and Alignment
python3 photoscene/preprocess.py --config $yamlFile
# Graph Selection
python3 photoscene/selectGraphFromCls.py --config $yamlFile
# First Round Material Optimization
python3 photoscene/optimizeMaterial.py --config $yamlFile --mode first
# Lighting Optimization
python3 photoscene/optimizeLight.py --config $yamlFile
# Second Round Material Optimization
python3 photoscene/optimizeMaterial.py --config $yamlFile --mode second
# Render Final PhotoScene Result
python3 photoscene/renderPhotoScene.py --config $yamlFile
You can first use provided Total3D examples to run the entire framework. If you want to try more scenes, please download Total3D preprocessed data by running the following script or following Total3D to download preprocessed SUN RGBD data (12G) to third_party/Total3D/data/sunrgbd/sunrgbd_train_test_data
and copy the scenes (<scene_id>.pkl) to data/total3d/inputs
.
bash scripts/download_total3d_data.sh
Please cite our paper if you find that our method is helpful!
@InProceedings{Yeh_2022_CVPR,
author = {Yeh, Yu-Ying and Li, Zhengqin and Hold-Geoffroy, Yannick and Zhu, Rui and Xu, Zexiang and Ha\v{s}an, Milo\v{s} and Sunkavalli, Kalyan and Chandraker, Manmohan},
title = {PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {18562-18571}
}