Official repository of the paper:
Plan2Scene: Converting floorplans to 3D scenes
- We use a conda environment initialized as described here.
- Setup the
command line libraryof Embark Studios texture-synthesis project.
- You can download a pre-built binary available here. Alternatively, you may build from the source.
- Download the seam mask available here.
./conf/plan2scene/seam_correct-example.jsonto 'seam_correct.json' and update the paths to the texture synthesis command line library binary, and the seam mask.
Use 'code/src' as the source root when running python scripts.
[Optional] We have provided 3D scenes pre-populated with CAD models of objects. If you wish to re-populate these scenes using the Object Placement approach we use, follow the instructions here.
To replicate our results, you should use the pre-extracted crops we provide. These crops are provided with the Rent3D++ dataset and are copied to the
[Optional] If you wish to extract new crops instead of using these provided crops, following these instructions.
Select ground truth reference crops and populate photo room assignment lists.
# Select ground truth reference crops. python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/train ./data/input/photo_assignments/train train python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/val ./data/input/photo_assignments/val val python code/scripts/plan2scene/preprocessing/generate_reference_crops.py ./data/processed/gt_reference/test ./data/input/photo_assignments/test test # We evaluate Plan2Scene by simulating photo un-observations. # Generate photoroom.csv files considering different photo un-observation ratios. python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/train ./data/input/photo_assignments/train ./data/input/unobserved_photos.json train python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/val ./data/input/photo_assignments/val ./data/input/unobserved_photos.json val python code/scripts/plan2scene/preprocessing/generate_unobserved_photo_assignments.py ./data/processed/photo_assignments/test ./data/input/photo_assignments/test ./data/input/unobserved_photos.json test
[Optional] Stationary Textures Dataset - We use one of the following datasets to train the texture synthesis model. Not required if you are using pre-trained models.
[Optional] Substance Mapped Textures dataset. Only used by the retrieve baseline.
Pretrained models are available here.
Inference on Rent3D++ dataset
Download and pre-process the Rent3D++ dataset as described in the data section.
Setup a pretrained model or train a new Plan2Scene network.
Synthesize textures for observed surfaces using the VGG textureness score.
# For test data without simulating photo unobservations. (drop = 0.0) python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0 python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.0 ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0 # Results are stored at ./data/processed/vgg_crop_select/test/drop_0.0
Propagate textures to unobserved surfaces using our texture propagation network.
python code/scripts/plan2scene/texture_prop/gnn_texture_prop.py ./data/processed/gnn_prop/test/drop_0.0 ./data/processed/vgg_crop_select/test/drop_0.0 test GNN_PROP_CONF_PATH GNN_PROP_CHECKPOINT_PATH --keep-existing-predictions --drop 0.0
To preview results, follow the instructions below.
- Complete inference steps.
- Correct seams of predicted textures and make them tileable.
# For test data without simulating photo unobservations. python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops ./data/processed/gnn_prop/test/drop_0.0/texture_crops test --drop 0.0
- Generate .scene.json files with embedded textures using embed_textures.py.
A scene.json file describes the 3D geometry of a house.
It can be previewed via a browser using the 'scene-viewer' of SmartScenesToolkit (You will have to clone and build the SmartScenesToolkit).
# For test data without simulating photo unobservations. python code/scripts/plan2scene/postprocessing/embed_textures.py ./data/processed/gnn_prop/test/drop_0.0/archs ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops test --drop 0.0 # scene.json files are created in the ./data/processed/gnn_prop/test/drop_0.0/archs directory.
- Render .scene.json files as .pngs using render_house_jsons.py.
- Download and build the SmartScenesToolkit.
./conf/render.jsonand update its fields to point to scene-toolkit.
- Run the following command to generate previews.
CUDA_VISIBLE_DEVICES=0 python code/scripts/plan2scene/render_house_jsons.py ./data/processed/gnn_prop/test/drop_0.0/archs --scene-json # A .png file is created for each .scene.json file in the ./data/processed/gnn_prop/test/drop_0.0/archs directory.
- Generate qualitative result pages with previews using preview_houses.py.
python code/scripts/plan2scene/preview_houses.py ./data/processed/gnn_prop/test/drop_0.0/previews ./data/processed/gnn_prop/test/drop_0.0/archs ./data/input/photos test --textures-path ./data/processed/gnn_prop/test/drop_0.0/tileable_texture_crops 0.0 # Open ./data/processed/gnn_prop/test/drop_0.0/previews/preview.html
Test on Rent3D++ dataset
[Optional] Download a pre-trained model or train the substance classifier used by the Subs metric. Training instructions are available here. Pre-trained weights are available here. Skip this step to omit the Subs metric.
Generate overall evaluation report at 60% photo unobservations. We used this setting in paper evaluations.
# Synthesize textures for observed surfaces using the VGG textureness score. # For the case: 60% (i.e. 0.6) of the photos unobserved. python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.6 test --drop 0.6 python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.6 ./data/processed/texture_gen/test/drop_0.6 test --drop 0.6 # Propagate textures to un-observed surfaces using our GNN. # For the case: 60% (i.e. 0.6) of the photos unobserved. python code/scripts/plan2scene/texture_prop/gnn_texture_prop.py ./data/processed/gnn_prop/test/drop_0.6 ./data/processed/vgg_crop_select/test/drop_0.6 test GNN_PROP_CONF_PATH GNN_PROP_CHECKPOINT_PATH --keep-existing-predictions --drop 0.6 # Correct seams of texture crops and make them tileable. # For test data where 60% of photos are unobserved. python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gnn_prop/test/drop_0.6/texture_crops test --drop 0.6 # Generate overall results at 60% simulated photo unobservations. python code/scripts/plan2scene/test.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test
Generate evaluation report for observed surfaces. No simulated unobservation of photos. We used this setting in paper evaluations.
# Run inference on using drop=0.0. python code/scripts/plan2scene/preprocessing/fill_room_embeddings.py ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0 python code/scripts/plan2scene/crop_select/vgg_crop_selector.py ./data/processed/vgg_crop_select/test/drop_0.0 ./data/processed/texture_gen/test/drop_0.0 test --drop 0.0 # Correct seams of texture crops and make them tileable by running seam_correct_textures.py. python code/scripts/plan2scene/postprocessing/seam_correct_textures.py ./data/processed/vgg_crop_select/test/drop_0.0/tileable_texture_crops ./data/processed/vgg_crop_select/test/drop_0.0/texture_crops test --drop 0.0 # Generate evaluation results for observed surfaces. python code/scripts/plan2scene/test.py ./data/processed/vgg_crop_select/test/drop_0.0/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test
Generate evaluation report for unobserved surfaces at 60% photo unobservations. We used this setting in the paper evaluations.
# It is assumed that the user has already generated the overall report at 0.6 drop fraction. # Generate results on unobserved surfaces at 60% simulated photo unobservations. python code/scripts/plan2scene/test.py ./data/processed/gnn_prop/test/drop_0.6/tileable_texture_crops ./data/processed/gt_reference/test/texture_crops test --exclude-prior-predictions ./data/processed/vgg_crop_select/test/drop_0.6/texture_crops
Generate evaluation report on FID metric as described here.
Inference on custom data
If you have floorplan vectors in another format, you can convert them to the raster-to-vector annotation format. Then, follow the same instructions as before to create textured 3D meshes of houses. The R2V annotation format is explained with examples in the data section of the raster-to-vector repository.
Training a new Plan2Scene network
Plan2Scene consists of two trainable components, 1) the texture synthesis stage and 2) the texture propagation stage. Each stage is trained separately. The training procedure is as follows.
- Train the texture synthesis stage as described here.
- Train the texture propagation stage as described here.
The baseline models are available here.