Vitruvio: Conditional Variational Autoencoder (CVAE) to Generate Building Meshes via Single Perspective Sketches

This method allows designers to automatically generate 3D representations in real-time based on their initial sketches and thus communicate effectively and intuitively to the client. Vitruvio adapts the Occupancy Network to perform single view reconstruction (SVR), a technique for creating 3D representations from a single image. Vitruvio achieves: (1) an 18% increase in the reconstruction accuracy and (2) a 26% reduction in the inference time compared to the Occupancy Network on one thousand buildings provided by the New York municipality. This research investigates the effect that the building orientation has on the reconstruction quality, discovering that Vitruvio can capture fine-grain details in complex buildings when their native orientation is preserved during training, as opposed to the SVR's standard practice that aligns every building to its canonical pose.

Contributions

• We introduced learning-based method for single perspective sketch-to-3D applications in the Architecture Engineering and Construction (AEC) industry.
• We adapted a previous state-of-the-art Conditional Variational Autoencoder (Occupancy Network) to scalable datasets.
• We showed qualitatively and quantitatively how the building orientation affects the reconstruction.

Orientation

	We need to consider the orientation of the building for this sketch to 3D method. Use this script to center and align the dataset, while tracking the orientation. Make sure to install all the packages in your environemnt and use python align.py --objs '<FOLDER PATH>'

Dataset

You can generate your own dataset taking the .obj files from RealCity3D and run the code in Occupancy Network to get the occupancy field. For any other inqueries please reach out to Alberto Tono using atono@stanford.edu

Main References and Material

This repository uses the code from the paper

• Occupancy Networks - Learning 3D Reconstruction in Function Space as a baseline.
• Paper Arxiv 2021 submission
• Dataset, Weights Request & Contribution Form [5GB]
• Dataset Split and Orientation Tracker File [26kb]

Acknowledgement

CIFE Stanford Seed Grant 2022/23, CIFE Stanford Seed Grant 2023/24, Stanford Human-Centered Artificial Intelligence - HAI, NVIDIA Omniverse Team.

At the beginning of a project, architects convey design ideas via quick 2D diagrams, front views, floor plans, and sketches. Consequently, many stakeholders have difficulty visualizing the 3D representation of the building mass, leading to varied interpretations thus inhibiting a shared understanding of the design. To alleviate the challenge, this paper proposes a deep learning-based method, Vitruvio, for creating a 3D model from a single perspective sketch

Cite

@misc{tono22vitruvio,
  doi = {10.48550/ARXIV.2210.13634},
  url = {https://arxiv.org/abs/2210.13634},
  author = {Tono, Alberto and Fischer, Martin},
  title = {Vitruvio: 3D Building Meshes via Single Perspective Sketches},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}