Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

transfer_city.mp4

Introduction

The repository contains the official implementation of source code and pre-trained models of our paper:"Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior". It is a novel pipeline for large-scale 3D urban scene generation!

Updates

• 2024.04.26: We release samples of layout in dataset/data and pretrained ControlNet weights!
• 2024.04.11: The 🔥🔥🔥 pre-print 🔥🔥🔥 is released! Refer to it for more details!
• 2024.04.10: The project page is created. Check it out for an overview of our work!

Environments

1. Main requirements:

• PyTorch (tested on torch-2.0)
• pytorch3d
• diffusers

2. Download the pretrained model weights of Stable Diffusion
3. Download the pretrained model weights of CLIP
4. Download the pretrained model weights of ControlNet
5. Other requirements are provided in requirements.txt

Generate a Scene

Coarse stage

Please refer to train.sh.

Refinement stage

Please refer to refine.sh.

Render video

Please refer to render.sh.

To-Do List

• Release 3D layout data
• Technical Report
• Project page

Related Work

• (ICCV 2023) Urban Radiance Field Representation with Deformable Neural Mesh Primitives, Fan Lu et al. [Paper], [Project Page]

Citation

If you find this project useful for your work, please consider citing:

@article{lu2024urban,
  title={Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior},
  author={Lu, Fan and Lin, Kwan-Yee and Xu, Yan and Li, Hongsheng and Chen, Guang and Jiang, Changjun},
  journal={arXiv preprint arXiv:2404.06780},
  year={2024}
}