SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model

Yukai Shi^1,3, Weiyu Li^2,4, Zihao Wang⁴, Hongyang Li³, Xingyu Chen³, Ping Tan^2,4, Lei Zhang³

¹ Tsinghua University    ² HKUST    ³ IDEA Research    ⁴ LightIllusions

Scene Image Normal Map 3D Scene	Scene Image Normal Map 3D Scene
Scene Image Normal Map 3D Scene	Scene Image Normal Map 3D Scene
Scene Image Normal Map 3D Scene	Scene Image Normal Map 3D Scene

Abstract

We propose a decoupled 3D scene generation framework called SceneMaker in this work. Due to the lack of sufficient open-set de-occlusion and pose estimation priors, existing methods struggle to simultaneously produce high-quality geometry and accurate poses under severe occlusion and open-set settings. To address these issues, we first decouple the de-occlusion model from 3D object generation, and enhance it by leveraging image datasets and collected de-occlusion datasets for much more diverse open-set occlusion patterns. Then, we propose a unified pose estimation model that integrates global and local mechanisms for both self-attention and cross-attention to improve accuracy. Besides, we construct an open-set 3D scene dataset to further extend the generalization of the pose estimation model. Comprehensive experiments demonstrate the superiority of our decoupled framework on both indoor and open-set scenes. Our codes and datasets will be released.

Framework

Our framework consists of three main components:

1. Scene Perception: Understanding the input scene structure
2. 3D Object Generation under Occlusion: Decoupled de-occlusion model for robust object generation
3. Pose Estimation: Unified pose estimation model with global and local attention mechanisms

We decouple the de-occlusion model from 3D object generation. We construct a unified pose estimation model that incorporates both global and local attention mechanisms.

Open Source Progress

• 🔄 Dataset: Uploading
• ⏳ Inference Code: Coming soon
• ⏳ Training Code: Coming soon

Citation

If you find our work useful in your research, please consider citing:

@article{
}

Acknowledgement

We would like to thank the authors of the following projects for their excellent work and open-source contributions:

• MoGe - Monocular depth estimation
• SAM - Segment Anything Model for image segmentation
• DINO-X - Grounding segementation
• CraftsMan - 3D object generation
• Step1x-3D - 3D object generation
• Hunyuan3D - 3D object generation
• MIDI3D - Multi-instance 3D scene generation
• InstPIFu - Indoor 3D scene generation

Their contributions have been invaluable to the development of SceneMaker.

License

See LICENSE file for details.