Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models

Project Page | Paper

Arxiv, 2024

Yuxuan Xue¹ , Xianghui Xie^{1, 2}, Riccardo Marin¹, Gerard Pons-Moll^{1, 2}

¹Real Virtual Human Group @ University of Tübingen & Tübingen AI Center
²Max Planck Institute for Informatics, Saarland Informatics Campus

News 🚩

• [2024/06/14] Human 3Diffusion paper is available on ArXiv.
• [2024/06/14] Inference code and model weights is scheduled to be released after CVPR 2024.

Key Insight 🙌

• 2D foundation models are powerful but output lacks 3D consistency!
• 3D generative models can reconstruct 3D representation but is poor in generalization!
• How to combine 2D foundation models with 3D generative models?:
  • they are both diffusion-based generative models => Can be synchronized at each diffusion step
  • 2D foundation model helps 3D generation => provides strong prior informations about 3D shape
  • 3D representation guides 2D diffusion sampling => use rendered output from 3D reconstruction for reverse sampling, where 3D consistency is guaranteed

Citation ✍️

@inproceedings{xue2023human3diffusion,
  title     = {{Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models}},
  author    = {Xue, Yuxuan and Xie, Xianghui and Marin, Riccardo and Pons-Moll, Gerard.},
  journal   = {Arxiv},
  year      = {2024},
}