Human-Aware Object Placement for Visual Environment Reconstruction. (CVPR2022)

[Project Page] [Paper] [MPI Project Page] [Youtube Video]

3D Scene and Humans Reconstruction Results from a single RGB video

What Can You Learn from MOVER?

• 3D Scene Initialization with water-tight mesh (benefit from Total3DUnderstand and OccupancyNet).
• Single Video Batch-wise SMPLify-X for single person.
• Ground-Plane & Camera Orientation Optimization with human contacted feet.
• Three HSIs Constraints: the ordering depth, collision and contact.

Installation

Please follow the Installation Instruction to setup all the required packages.

Data

Please register SMPL-X at first, and then download smpl-x_model.tar.gz from our webpage, put it under ${MOVER_REPORSITORY}/data/.

We provide demo sequences and MOVER reconstructed humans and 3D scenes of PROX qualitative and quantitative in our webpage.

See more details in data document

Get Started

We provide the core part of MOVER, use three different kinds of HSI constraints (depth, collision, and contact) to help understand the 3D scene.

Scene optimization with 2D cues and HSIs:

cd ./demo
bash run.sh

We also provide the visualization for the final reconstructed 3D scenes and 3D humans.

Visualize reconstructed 3D humans and 3D scene results:

cd ./demo
bash run_rendering.sh

Scene Initialization

See more details in scene initialization document

Human Pose ans Shape (HPS) Initialization

See more details in HPS initialization document

Citation

@inproceedings{yi2022mover,
title = {Human-Aware Object Placement for Visual Environment Reconstruction},
author = {Yi, Hongwei and Huang, Chun-Hao P. and Tzionas, Dimitrios and Kocabas, Muhammed and 
  Hassan, Mohamed and Tang, Siyu and Thies, Justus and Black, Michael J.},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2022},
month_numeric = {6}}

Acknowledgments

We thank Yixin Chen, Yuliang Xiu for the insightful discussions, Yao Feng, Partha Ghosh and
Maria Paola Forte for proof-reading, and Benjamin Pellkofer for IT support. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tubingen AI Center, FKZ: 01IS18039B.

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Contact

For more questions, please contact mover@tue.mpg.de

For commercial licensing, please contact ps-licensing@tue.mpg.de