Official code repository for the paper "Unite the People – Closing the Loop Between 3D and 2D Human Representations".
Switch branches/tags
Nothing to show
Clone or download
Latest commit 9418150 Jul 21, 2018
Permalink
Failed to load latest commit information.
3dfit Fixed segmented rendering. Jul 21, 2018
direct3d Added 3D model. Oct 12, 2017
models Repaired segmentation. Oct 27, 2017
patches Setup. Jul 5, 2017
pose Added 3D model. Oct 12, 2017
segmentation Repaired segmentation. Oct 27, 2017
tests Setup. Jul 5, 2017
up_tools Update mesh.py Jul 11, 2018
.gitignore Fixed segmented rendering. Jul 21, 2018
LICENSE.txt Updated license. Sep 5, 2017
README.md Added WebMonks Docker to README. Feb 19, 2018
config.py setup fix Jul 20, 2017
requirements.txt Added cython requirement. Jul 20, 2017
setup.py setup fix Jul 20, 2017

README.md

Unite the People code repository

Requirements:

  • OpenCV (on Ubuntu, e.g., install libopencv-dev and python-opencv).
  • SMPL (download at http://smpl.is.tue.mpg.de/downloads) and unzip to a place of your choice.
  • OpenDR (just run pip install opendr, unfortunately can't be done automatically with the setuptools requirements.
  • If you want to train a segmentation model, Deeplab V2 (https://bitbucket.org/aquariusjay/deeplab-public-ver2) with a minimal patch applied that can be found in the subdirectory patches, to enable on the fly mirroring of the segmented images. Since I didn't use the MATLAB interface and did not care about fixing related errors, I just deleted src/caffe/layers/mat_{read,write}_layer.cpp as well as src/caffe/util/matio_io.cpp and built with -DWITH_matlab=Off.
  • If you want to train a pose model, the Deepercut caffe (https://github.com/eldar/deepcut-cnn).
  • If you want to get deepercut-cnn predictions, download the deepercut .caffemodel file and place it in models/pose/deepercut.caffemodel.
  • Edit the file config.py to set up the paths.

The rest of the requirements is then automatically installed when running:

python setup.py develop

Folder structure

For each of the tasks we described, there is one subfolder with the related executables. All files that are being used for training or testing models are executable and provide a full synposis when run with the --help option. In the respective tools subfolder for each task, there is a create_dataset.py script to summarize the data in the proper formats. This must be usually run before the training script. The models folder contains pretrained models and infos, patches a patch for deeplab caffe, tests some Python tests and up_tools some Python tools that are shared between modalities.

There is a Docker image available that has been created by TheWebMonks here (not affiliated with the authors): https://github.com/TheWebMonks/demo-2d3d .

Bodyfit

The adjusted SMPLify code to fit bodies to 91 keypoints is located in the folder 3dfit. It can be used for 14 or 91 keypoints. Use the script 3dfit/render.py to render a fitted body.

Direct 3D fitting using regression forests

The relevant files are in the folder direct3d. Run run_partforest_training.sh to train all regressors. After that, you can use bodyfit.py to get predictions from estimated keypoints of the 91 keypoint pose predictor.

91 keypoint pose prediction

The pose folder containes infrastructure for 91 keypoint pose prediction. Use the script pose/tools/create_dataset.py with a dataset name of your choice and a target person size of 500 pixels to create the pose data from UP-3D, alternatively download it from our website.

Configure a model by creating the model configuration folder pose/training/config/modelname by cloning the pose model. Then you can run run.sh {train,test,evaluate,trfull,tefull,evfull} modelname to run training, testing or evaluation on either the reduced training set with the held-out validation set as test data or the full training set and real test data. We initialized our training from the original Resnet models (https://github.com/KaimingHe/deep-residual-networks). You can do so by downloading the model and saving it as pose/training/config/modelname/init.caffemodel.

The pose.py script will produce a pose prediction for an image. It assumes that a model with name pose has been trained (or downloaded). We normalize the training images w.r.t. person size, that's why the model works best for images with a rough person height of 500 pixels. Multiple people are not taken into account; for every joint the arg max position is used over the full image.

31 part segmentation

The folder setup is just as for the keypoint estimation: use segmentation/tools/create_dataset.py to create a segmentation dataset from the UP-3D data or download it (again, we used target person size 500). Then use run.sh {train,test,evaluate,trfull,tefull,evfull} modelname as described above to create your models. The segmentation.py script can be used to get segmentation results for the model named segmentation from and image. We initialized our models from the Deeplab trained models available here. Move the model file to segmentation/training/modelname/init.caffemodel.

Website, citation, license

You can find more information on the website. If you use this code for your research, please consider citing us:

@inproceedings{Lassner:UP:2017,
  title = {Unite the People: Closing the Loop Between 3D and 2D Human Representations},
  author = {Lassner, Christoph and Romero, Javier and Kiefel, Martin and Bogo, Federica and Black, Michael J. and Gehler, Peter V.},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  month = jul,
  year = {2017},
  url = {http://up.is.tuebingen.mpg.de},
  month_numeric = {7}
}

License: Creative Commons Non-Commercial 4.0.

The code for 3D fitting is based on the SMPLify code. Parts of the files in the folder up_tools (capsule_ch.py, capsule_man.py, max_mixture_prior.py, robustifiers.py, sphere_collisions.py) as well as the model models/3D/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl fall under the SMPLify license conditions.