Identity-Aware Hand Mesh Estimation and Personalization from RGB Images

[ECCV2022] "Identity-Aware Hand Mesh Estimation and Personalization from RGB Images".

Introduction

This repo is the PyTorch implementation of ECCV2022 paper "Identity-Aware Hand Mesh Estimation and Personalization from RGB Images". You can find this paper from paper link.

Install

• Environment for IDentity-Aware Hand Mesh Estimation (IdHandMesh)

  conda create -n IdHandMesh python=3.8   
  conda activate IdHandMesh
  

• Please follow official suggestions to install pytorch and torchvision. We use pytorch=1.7.1, torchvision=0.8.2

• Requirements

  Please see environment.yaml.

• Install MANO from MANO.

• Note that torch_sparse and MPI-IS Mesh are required for the CMR model (one of the methods our method is compared to).

  a) If you have difficulty in installing torch_sparse etc., please use whl file from here.

  b) MPI-IS Mesh: We suggest to install this library from the source.

  More details can be found at CMR.

Dataset

Dex_YCB

• Please download DEX-YCB dataset from this link, and create a soft link in data, i.e., data/dex_ycb. Inside the folder 'datasets', run the following scripts to process the data.

  python split_annotations_with_cropping.py
  python get_rest_hand_mesh.py
  

• After processing the data, you should have the following directory struction.

  ${ROOT} 
  |-- conv
  |   |-- ...
  |-- data
  |   |-- dex_ycb
  |   |   |-- 20200709-subject-01
  |   |   |-- 20200813-subject-02
  |   |   |-- ...
  |   |   |-- 20201022-subject-10
  |   |   |-- 20201022-subject-10
  |   |   |-- 20201022-subject-10
  |   |   |-- bop
  |   |   |-- calibration
  |   |   |-- cropped_hand_size_2         # generated by our .py file
  |   |   |-- models
  |   |   |-- split_annotations           # generated by our .py file
  |-- datasets
  |   |-- dex_ycb
  |   |   |-- ...
  |-- options
  |-- out
  |-- scripts
  |-- src
  |-- template
  |   |-- dex_ycb_j_regressor.npy
  |   |-- MANO_RIGHT.pkl
  |   |-- template.ply 
  |   |-- transform.pkl
  |-- utils
  |   |-- ...
  |-- ....py
  

Reproducing the baseline and our method.

• Training

  a) Train the baseline model.

  ./scripts/train_dex_ycb_mano_based_baseline.sh
  

  b) Train the our model with ground truth hand shape.

  ./scripts/train_dex_ycb_mano_based_our_model_with_gt_shape.sh
  

  c) Train baseline with confidence branch. (only train the confidence branch, other parts are frozen).

  ./scripts/train_dex_ycb_mano_based_conf_branch.sh
  

• Run hand shape calibration.

  a) Get results from the baseline model

  python mis_dex_ycb_get_predictions_baseline_with_conf.py
  

  b) Perform calibration

  python calibrate_from_shape_params.py
  

• Evaluate the performance without optimizatin module.

  a) Baseline performance

  ./scripts/eval_dex_ycb_mano_based_baseline.sh
  

  b) Our model when fed with groundtruth hand shape

  ./scripts/eval_dex_ycb_ours_gt_hand_shape.sh
  

  c) Our model when fed with calibrated hand shape

  ./scripts/eval_dex_ycb_ours_calibrated_hand_shape.sh
  

• Optimizatin module during inference.

  Get 2d predictions.

  python mis_dex_ycb_get_predictions_2d.py
  

  Run optimization and evaluate at the same time.

  a) Baseline performance

  python optimization_dex_ycb_baseline.py
  

  b) Our model when fed with groundtruth hand shape

  python optimization_dex_ycb_ours_with_gt_hand_shape.py
  

  c) Our model when fed with calibrated hand shape

  python optimization_dex_ycb_ours_with_calibrated_hand_shape.py
  

Reproducing the methods being compared to.

• CMR

  ./scripts/train_dex_ycb_cmr.sh
  ./scripts/eval_dex_ycb_cmr.sh
  

• Boukhayma’s model

  ./scripts/train_dex_ycb_boukhayma
  ./scripts/eval_dex_ycb_boukhayma.sh
  

• Metro

  Install Metro and then use the following script to generate the tsv file for DEX_YCB.

  metro_utils/tsv_demo_dex_ycb.py
  

Reference

@inproceedings{kong2022identity,
title={Identity-Aware Hand Mesh Estimation and Personalization from RGB Images},
author={Kong, Deying and Zhang, Linguang and Chen, Liangjian and Ma, Haoyu and Yan, Xiangyi and Sun, Shanlin and Liu, Xingwei and Han, Kun and Xie, Xiaohui},
booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part V},
pages={536--553},
year={2022}
}

Acknowledgement

Our implementation is developed with the help of the following open sourced projects:

• spiralnet_plus.
• CMR
• Boukhayam's MANO-based Model
• Metro
• MANO

Please also consider cite the above projects if you find them helpful.