Skip to content
/ CPF Public

[ICCV 2021] CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

License

Notifications You must be signed in to change notification settings

lixiny/CPF

Repository files navigation


CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

Logo

ICCV, 2021
Lixin Yang · Xinyu Zhan · Kailin Li · Wenqiang Xu · Jiefeng Li · Cewu Lu

Paper PDF Project Page Youtube Video


This repo contains model, demo, and test codes.

Table of Contents
  1. Installation
  2. Demo
  3. Evaluation
  4. Anatomical Constrained A-MANO
  5. TODO
  6. Citation


Installation

Following the Installation Instruction to setup environment, assets, datasets and models.

Demo

Notice: require active screen.

visualize GeO fitting pipeline

We create a FHBExample dataset in hocontact/hodatasets/fhb_example.py that only contains 10 samples to demonstrate our pipeline.

# Only support single GPU !
$ python scripts/run_demo.py \
    --gpu 0 \
    --init_ckpt CPF_checkpoints/picr/fhb/checkpoint_200.pth.tar \
    --honet_mano_fhb_hand


Press q in the "runtime hand" window to start fitting

visualize A-MANO'S anchor position

This demo shows the anochrs positions on MANO hand's surface

$ python scripts/recover_anchor.py --render

Evaluation

We provide shell srcipts for evaluating on FHB, HO3Dv1 and v2.

FHB dataset

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_FHB.sh

and fit GeO optimizer:

# setting 1: hand-only 
# recommend 4 GPUs
$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handonly_FHB.sh

# setting 2: hand-obj
$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handobj_FHB.sh

HO3Dv1

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv1.sh

and fit GeO optimizer:

# hand-only
# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handonly_HO3Dv1.sh

# hand-obj
# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv1.sh

HO3Dv2 (version 2)

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv2.sh

and fit GeO optimizer:

# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv2.sh

evaluation results

Above scripts may take a while ( ~ 1 day ). We also provide the results in fitting_res.

Anatomical Constrained A-MANO

We provide pytorch implementation of our Anatomical Constrained MANO in lixiny/manopth, which is modified from the original hassony2/manopth.

TODO

  • testing code and pretrained models
    • HoNet (FHB, HO3Dv1/v2)
    • PiCR (FHB, HO3Dv1/v2)
  • fitting code of GeO, both hand-only and hand-object (FHB, HO3Dv1/v2)
  • training code
  • contact region visualization

Citation

If you find this work helpful, please consider citing us:

@inproceedings{yang2021cpf,
    title={{CPF}: Learning a Contact Potential Field to Model the Hand-Object Interaction},
    author={Yang, Lixin and Zhan, Xinyu and Li, Kailin and Xu, Wenqiang and Li, Jiefeng and Lu, Cewu},
    booktitle={ICCV},
    year={2021}
}

About

[ICCV 2021] CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

Resources

License

Stars

Watchers

Forks

Packages