MFA-Net: Motion Feature Augmented Network for Dynamic Hand Gesture Recognition from Skeletal Data

Xinghao Chen¹, Guijin Wang¹, Hengkai Guo², Cairong Zhang¹, Hang Wang³, Li Zhang¹

¹Department of Electronic Engineering, Tsinghua University     ²AI Lab, Bytedance Inc     ³Beijing Huajie IMI Technology Co., Ltd

[PDF @ ICIP'17]    [PDF @ Sensors'19]

Introduction

This repository contains the demo code for MFA-Net, an accurate method for skeleton-based dynamic hand gesture recognition.

Figure 1: Framework of MFA-Net.

Citation

If you find our work useful in your research, please consider citing:

@article{chen2019mfanet,
  title={MFA-Net: Motion Feature Augmented Network for Dynamic Hand Gesture Recognition from Skeletal Data},
  author={Chen, Xinghao and Wang, Guijin and Guo, Hengkai and Zhang, Cairong and Wang, Hang and Zhang, Li},
  journal={Sensors},
  year={2019},
  volume = {19},
  number = {2},
  ARTICLE-NUMBER = {239},
  doi = {10.3390/s19020239}
}

@inproceedings{chen2017motion,
  title={Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition},
  author={Chen, Xinghao and Guo, Hengkai and Wang, Guijin and Zhang, Li},
  booktitle={Image Processing (ICIP), 2017 IEEE International Conference on},
  pages={2881--2885},
  year={2017},
  organization={IEEE}
}

Dependencies

• keras 2.0.6
• theano 0.9.0 or tensorflow 1.2.0
• seaborn (For drawing confusion matrix)

Datasets

• DHG-14/28 Dataset [1]
• SHREC17 Dataset [2]

Training

bash sh/train_MFA_Net_shrec17_vae_14.sh  # For DHG-14/28 dataset (14 gestures)
bash sh/train_MFA_Net_shrec17_vae_14.sh  # For DHG-14/28 dataset (28 gestures)
bash sh/train_MFA_Net_shrec17_vae_14.sh  # For SHREC17 dataset (14 gestures)
bash sh/train_MFA_Net_shrec17_vae_14.sh  # For SHREC17 dataset (28 gestures)

Testing:

bash sh/test_MFA_Net_shrec17_vae_14.sh  # For DHG-14/28 dataset (14 gestures)
bash sh/test_MFA_Net_shrec17_vae_14.sh  # For DHG-14/28 dataset (28 gestures)
bash sh/test_MFA_Net_shrec17_vae_14.sh  # For SHREC17 dataset (14 gestures)
bash sh/test_MFA_Net_shrec17_vae_14.sh  # For SHREC17 dataset (28 gestures)

Use the following command to draw confusion matrix for the predicted results:

python src/cnf_DHG.py  # For DHG-14/28 dataset
python src/cnf_SHREC.py  # For SHREC17 dataset

Reference

• [1] Skeleton-based Dynamic hand gesture recognition, Quentin De Smedt, Hazem Wannous and Jean-Philippe Vandeborre, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
• [2] SHREC'17 Track: 3D Hand Gesture Recognition Using a Depth and Skeletal Dataset, Quentin De Smedt, Hazem Wannous and Jean-Philippe Vandeborre, Joris Guerry, Bertrand Le Saux, David Filliat, Eurographics Workshop on 3D Object Retrieval (2017).