This code is for the paper: Temporal Hierarchical Dictionary Guided Decodingfor Online Gesture Segmentation and Recognition. We propose a novel hybrid HMM-DNN framework for online segmentation and recognition of skeleton-based human gestures. The network is tested on the four datasets, MSRA, OAD action dataset, DHG gesture dataset and Chalearn 2014 dataset. We report state-of-the-art performances on all these datasets.
Code written by Chen Haoyu, University of Oulu.
The original code was written in Theano, we re-implemented it with Keras-Tensorflow.
We train and evaluate on Ubuntu 16.04, it will also work for Windows and OS.
- step1_generateEntropyMap is used for generating THD-HMM with entropy maps
- step2_THD_HMM-LSTM is used for training the BiLSTM with the generated THD-HMM for recognition and segmentation
-
We already prepare the pretrained THD-HMM dictionary for OAD dataset, which can be used for training and validing the networkds to recognize the gestures directly. So you can skip step1_generateEntropyMap and move to step2_THD_HMM-LSTM directly.
-
- Download the dataset and put it into folder ./step2_THD_HMM-LSTM, OAD can be download from here
-
- run the code with:
python main.py
- run the code with:
-
- check the experiment results in the folder
./experi/
.
- check the experiment results in the folder
-
- Download the dataset and put it into folder ./step1_THD_HMM-LSTM, Chalearn dataset can be download from here, note that we use Track 3: gesture recognition, for validating.
-
- run the code
main_THD.m
with matlab.
- run the code
-
- check the generated THD-HMM and calculated Entropy maps in the folder
./template/
.
- check the generated THD-HMM and calculated Entropy maps in the folder
Ubuntu 16.04
Python 3.6.5
Keras 2.3.1
Tensorflow-gpu==1.15.0
cuda ==10.0 (cuda 10.1 is not compatible)
Please cite these papers in your publications if it helps your research.
@article{chen2020temporal, title={Temporal Hierarchical Dictionary Guided Decoding for Online Gesture Segmentation and Recognition}, author={Chen, Haoyu and Liu, Xin and Shi, Jingang and Zhao, Guoying}, journal={IEEE Transactions on Image Processing}, volume={29}, pages={9689--9702}, year={2020}, publisher={IEEE} }
Copyright@ University of Oulu