STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

The implementation of STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising in ISMAR 2021.

We propose a method for denoising hand motion data in mixed reality using a spatial-temporal graph auto-encoder. Our approach models the articulated hand structure with a partition strategy and extracts structural constraints using graph convolution and self-attention. We combine graph and temporal convolutions in an hourglass residual auto-encoder to preserve structural constraints while denoising hand motion data. Our method outperforms state-of-the-art approaches in both quantitative and qualitative experiments.

Paper Address | Download Paper | Supplementary Video | Multi-STGAE

Framework

The pipeline of the proposed method for hand motion denoising using STGAE is shown in the above figure. For more details, please refer to the supplementary video.

Experiments

• Denoising results

More demos can be seen in our supplementary video.

Environments

• LaTeX tool

sudo apt-get install texlive-full

• FFmpeg

sudo apt-get install ffmpeg

• pydot & graphviz

sudo pip3 install pydot
sudo pip3 install graphviz

App

The core code of the client and server is available, and detailed development tips can be found at the link

Citation

@inproceedings{zhou2021stgae,
  title={STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising},
  author={Zhou, Kanglei and Cheng, Zhiyuan and Shum, Hubert PH and Li, Frederick WB and Liang, Xiaohui},
  booktitle={2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
  pages={41--49},
  year={2021},
  organization={IEEE}
}

Contact

Feel free to contact me via zhoukanglei[at]qq.com.