STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition

This repository (work-in-progress) contains the code for the following CVPR'23 paper:

STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition

Introduction

We study the problem of human action recognition using motion capture (MoCap) sequences. Unlike existing techniques that take multiple manual steps to derive standardized skeleton representations as model input, we propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The model uses a hierarchical transformer with intra-frame off-set attention and inter-frame self-attention. The attention mechanism allows the model to freely attend between any two vertex patches to learn non-local relationships in the spatial-temporal domain. Masked vertex modeling and future frame prediction are used as two self-supervised tasks to fully activate the bi-directional and auto-regressive attention in our hierarchical transformer. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models on common MoCap benchmarks.

Citation

If you find our work useful in your research, please consider citing:

@InProceedings{Zhu_2023_CVPR,
    author    = {Zhu, Xiaoyu and Huang, Po-Yao and Liang, Junwei and de Melo, Celso M. and Hauptmann, Alexander G.},
    title     = {STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {1526-1536}
}

Dataset

• Please download the KIT and BABEL dataset from this link.

• The dataset split is available at Google Drive.

Code

Training and Inference

To perform training and inference, please run:

$ python3 main.py --root_path ./data/kit_pt_processed_dataset --save_root_dir ./ckpt/stmt_training --framenum 24

Pre-Trained Model

Please use this link to download our pre-trained model: Google Drive

License

Our code and models are only for ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY.

Acknowledgements

Our STMT is based on Point-Spatio-Temporal-Convolution, P4Transformer and SequentialPointNet.