BRL

PyTorch implementation of "Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition" [PDF]

Dependencies

• Python >= 3.6
• PyTorch >= 1.5.0
• PyYAML, tqdm, tensorboardX
• We provide the dependency file of the experimental environment, you can install all dependencies by creating a new anaconda virtual environment and running pip install -r requirements.txt
• Run pip install -e .

Data Preparation

Download datasets

There are 4 datasets to download:

• NTU RGB+D 60 Skeleton
• NTU RGB+D 120 Skeleton
• NW-UCLA
• Kinetics 400 Skeleton

Please download the raw data from NTU RGB+D 60, NTU RGB+D 120, NW-UCLA, and Kinetics Skeleton 400.

For detailed download instructions, please go to the README.

Data Processing

Generating Data

Generate NTU RGB+D 60, NTU RGB+D 120, NW-UCLA, and Kinetics Skeleton 400 datasets:

cd ./data
python ntu_gendata.py
python ntu120_gendata.py
python ucla_data.py
python kinetics_gendata.py

Meanwhile, PYSKL provides the processed skeleton data for all datasets as pickle files (which can be directly used for training and testing). Please check Data Doc for the download links and descriptions of the annotation format. You can also use the provided script to generate the processed pickle files.

Constructing Long-tailed Datasets

Construct NTU 60-LT, NTU 120-LT, and NW-UCLA-LT：

cd ./data
python imblance_gentxt.py
python imblance_gendata_from_txt.py

Training & Testing

You can use following commands for training and testing. Basically, we support distributed training on a single server with multiple GPUs.

# Training
bash tools/dist_train.sh {config_name} {num_gpus} {other_options}
# For example: train BRL on NTURGB+D XSub (Joint Modality) with one GPU, with validation, and test the last and the best (with best validation metric) checkpoint.
bash tools/dist_train.sh configs/BRL/ntu60_xsub_LT/j.py 1 --validate --test-last --test-best

# Testing
bash tools/dist_test.sh {config_name} {checkpoint} {num_gpus} --out {output_file} --eval {optional_arguments}
# For example: test BRL on NTURGB+D XSub (Joint Modality) with metrics `top_k_accuracy`, and dump the result to `result.pkl`.
bash tools/dist_test.sh configs/BRL/ntu60_xsub_LT/j.py checkpoints/SOME_CHECKPOINT.pth 1 --eval top_k_accuracy --out result.pkl

# Ensemble the results
cd ./tools
python ensemble.py

Experimental results

We provide the results trained on NTURGB+D 60 and NTURGB+D 120 in the long-tailed training setting. We also provide checkpoints for six modalities: Joint, Bone, Skip, Joint Motion, Bone Motion, and Skip Motion. The accuracy of each modality links to the weight file.

BRL Model Zoo

We release numerous checkpoints for BRL trained with various modalities, annotations on NTURGB+D and NTURGB+D 120. The accuracy of each modality links to the weight file.

Dataset	Annotation	Joint Top1	Bone Top1	Skip Top1	Joint Motion Top1	Bone Motion Top1	Skip Motion Top1	Two-Stream Top1	Four Stream Top1	Six Stream Top1
NTURGB+D XSub	Official 3D Skeleton	joint_config: 76.7	bone_config: 76.1	skip_config: 77.7	joint_motion_config: 75.0	bone_motion_config: 72.8	skip_motion_config: 73.4	79.6	81.0	81.8
NTURGB+D XView	Official 3D Skeleton	joint_config: 81.4	bone_config: 80.3	skip_config: 81.1	joint_motion_config: 78.5	bone_motion_config: 76.2	skip_motion_config: 77.2	84.0	84.9	85.4
NTURGB+D 120 XSub	Official 3D Skeleton	joint_config: 65.3	bone_config: 65.3	skip_config: 64.2	joint_motion_config: 59.7	bone_motion_config: 59.8	skip_motion_config: 59.6	68.7	69.4	69.7
NTURGB+D 120 XSet	Official 3D Skeleton	joint_config: 66.8	bone_config: 66.6	skip_config: 65.9	joint_motion_config: 63.5	bone_motion_config: 62.2	skip_motion_config: 61.6	69.7	71.0	71.3
Kinetics-400	HRNet 2D Pose	joint_config: 45.6	bone_config: 45.2		joint_motion_config: 42.0	bone_motion_config: 41.2		48.1 (1:1)	48.6 (3:3:1:1)

Supported Algorithms

• ST-GCN (AAAI 2018) [Model Zoo]
• AAGCN (TIP) [Model Zoo]
• MS-G3D (CVPR 2020 Oral) [Model Zoo]
• CTR-GCN (ICCV 2021) [Model Zoo]

For specific examples and other pre-trained models, please go to the README.

Acknowledgements

This repo is based on MS-G3D, CTR-GCN, and PYSKL.

Thanks to the original authors for their work!

Citation

Please cite this work if you find it useful:

  @article{liu2023balanced,
    title={Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition},
    author={Liu, Hongda and Wang, Yunlong and Ren, Min and Hu, Junxing and Luo, Zhengquan and Hou, Guangqi and Sun, Zhenan},
    journal={arXiv preprint arXiv:2308.14024},
    year={2023}
  }

Contact

For any questions, feel free to contact: hongda.liu@cripac.ia.ac.cn