Hierarchical Skeleton Meta-Prototype Contrastive Learning with Hard Skeleton Mining for Unsupervised Person Re-Identification

By Haocong Rao, Cyril Leung, and Chunyan Miao. In International Journal of Computer Vision (IJCV), 2023 (In Press) (Arxiv), (Paper).

Introduction

This is the official implementation of Hi-MPC model presented by "Hierarchical Skeleton Meta-Prototype Contrastive Learning with Hard Skeleton Mining for Unsupervised Person Re-Identification". The codes are used to reproduce experimental results of the proposed Multi-level Skeleton Meta-Representation (MSMR) in the paper.

Abstract: With rapid advancements in depth sensors and deep learning, skeleton-based person re-identification (re-ID) models have recently achieved remarkable progress with many advantages. Most existing solutions learn single-level skeleton features from body joints with the assumption of equal skeleton importance, while they typically lack the ability to exploit more informative skeleton features from various levels such as limb level with more global body patterns. The label dependency of these methods also limits their flexibility in learning more general skeleton representations. This paper proposes a generic unsupervised Hierarchical skeleton Meta-Prototype Contrastive learning (Hi-MPC) approach with Hard Skeleton Mining (HSM) for person re-ID with unlabeled 3D skeletons. Firstly, we construct hierarchical representations of skeletons to model coarse-to-fine body and motion features from the levels of body joints, components, and limbs. Then a hierarchical meta-prototype contrastive learning model is proposed to cluster and contrast the most typical skeleton features ("prototypes") from different-level skeletons. By converting original prototypes into meta-prototypes with multiple homogeneous transformations, we induce the model to learn the inherent consistency of prototypes to capture more effective skeleton features for person re-ID. Furthermore, we devise a hard skeleton mining mechanism to adaptively infer the informative importance of each skeleton, so as to focus on harder skeletons to learn more discriminative skeleton representations. Extensive evaluations on five datasets demonstrate that our approach outperforms a wide variety of state-of-the-art skeleton-based methods. We further show the general applicability of our method to cross-view person re-ID and RGB-based scenarios with estimated skeletons.

Environment

• Python >= 3.5
• Tensorflow-gpu >= 1.14.0
• Pytorch >= 1.1.0
• Faiss-gpu >= 1.6.3

Here we provide a configuration file to install the extra requirements (if needed):

conda install --file requirements.txt

Note: This file will not install tensorflow/tensorflow-gpu, faiss-gpu, pytroch/torch, please install them according to the cuda version of your graphic cards: Tensorflow, Pytorch. Take cuda 9.0 for example:

conda install faiss-gpu cuda90 -c pytorch
conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
conda install tensorflow-gpu==1.14
conda install scikit-learn

Datasets and Models

We provide three already pre-processed datasets (IAS-Lab, BIWI, KGBD) with various sequence lengths (f=4/6/8/10/12) here (pwd: 7je2) and the pre-trained models here (pwd: xqho). Since we report the average performance of our approach on all datasets, here the provided models may produce better results than the paper.

Please download the pre-processed datasets and model files while unzipping them to Datasets/ and ReID_Models/ folders in the current directory.

Note: The access to the Vislab Multi-view KS20 dataset and large-scale RGB-based gait dataset CASIA-B are available upon request. If you have signed the license agreement and been granted the right to use them, please email us with the signed agreement and we will share the complete pre-processed KS20 and CASIA-B data. The original datasets can be downloaded here: IAS-Lab, BIWI, KGBD, KS20, CASIA-B. We also provide the Preprocess.py for directly transforming original datasets to the formated training and testing data.

Dataset Pre-Processing

To (1) extract 3D skeleton sequences of length f=6 from original datasets and (2) process them in a unified format (.npy) for the model inputs, please simply run the following command:

python Preprocess.py 6

Note: If you hope to preprocess manually (or you can get the already preprocessed data (pwd: 7je2)), please frist download and unzip the original datasets to the current directory with following folder structure:

[Current Directory]
├─ BIWI
│    ├─ Testing
│    │    ├─ Still
│    │    └─ Walking
│    └─ Training
├─ IAS
│    ├─ TestingA
│    ├─ TestingB
│    └─ Training
├─ KGBD
│    └─ kinect gait raw dataset
└─ KS20
     ├─ frontal
     ├─ left_diagonal
     ├─ left_lateral
     ├─ right_diagonal
     └─ right_lateral

After dataset preprocessing, the auto-generated folder structure of datasets is as follows:

Datasets
├─ BIWI
│    └─ 6
│      ├─ test_npy_data
│      │    ├─ Still
│      │    └─ Walking
│      └─ train_npy_data
├─ IAS
│    └─ 6
│      ├─ test_npy_data
│      │    ├─ A
│      │    └─ B
│      └─ train_npy_data
├─ KGBD
│    └─ 6
│      ├─ test_npy_data
│      │    ├─ gallery
│      │    └─ probe
│      └─ train_npy_data
└─ KS20
    └─ 6
      ├─ test_npy_data
      │    ├─ gallery
      │    └─ probe
      └─ train_npy_data

Note: KS20 data need first transforming ".mat" to ".txt". If you are interested in the complete preprocessing of KS20 and CASIA-B, please contact us and we will share. We recommend to directly download the preprocessed data here (pwd: 7je2).

Model Usage

To (1) train the unsupervised Hi-MPC model to obtain skeleton representations (MSMR) and (2) validate their effectiveness on the person re-ID task on a specific dataset (probe), please simply run the following command:

python Hi-MPC.py --dataset KS20 --probe probe

# Default options: --dataset KS20 --probe probe --length 6  --gpu 0
# --dataset [IAS, KS20, BIWI, KGBD]
# --probe ['probe' (the only probe for KS20 or KGBD), 'A' (for IAS-A probe), 'B' (for IAS-B probe), 'Walking' (for BIWI-Walking probe), 'Still' (for BIWI-Still probe)] 
# --length [4, 6, 8, 10, 12] 
# --(H, M, eps, min_samples, lr, etc.) with default settings for each dataset
# --mode [Train (for training), Eval (for testing)]
# --gpu [0, 1, ...]

To print Rank-1 accuracy, Rank-5 accuracy, Rank-10 accuracy and mAP when applying different level representations (joint-level, component-level, limb-level, MSMR) of the best model saved in default directory (ReID_Models/(Dataset)/(Probe)), run:

python Hi-MPC.py --dataset KS20 --probe probe --mode Eval

Application to Model-Estimated Skeleton Data

Estimate 3D Skeletons from RGB-Based Scenes

To apply our Hi-MPC to person re-ID under the large-scale RGB scenes (CASIA B), we exploit pose estimation methods to extract 3D skeletons from RGB videos of CASIA B as follows:

• Step 1: Download CASIA-B Dataset
• Step 2: Extract the 2D human body joints by using OpenPose
• Step 3: Estimate the 3D human body joints by using 3DHumanPose

We provide already pre-processed skeleton data of CASIA B for single-condition (Nm-Nm, Cl-Cl, Bg-Bg) and cross-condition evaluation (Cl-Nm, Bg-Nm) (f=40/50/60) here (pwd: 07id). Please download the pre-processed datasets into the directory Datasets/.

Usage

To (1) train the Hi-MPC model to obtain skeleton representations (MSMR) and (2) validate their effectiveness on the person re-ID task on CASIA B under single-condition and cross-condition settings, please simply run the following command:

python Hi-MPC.py --dataset CASIA_B --probe_type nm.nm --length 40

# --length [40, 50, 60] 
# --probe_type ['nm.nm' (for 'Nm' probe and 'Nm' gallery), 'cl.cl', 'bg.bg', 'cl.nm' (for 'Cl' probe and 'Nm' gallery), 'bg.nm']  
# --(H, M, eps, min_samples, lr, etc.) with default settings
# --gpu [0, 1, ...]

Please see Hi-MPC.py for more details.

Citation

If you found this repository useful, please consider citing:

@article{rao2023hierarchical,
  title={Hierarchical Skeleton Meta-Prototype Contrastive Learning with Hard Skeleton Mining for Unsupervised Person Re-Identification},
  author={Rao, Haocong and Leung, Cyril and Miao, Chunyan},
  journal={arXiv preprint arXiv:2307.12917},
  year={2023}
}

License

Hi-MPC is released under the MIT License. Our models and codes must only be used for legitimate research.