Skip to content
/ UNIK Public

[BMVC 2021 Oral] Official implementation of our paper "A Unified Framework for Real-world Skeleton-based Action Recognition" on Toyota Smarthome/Penn Action/NTU-RGB+D/Posetics datasets

License

Notifications You must be signed in to change notification settings

YangDi666/UNIK

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Unified Framework for Real-world Skeleton Action Recognition

SSTA-PRS: Selective Spatio-Temporal Aggregation Based Pose Refinement System | Project page
UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition | Project page
ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting | Project page

Evironment

-- Python3 with PyTorch version >=Pytorch0.4.

Data Preparation

  • mkdir data

  • Posetics: please contact us (di.yang@inria.fr) for Data Request. ad

  • Toyota Smarthome: download the raw data (skeleton-v2.0 refined by SSTA-PRS).

  • Penn Action: download the raw skeleton data.

  • For other datasets: NTU-RGB+D/Skeleton-Kinetics.

  • Put them under the data directory:

     -data\
      -smarthome_raw\
         -smarthome_skeletons\
          - ... .json
            ... .json
            ...               
      -pennAction_raw\
         -skeletons\
          - ... .json
            ...
      -posetics_raw\
         -posetics_train_label.json
         -posetics_val_label.json  
         -posetics_train\
          - ... .json
            ...        
         -posetics_val\
          - ... .json
            ... 
      -nturgbd_raw\
         -samples_with_missing_skeletons.txt
         -nturgb+d_skeletons\
          - ... .skeleton
          - ...
      -kinetics_raw\             
             ...
      -...
    
  • Preprocess the data with

    cd data_gen
    python smarthome_gendata.py
    python penn_gendata.py
           ...
    
  • Generate the bone data with:

    python gen_bone_data.py

Pre-training on Posetics

python run_unik.py --config ./config/posetics/train_joint.yaml

Pre-trained model is now avalable here. Move it to

./weights/

Training (Fine-tuining) & Testing

Change the config file depending on what you want (e.g., for Smarthome).

python run_unik.py --config ./config/smarthome-cross-subject/train_joint.yaml

python run_unik.py --config ./config/smarthome-cross-subject/train_bone.yaml

To ensemble the results of joints and bones, run test firstly to generate the scores of the softmax layer.

python run_unik.py --config ./config/smarthome-cross-subject/test_joint.yaml

python run_unik.py --config ./config/smarthome-cross-subject/test_bone.yaml

Then combine the generated scores with:

python ensemble.py --datasets smarthome/xsub

For evaluation on Smarthome:

  • Cross-subject:

    python evaluation-cs.py runs/smarthome/smarthome_cs_unik_test_joint_right.txt 31
    
  • Cross-view:

    python evaluation-cv.py runs/smarthome/smarthome_cv2_unik_test_joint_right.txt 19
    python evaluation-cv.py runs/smarthome/smarthome_cv1_unik_test_joint_right.txt 19
    

Citation

If you find this code useful for your research, please consider citing our paper:

@article{yang2021unik,
      title={UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition}, 
      author={Di Yang and Yaohui Wang and Antitza Dantcheva and Lorenzo Garattoni and Gianpiero Francesca and Francois Bremond},
      year={2021},
      journal={BMVC}
}

@article{yang2022via,
      title={ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting}, 
      author={Di Yang and Yaohui Wang and Antitza Dantcheva and Lorenzo Garattoni and Gianpiero Francesca and Francois Bremond},
      year={2022},
      journal={arXiv preprint arXiv:2209.00065}
}

About

[BMVC 2021 Oral] Official implementation of our paper "A Unified Framework for Real-world Skeleton-based Action Recognition" on Toyota Smarthome/Penn Action/NTU-RGB+D/Posetics datasets

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages