Skip to content
/ CASA Public

Context-Aware Sequence Alignment using 4D Skeletal Augmentation CVPR 2022


Notifications You must be signed in to change notification settings


Repository files navigation

Context-Aware Sequence Alignment using 4D Skeletal Augmentation

Taein Kwon, Bugra Tekin, Sigyu Tang, and Marc Pollefeys CASA

This code is for Context-Aware Sequence Alignment using 4D Skeletal Augmentation, CVPR 2022 (oral). You can see more details about more paper in our project page. Note that we referred LoFTR to implement our framework.

Environment Setup

To setup the env,

git clone
conda env create -f env.yml
conda activate CASA

External Dependencies

Folder Structures

├── bodymocap
├── extra_data
│   └── body_module
│         └── J_regressor_extra_smplx.npy
├── human_body_prior
│   └── ...
├── manopth
│   ├──
│   ├──
│   └── ...
├── smpl
│   └── models
│         └── SMPLX_NEUTRAL.pkl
├── mano 
│   ├── models
│   │     ├── MANO_LEFT.pkl
│   │     └── MANO_RIGHT.pkl
│   ├── websuers
│   ├──
│   └── LICENSE.txt
├── npyrecords
├── sripts
├── src
└── ...

Install MANO and Manopth

In this repository, we use the MANO model from MPI and some part of Yana's code for hand pose alignment.

  • Clone manopth git clone and copy manopth and mano folder (inside) into the CASA folder.
  • Go to the mano website and download models and code and put them in CASA/mano/models.
  • In, please change following lines to run in python3. L23:import CPickle as pickle -> import pickle, L30: dd = pickle.load(open(fname_or_dick)) -> dd = pickle.load(open(fname_or_dict, 'rb'), encoding='latin1'), L144: print 'FINTO' -> print('FINTO').

Install Vposer

We used Vposer to augment body pose.

  • In the Vposer repository, clone it git clone
  • Copy the human_body_prior folder into the CASA folder.
  • Go into the human_body_prior folder and run the
    cd human_body_prior
    python develop

Install SMPL

We use the SMPL model for body pose alignment.

  • Download SMPL-X ver 1.1. and VPoser V2.0.
  • Put the SMPLX_NUETRAL.pkl into the CASA/smpl/models folder.
  • Copy VPoser files in CASA/human_body_prior/support_data/downlaods/vposer_v2_05/
  • In order to use the joints from FrankMocap, you need to additionally clone FrankMocap git clone and put the bodymocap fodler into CASA fodler.
  • After then, run sh to get the J_regressor_extra_smplx.npy file.


You can download npy files here. In the npy files, normalized joints and labels are included. In order to get the original data, you should go to each dataset websites and download the datasets there.


We selected pouring milk sequences and manually divided into train and test set with the new labels we set. Please go to the H2O project page and download the dataset there.


We estimated 3D joints using FrankMocap for the Penn Action dataset. Penn Action has 13 different actions: baseball_pitch, baseball_swing, bench_press, bowling, clean_and_jerk, golf_swing, jumping_jacks, pushups, pullups, situp, squats, tennis_forehand, tennis_serve.


We downloaded and used the 3D joints from triangulation of 2D poses in the IkeaASM dataset.


To train the Penn Action dataset,

sh scripts/train/ ${dataset_name}

For example,

sh scripts/train/ tennis_serve


We also provide pre-trained models. To evalate the pre-trained model

sh scripts/eval/ ${dataset_name} ${eval_model_path}

For example,

sh scripts/eval/ tennis_serve logs/tennis_serve/CASA=64/version_0/checkpoints/last.ckpt


Note that our code follows the Apache License 2.0. However, external libraries follows their own licenses.


Context-Aware Sequence Alignment using 4D Skeletal Augmentation CVPR 2022







No releases published