Action Anticipation using Latent Goal Learning

Code accompanying the IEEE WACV 2022 paper "Action anticipation using latent goal learning"

EPIC-KITCHENS 55

• Training

  Download RGB features from RULSTM project, specifically this script

  https://github.com/fpv-iplab/rulstm/blob/master/RULSTM/scripts/download_data_ek55.sh

  and these lines

  mkdir -p data/ek55/rgb
  curl https://iplab.dmi.unict.it/sharing/rulstm/features/rgb/data.mdb -o data/ek55/rgb/data.mdb
  

  The data is now in /data/ek55/rgb. Next, fetch the training.csv and validation.csv from the RULSTM project ek55 directory

  Run the training script - tsnrgb_feat_latent_goal_action_max_current_action.py

• Testing on test set

  • Fetch the test_seen.csv and test_unseen.csv from the RULSTM project ek55 directory

  • The CSV format is different in training.csv and test_seen.csv. For training.csv, the columns are - segment_id, video_id, start_frame, end_frame, verb, noun, action For test_seen/unseen.csv, the columns are - segment_id, video_id, start_frame, end_frame

  • We need to train 2 models - one with RGB features as above and another with OBJ features Download OBJ features from RULSTM project, specifically this script

    mkdir -p data/ek55/obj
    curl https://iplab.dmi.unict.it/sharing/rulstm/features/obj/data.mdb -o data/ek55/obj/data.mdb
    

    The data is now in /data/ek55/obj.

  Run the training script tsnrgb_feat_latent_goal_action_max_current_action.py. Change feat_dim in main() to 352

Breakfast and 50 Salads

I3D features were obtained from this repo for both 50Salads and Breakfast.

https://github.com/yabufarha/ms-tcn

Download the data folder, which contains the features and the ground truth labels. (~30GB) (If you cannot download the data from the previous link, try to download it from here)

Then run [i3d_latent_goal_bf.py] (https://github.com/debadityaroy/LatentGoal/blob/main/i3d_latent_goal_bf.py)

We have actions instead of verb and nouns for Breakfast and 50Salads.

Acknowledgment

This research/project is supported in part by the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: AISG2-RP-2020-016) and the National Research Foundation Singapore under its AI Singapore Program (Award Number: AISG-RP-2019-010).

In case of issues, please write to roy_debaditya [at] ihpc [dot] a-star [dot] edu [dot] sg

Please cite this work if you use this code

@inproceedings{
wacv22,
title={Action anticipation using latent goal learning},
author={Debaditya Roy and Basura Fernando},
booktitle={WACV},
year={2022},
}