Future Transformer for Long-term Action Anticipation (CVPR 2022)

Project Page | Paper

This repository contains the official source code and data for our paper:

Future Transformer for Long-term Action Anticipation
Dayoung Gong, Joonseok Lee, Manjin Kim, Seong Jong Ha, and Minsu Cho POSTECH & NCSOFT CVPR, New Orleans, 2022.

Citation

If you find our code or paper useful, please consider citing our paper:

@inproceedings{gong2022future,
  title={Future Transformer for Long-term Action Anticipation},
  author={Gong, Dayoung and Lee, Joonseok and Kim, Manjin and Ha, Seong Jong and Cho, Minsu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3052--3061},
  year={2022}
}

Experiments

We conduct experiments on Breakfast with 4 splits and 50Salads with 5 splits.

Dataset	Model	obs 0.2, pred 0.1	obs 0.2, pred 0.2	obs 0.2, pred 0.3	obs 0.2, pred 0.4	obs 0.3, pred 0.1	obs 0.3, pred 0.2	obs 0.3, pred 0.3	obs 0.3, pred 0.4	Checkpoint (Splits)
Breakfast	FUTR	27.71	24.56	22.84	22.05	32.27	29.89	27.49	25.88	1 2 3 4
50Salads	FUTR	37.01	27.81	22.46	16.75	33.32	23.17	22.14	15.49	1 2 3 4 5

Environmental setup

• Conda environment settings:

conda env export > futr.yaml
conda activate futr

Dataset

Download the data from https://mega.nz/file/O6wXlSTS#wcEoDT4Ctq5HRq_hV-aWeVF1_JB3cacQBQqOLjCIbc8 .
Create a directory './datasets' for the two datasets and place each dataset to have following directory structure:

    ../                         # parent directory
    ├── ./                      # current (project) directory
    │   ├── data/               # (dir.) dataloaders for action anticipation dataset
    │   ├── model/              # (dir.) implementation of Hypercorrelation Squeeze Network model 
    │   ├── README.md           # intstruction for reproduction
    │   ├── train.py            # code for training FUTR
    │   ├── predict.py          # code for testing FUTR
    │   ├── otps.py             # code for arguments
    │   └── utils.py            # code for helper functions
    └── datasets/
        ├── breakfast/          # Breakfast dataset
        │   ├── groundTruth/
        │   ├── features/
        │   ├── mapping.txt
        │   └── ...
        ├── 50salads/          # 50salads dataset
        │   ├── groundTruth/
        │   ├── features/
        │   ├── mapping.txt
        │   └── ...

Training

1. Breakfast

./scripts/train.sh $split_num

2. 50salads

./scripts/50s_train.sh $split_num

Testing

1. Breakfast

./scripts/predict.sh $split_num

2. 50salads

./scripts/50s_predict.sh $split_num

Acknowledgement

We thank Yazan Abu Farha for providing the code of Long-term anticipation of activities with cycle consistency and for helping us to reproduce experiments.