Proposal-Free Temporal Action Detection with Global Segmentation Mask Learning

Sauradip Nag^1,2,+  Xiatian Zhu^1,3  Yi-Zhe Song^1,2  Tao Xiang^1,2 

¹CVSSP, University of Surrey, UK  ²iFlyTek-Surrey Joint Research Center on Artificial Intelligence, UK 
³Surrey Institute for People-Centred Artificial Intelligence, UK

⁺corresponding author

Accepted to ECCV 2022

Paper | Project Page

Updates

• (June, 2022) We released TAGS training and inference code for ActivityNetv1.3 dataset.
• (June, 2022) TAGS is accepted by ECCV 2022.

Summary

• First proposal-free framework for Temporal Action Detection (TAD) task.
• Converted the traditional regression based Boundary Prediction into a classification problem.
• Predicts Global Masks for action instances and generates start/end points from mask start/end points.
• A novel Boundary Refinement and Alignment strategy is proposed which ensures inter-branch consistency.
• Efficient and Faster than all the existing Anchor-based and Anchor-Free approaches.

Contact

Regarding any issues on the implementation you can contact the first author at s.nag@surrey.ac.uk

Abstract

Existing temporal action detection (TAD) methods rely on generating an overwhelmingly large number of proposals per video. This leads to complex model designs due to proposal generation and/or per-proposal action instance evaluation and the resultant high computational cost. In this work, for the first time, we propose a proposal-free Temporal Action detection model with Global Segmentation mask (TAGS). Our core idea is to learn a global segmentation mask of each action instance jointly at the full video length. The TAGS model differs significantly from the conventional proposal-based methods by focusing on global temporal representation learning to directly detect local start and end points of action instances without proposals. Further, by modeling TAD holistically rather than locally at the individual proposal level, TAGS needs a much simpler model architecture with lower computational cost. Extensive experiments show that despite its simpler design, TAGS outperforms existing TAD methods, achieving new state-of-the-art performance on two benchmarks. Importantly, it is ~ 20x faster to train and ~1.6x more efficient for inference.

Architecture

Getting Started

Requirements

• Python 3.7
• PyTorch == 1.9.0 (Please make sure your pytorch version is atleast 1.8)
• NVIDIA GPU
• Kornia

Environment Setup

It is suggested to create a Conda environment and install the following requirements

pip3 install -r requirements.txt

Download Features

Download the video features and update the Video paths/output paths in config/anet.yaml file. For now ActivityNetv1.3 dataset config is available. We are planning to release the code for THUMOS14 dataset soon. We obtain the I3D features from ACM-Net and also process the annotations similar to TSP

Dataset	Feature Backbone	Pre-Training	Link
ActivityNet	TSN	Kinetics-400	Google Drive
THUMOS	TSN	Kinetics-400	Google Drive
ActivityNet	I3D	Kinetics-400	Google Drive
THUMOS	I3D	Kinetics-400	Google Drive

Model Training

To train TAGS from scratch run the following command. The training configurations can be adjusted from config/anet.yaml file.

python tags_train.py

Model Inference

We provide the pretrained models containing the checkpoint for I3D features on ActivityNetv1.3 . It can be found in the Link

After downloading the checkpoints, the checkpoints path can be saved in config/anet.yaml file. The model inference can be then performed using the following command

python tags_inference.py

Model Evaluation

To evaluate our TAGS model run the following command.

python eval.py

Performance

TO-DO Checklist

• Fix the multi-scale Transformer code in Snippet Embedding
• Support for THUMOS14 dataset
• Enable multi-gpu training

Citation

If you find this project useful for your research, please use the following BibTeX entry.

@article{nag2022temporal,
  title={Temporal Action Detection with Global Segmentation Mask Learning},
  author={Nag, Sauradip and Zhu, Xiatian and Song, Yi-Zhe and Xiang, Tao},
  journal={arXiv preprint arXiv:2207.06580},
  year={2022}
}
}