MiniROAD: Minimal RNN Framework for Online Action Detection

Introduction

This is a pytorch implementation for our ICCV 2023 paper "MiniROAD: Minimal RNN Framework for Online Action Detection".

Data Preparation

THUMOS14 and TVSeries

To prepare the features and targets by yourself, please refer to LSTR. You can also directly download the pre-extracted features and targets from TeSTra.

FineAction

Download the officially available pre-extracted features from FineAction. As mentioned in the paper, the temporal dimensions have been linearly interpolated by a factor of four as the officially available feature is too condensed (16 frames being converted into one feature).

Data Structure

1. If you want to use our dataloaders, please make sure to put the files as the following structure:

   • THUMOS'14 dataset:

     $YOUR_PATH_TO_THUMOS_DATASET
     ├── rgb_FEATURETYPE/
     |   ├── video_validation_0000051.npy 
     │   ├── ...
     ├── flow_FEATURETYPE/ 
     |   ├── video_validation_0000051.npy 
     |   ├── ...
     ├── target_perframe/
     |   ├── video_validation_0000051.npy (of size L x 22)
     |   ├── ...
     

   • TVSeries dataset:

     $YOUR_PATH_TO_TVSERIES_DATASET
     ├── rgb_FEATURETYPE/
     |   ├── Breaking_Bad_ep1.npy 
     │   ├── ...
     ├── flow_FEATURETYPE/
     |   ├── Breaking_Bad_ep1.npy 
     |   ├── ...
     ├── target_perframe/
     |   ├── Breaking_Bad_ep1.npy (of size L x 31)
     |   ├── ...
     

   • FineAction dataset:

     $YOUR_PATH_TO_FINEACTION_DATASET
     ├── rgb_kinetics_i3d/
     |   ├── v_00008645.npy (of size L x 2048)
     │   ├── ...
     ├── flow_kinetics_i3d/
     |   ├── v_00008645.npy (of size L x 2048)
     |   ├── ...
     ├── target_perframe/
     |   ├── v_00008645.npy (of size L x 107)
     |   ├── ...
     

   For appropriate FEATURETYPE, please refer to (datasets/dataset.py)

2. Create softlinks of datasets:

   cd MiniROAD
   ln -s $YOUR_PATH_TO_THUMOS_DATASET data/THUMOS
   ln -s $YOUR_PATH_TO_TVSERIES_DATASET data/TVSERIES
   ln -s $YOUR_PATH_TO_FINEACTION_DATASET data/FINEACTION
   

Training

```
cd MiniROAD
python main.py --config $PATH_TO_CONFIG_FILE 
```

Inference from checkpoint

```
cd MiniROAD
python main.py --config $PATH_TO_CONFIG_FILE --eval $PATH_TO_CHECKPOINT
```

Main Results and checkpoints

THUMOS14

method	feature	mAP (%)	config	checkpoint
MiniROAD	kinetics	71.8	yaml	Download
MiniROAD	nv_kinetics	68.4	yaml	Download

FINEACTION

method	feature	mAP (%)	config	checkpoint
MiniROAD	kinetics	37.1	yaml	Download

TVSERIES

method	feature	mcAP (%)	config	checkpoint
MiniROAD	kinetics	89.6	yaml	Download

Citations

If you are using the data/code/model provided here in a publication, please cite our paper:

@inproceedings{miniroad,
	title={MiniROAD: Minimal RNN Framework for Online Action Detection},
	author={An, Joungbin and Kang, Hyolim and Han, Su Ho and Yang, Ming-Hsuan and Kim, Seon Joo},
	booktitle={International Conference on Computer Vision (ICCV)},
	year={2023}
}

License

This project is licensed under the Apache-2.0 License.

Acknowledgements

Many of the codebase is from LSTR.