Code release for "Spatiotemporal Residual Networks for Video Action Recognition", NIPS16 & "Spatiotemporal Multiplier Networks for Video Action Recognition", CVPR17.
Switch branches/tags
Nothing to show
Clone or download
Latest commit 8b4f284 Sep 13, 2017
Permalink
Failed to load latest commit information.
base_streams update to newest mcn Sep 13, 2017
hmdb51_splits init Jul 25, 2017
matconvnet @ 7543221 updated mcn Sep 13, 2017
ucf101_splits init Jul 25, 2017
.gitmodules init Jul 25, 2017
README.md Update README.md Jul 25, 2017
STMuNet_test.m add test Sep 13, 2017
STMulNet.m fix Sep 13, 2017
STResNet_stage1.m init Jul 25, 2017
STResNet_stage2.m init Jul 25, 2017
STResNet_test.m init Jul 25, 2017
cnn_get_im_flow_batch.m update to newest mcn Sep 13, 2017
cnn_setup_data.m init Jul 25, 2017
cnn_setup_environment.m add test Sep 13, 2017
cnn_train_dag.m update to newest mcn Sep 13, 2017
compile.m init Jul 25, 2017
getBatchWrapper_rgbflow.m init Jul 25, 2017

README.md

================================================================================

Spatiotemporal Residual Networks for Video Action Recognition

This repository contains the code for our NIPS'16 and CVPR'17 papers:

Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes
"Spatiotemporal Residual Networks for Video Action Recognition"
in Proc. NIPS 2016

Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes
"Spatiotemporal Multiplier Networks for Video Action Recognition"
in Proc. CVPR 2017

If you use our code/models/data for your research, please cite our papers:

    @inproceedings{feichtenhofer2016spatiotemporal,
      title={Spatiotemporal residual networks for video action recognition},
      author={Feichtenhofer, Christoph and Pinz, Axel and Wildes, Richard},
      booktitle={Advances in Neural Information Processing Systems (NIPS)},
      pages={3468--3476},
      year={2016}
    }

    @inproceedings{feichtenhofer2017multiplier,
      title={Spatiotemporal multiplier networks for video action recognition},
      author={Feichtenhofer, Christoph and Pinz, Axel and Wildes, Richard P}
      booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2017}
    }

Requirements

The code was tested on Ubuntu 14.04, 16.04 and Windows 10 using MATLAB R2016b and NVIDIA Titan X GPUs.

If you have questions regarding the implementation please contact:

Christoph Feichtenhofer <feichtenhofer AT tugraz.at>

================================================================================

Setup

  1. Download the code git clone --recursive https://github.com/feichtenhofer/st-resnet

  2. Compile the code by running compile.m.

  3. Edit the file cnn_setup_environment.m to adjust the models and data paths.

  4. (Optional) Download pretrained model files and the datasets, linked below and unpack them into your models/data directory. Otherwise the scripts will attempt to download the models at runtime.

  • Optionally you can also pretrain your own twostream base models by running
    1. base_streams/cnn_ucf101_spatial(); to train the appearance network streams.
    2. base_streams/cnn_ucf101_temporal(); to train the optical flow network streams.
  1. Training STResNet_stage1();, STResNet_stage2(); to train the architecture in our NIPS 2016 paper. STMulNet(); to train the architecture in our CVPR 2017 paper.
    • In case you did not download or trained the base models, the script will attempt to download these accordingly.
    • In case you would like to train on the CPU, clear the variable opts.train.gpus
    • In case you encounter memory issues on your GPU, consider decreasing the cudnnWorkspaceLimit (512MB is default)

Models: ST-ResNet

  • Download final models here: (in case you do not, STResNet_stage1();, STResNet_stage2();, and STResNet_test(); will attempt to download the respective models at runtime.)

Models: ST-MulNet

  • Download final models here: (in case you do not, STMulNet(); and STMulNet_test(); will attempt to download the models at runtime.)

Data

Pre-computed optical flow images and resized rgb frames for the UCF101 and HMDB51 datasets

Use it on your own dataset