Skip to content


Folders and files

Last commit message
Last commit date

Latest commit


Repository files navigation

T-CNN: Tubelets with Convolution Neural Networks


The TCNN framework is a deep learning framework for object detection in videos. This framework was orginally designed for the ImageNet VID chellenge in ILSVRC2015.

Citing T-CNN

If you are using the T-CNN code in you project, please cite the following works.

  Title = {Object Detection from Video Tubelets with Convolutional Neural Networks},
  Author = {Kang, Kai and Ouyang, Wanli and Li, Hongsheng and Wang, Xiaogang},
  Booktitle = {CVPR},
  Year = {2016}
  title={T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos},
  author={Kang, Kai and Li, Hongsheng and Yan, Junjie and Zeng, Xingyu and Yang, Bin and Xiao, Tong and Zhang, Cong and Wang, Zhe and Wang, Ruohui and Wang, Xiaogang and Ouyang, Wanli},
  journal={arXiv preprint},


T-CNN is released under the MIT License.

ImageNet 2015 VID detection results

Track Validation Set Test Set Rank in ILSVRC2015
Provided 73.8 67.8 #1
Additional 77.0 69.7 #2



  1. caffe with Python layer and pycaffe
  2. GNU Parallel
  3. matutils
  4. FCN tracker
  5. Matlab with python engine


  1. Clone the repository and sub-repositories from GitHub, let $TCNN_ROOT represents the root directory of the repository.

        $ # clone the repository
        $ git clone --recursive
        $ cd $TCNN_ROOT
        $ # checkout the ImageNet 2015 VID branch
        $ git checkout ilsvrc2015vid
  2. Compilation for vdetlib

        $ cd $TCNN_ROOT/vdetlib
        $ make
        $ export PYTHONPATH=$TCNN_ROOT/vdetlib:$PYTHONPATH
  3. Download and install caffe in the External directory

        $ git clone External/caffe
        $ # modify `Makefile.config` and build with Python layer and pycaffe
        $ # detailed instruction, please follow
        $ export PYTHONPATH=$TCNN_ROOT/External/caffe/python:$PYTHONPATH
  4. Download a modified version of FCN Tracker originally developed by Lijun Wang et. al.

        $ git clone --recursive -b T-CNN External/fcn_tracker_matlab
        $ # compile the caffe-fcn_tracking and configure FCNT


  1. Extract the sample data and still-image detection results

        $ cd $TCNN_ROOT
        $ unzip -d data/
  2. Generate optical flow for the videos

        $ mkdir ./data/opt_flow
        $ ls ./data/frames |
            parallel python tools/data_proc/ ./data/frames/{} ./data/opt_flow/{} --merge
  3. Multi-context suppression and motion-guided propagation in Matlab

        >> addpath(genpath('tools/mcs_mgp'));
        >> mcs_mgp('data/opt_flow', 'data/scores', 'data/mcs_mgp')
  4. Tubelet tracking and re-scoring

        $ # generate .vid protocol files
        $ ls data/frames | parallel python vdetlib/tools/ {} $PWD/data/frames/{} data/vids/{}.vid
        $ # tracking from raw detection files
        $ find data/vids -type f -name *.vid | parallel -j1 python tools/tracking/ {} data/mcs_mgp/window_size_7_time_step_1_top_ratio_0.000300_top_bonus_0.400000_optflow/{/.} data/tracks/{/.} --thres 3.15 --max_frames 100 --num 30
        $ # spatial max-pooling
        $ find data/vids -type f | parallel python tools/scoring/ {} data/tracks/{/.} data/mcs_mgp/window_size_7_time_step_1_top_ratio_0.000300_top_bonus_0.400000_optflow/{/.} data/score_proto/window_size_7_time_step_1_top_ratio_0.000300_top_bonus_0.400000_optflow_max_pooling/{/.} --overlap_thres 0.5
  5. Tubelet visualization

        $ python tools/visual/ data/vids/ILSVRC2015_val_00007011.vid data/score_proto/window_size_7_time_step_1_top_ratio_0.000300_top_bonus_0.400000_optflow_max_pooling/ILSVRC2015_val_00007011/ILSVRC2015_val_00007011.airplane.score

Beyond demo

  1. Optical flow extraction

        $ python tools data_proc/ -h
  2. vdetlib for tracking and rescoring

  3. Visualization tools in tools/visual.

Known Issues

  1. Matlab engines may stall after long periods of tracking. Please consider to kill the certain matlab session to continue.

To-do list

  • Tubelet Bayesian classifier