Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

SCN for video captioning

This repo contains the code of using SCN for video captioning, based on the CVPR 2017 paper “Semantic Compositional Networks for Visual Captioning”.

To keep things simple, SCN for image captioning is provided in another separate repo.


This code is written in python. To use it you will need:

  • Python 2.7 (do not use Python 3.0)
  • Theano 0.7 (you can also use the most recent version)
  • A recent version of NumPy and SciPy

Getting started

We provide the code on how to train SCN for video captioning on the Youtube2Text dataset. The SCN used in this experiment is a slightly different version of the original SCN. That is, we feed the video features to each step of the SCN-LSTM, instead of only the first step.

  • In order to start, please first download the C3D, ResNet features and tag features for the Youtube2Text dataset we used in the experiments. Put the youtube2text folder inside the ./data folder.

  • We also provide our pre-trained model on Youtube2Text. Put the pretrained_model folder into the current directory.

  • In order to evaluate the model, please download the standard coco-caption evaluation code. Copy the folder pycocoevalcap into the current directory.

  • Now, everything is ready.

How to use the code

  1. Run to start training.
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python 
  1. Based on our pre-trained model, run to generate captions on the Youtube2Text test set. The generated captions are also provided, named youtube2text_scn_test.txt.

  2. Now, run to evaluate the model. The code will output

CIDEr: 0.777, Bleu-4: 0.511, Bleu-3: 0.606, Bleu-2: 0.697, Bleu-1: 0.810, ROUGE_L: 0.706, METEOR: 0.335. 

Train the video tagger

The above results depend on training a good video tagging model. Below, we introduce how we implement this in our paper. In our experiments, in order to train the video tagger, we use data both from Youtube2Text and MSR-VTT. The code is inside the video_tagging folder.

  1. In order to start, please first download the ResNet features and captions for the Youtube2Text and MSR-VTT dataset we used in the experiments. They are included in the data folder. Further, inside the preprocess_raw_data folder, we also include how we preprocess the raw data into the format we used in running the code.

  2. Run to obtain the ground-truth 300 tags for the Youtube2Text dataset.

  3. Run to obtain the ground-truth 300 tags for the MSR-VTT dataset. This provides us additional data to train a more robust tagger, compared with using Youtube2Text alone.

  4. Run to train the video tagger for the Youtube2Text dataset.

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python 
  1. The trained model and the resulting predicted tag features are also provided in the above folder, i.e., youtube_tagging_learned_params.npz and tag_feats_pred.mat. The latter is the tag feats we used in running the video captioning code.

  2. Now, go back to run to train the video captioning model.

Citing SCN

Please cite our CVPR paper in your publications if it helps your research:

  Author = {Gan, Zhe and Gan, Chuang and He, Xiaodong and Pu, Yunchen and Tran, Kenneth and Gao, Jianfeng and Carin, Lawrence and Deng, Li},
  Title = {Semantic Compositional Networks for Visual Captioning},
  Year  = {2017}


Using Semantic Compositional Networks for Video Captioning



No releases published


No packages published