Skip to content
ICCV2019: Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
pos_src pos generator Oct 16, 2019
.DS_Store change Oct 16, 2019

Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network


In this paper, we propose to guide the video caption generation with POS information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. This code is a Pytorch implement of this work. image


  • Python 2.7
  • Pytorch 0.3.1.post3
  • Cuda 8.0
  • Cudnn 7.0.5


  1. Download Inception_ResNet_V2 features of MSRVTT-10K RGB frames and I3D features of MSRVTT-10K optical flows, and put them in datas folder.
  2. Download pre-trained models, and put them in results folder.
  3. Download the automatic evaluation metrics -- coco-caption, and link it to caption_src/ as well as pos_src/.
ln -s coco-caption caption_src/coco-caption
ln -s coco-caption pos_src/coco-caption
  1. Finally, the document structure of the root path should be like this: image


We provide the pre-trained models of "Ours(IR+M)" and "Ours_RL(IR+M)" in paper to reproduce the result reported in paper. Users can change the command in to reproduce "Ours(IR+M)" or "Ours_RL(IR+M)".

Metrics Ours(IR_M) Ours_RL(IR+M)
BLEU@1 0.7875 0.8175
BLEU@2 0.6601 0.6788
BLEU@3 0.5339 0.5376
BLEU@4 0.4194 0.4128
METEOR 0.2819 0.2869
ROUGE-L 0.6161 0.6210
CIDEr 0.4866 0.5337
cd caption_src/


Actually, training in this repository is divided into two steps:

  1. Train a global pos generator and extract the global postag features.
cd pos_src/

After early stopping, extract and store the postag features in pos_src/globalpos_features/xxx.hdf5, where xxx.hdf5 can be customized at line36 of pos_src/


Rember to copy the postag features hdf5 into datas/.

  1. Train the caption model.
cd caption_src/


If you use our code in your research or wish to refer to the baseline results, please use the use the following BibTeX entry.

  title={Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network},
  author={Wang, Bairui and Ma, Lin and Zhang, Wei and Jiang, Wenhao and Wang, Jingwen and Liu, Wei},
  journal={arXiv preprint arXiv:1908.10072},


Special thanks to Ruotian Luo, as our codes about Self-critical Sequence Training was inspired by and references to his repository.

You can’t perform that action at this time.