Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy

Table of Contents

1. Description
2. Dependencies
3. Manual
4. Data
5. Results
   1. Comparison on Youtube2Text
   2. Comparison on MSR-VTT
6. Results and Data for the Final Version of the Paper
7. Citation

---

Description

This repo contains the code of Semantics-Assisted Video Captioning Model, based on the paper "Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy". It is under review at Frontiers in Robotics and AI.

We propose three ways to improve the video captioning model. First of all, we utilize both spatial features and dynamic spatio-temporal features as inputs for semantic detection network in order to generate meaningful semantic features for videos. Then, we propose a scheduled sampling strategy which gradually transfers the training phase from a teacher guiding manner towards a more self teaching manner. At last, the ordinary logarithm probability loss function is leveraged by sentence length so that short sentence inclination is alleviated. Our model achieves state-of-the-art results on the Youtube2Text dataset and is competitive with the state-of-the-art models on the MSR-VTT dataset.

The overall structure of our model looks like this . Here is some captions generated by our model.

---

If you need a newer and more powerful model, please refer to Delving-Deeper-into-the-Decoder-for-Video-Captioning.

---

Dependencies

• Python3.6
• TensorFlow 1.13
• NumPy
• sklearn
• pycocoevalcap(Python3)

---

Manual

1. Make sure you have installed all the required packages.
2. Download pycocoevalcap and put it along with msrvttt, msvd, tagging folders.
3. Download files in the Data section.
4. cd path_to_directory_of_model; mkdir saves
5. run_model.sh is used for training models and test_model.sh is used for testing models. Specify the GPU you want to use by modifying CUDA_VISIBLE_DEVICES value. Specify the needed data paths by modifying corpus, ecores, tag and ref values. The words will be sampled by argmax strategy if argmax is 1 and they will be sampled by multinomial strategy if argmax is 0. name is the name which you give to the model. test refers to the path of the saved model which is to be tested. Do not give a parameter to test if you want to train a model.
6. After completing the configuration of the bash file, then bash run_model.sh for training, bash test_model.sh for testing.

---

Results

Comparison on Youtube2Text

Model	B-4	C	M	R	Overall
LSTM-E	45.3		31.0
h-RNN	49.9	65.8	32.6
aLSTMs	50.8	74.8	33.3
SCN	51.1	77.7	33.5
MTVC	54.5	92.4	36.0	72.8	0.9198
ECO	53.5	85.8	35.0
SibNet	54.2	88.2	34.8	71.7	0.8969
Our Model	61.8	103.0	37.8	76.8	1.0000

Comparison on MSR-VTT

Model	B-4	C	M	R	Overall
v2t_navigator	40.8	44.8	28.2	60.9	0.9325
Aalto	39.8	45.7	26.9	59.8	0.9157
VideoLAB	39.1	44.1	27.7	60.6	0.9140
MTVC	40.8	47.1	28.8	60.2	0.9459
CIDEnt-RL	40.5	51.7	28.4	61.4	0.9678
SibNet	40.9	47.5	27.5	60.2	0.9374
HACA	43.4	49.7	29.5	61.8	0.9856
TAMoE	42.2	48.9	29.4	62.0	0.9749
Our Model	43.8	51.4	28.9	62.4	0.9935

---

Data

MSVD

• MSVD tag index2word and word2index mappings(ExternalRepo)
  • We use the same word-index mapping in semantic tag to the code in this link.

MSRVTT

• MSR-VTT Dataset:

  • train_val_test_annotation.zip (GoogleDrive)
    • SHA-256: ce2d97dd82d03e018c6f9ee69c96eb784397d1c83f734fdb8c17aafa5e27da31
  • msr-vtt-v1.part1.rar (GoogleDrive)
    • SHA-256: 3445e0d1bffda3739110dfcf14182b63222731af8a4d7153f0ac09dbec39a0d3
  • msr-vtt-v1.part2.rar (GoogleDrive)
    • SHA-256: b550997526272ab68a42f1bd93315aa2bbb521c71f33d0cb922fbbfb86f15aae
  • msr-vtt-v1.part3.rar (GoogleDrive)
    • SHA-256: debbd0e535e77d9927ffb375299c08990519e22ba7dac542b464b70d440ef515

• Data and Models for both MSVD and MSR-VTT

  • data.zip
  • SHA-256: fadd721eaa0f13aff7c3505e4784a003514c33ffa5a934a9dcf13955285df11f

ECO

• Source Code: GitHub.
• ECO_full_kinetics.caffemodel (GoogleDrive)
  • MD5 31ed18d5eadfd59cb65b7dcdadc310b4
  • SHA-1 b749384d2dac102b8035965566e3030fce465c20

Results and Data for the Final Version of the Paper (Updating)

Results

1. MSVD Results

2. MSR-VTT Results

Data and Models

GoogleDrive

• SHA256: d2a731794ef1bc90c9ccd6c7fe5e92fa7ad104f9e9188ac751c984b23d3a939b

---

Citation

@ARTICLE{10.3389/frobt.2020.475767,
    AUTHOR={Chen, Haoran and Lin, Ke and Maye, Alexander and Li, Jianmin and Hu, Xiaolin},   
    TITLE={A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling},      
    JOURNAL={Frontiers in Robotics and AI},      
    VOLUME={7},      
    PAGES={129},     
    YEAR={2020},      
    URL={https://www.frontiersin.org/article/10.3389/frobt.2020.475767},       
    DOI={10.3389/frobt.2020.475767},      
    ISSN={2296-9144},   
    ABSTRACT={Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.}
}