Skip to content
Review Network for Caption Generation
Lua Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
code_caption fix string indexing bug in code caption eval Jan 2, 2018
image_caption_offline add mscoco online Oct 26, 2016
image_caption_online Update Oct 27, 2016
.gitignore add extractor Sep 14, 2016
LICENSE Update LICENSE Oct 26, 2016

Review Network for Caption Generation

Image Captioning on MSCOCO

You can use the code in this repo to genearte a MSCOCO evaluation server submission with CIDEr=0.96+ with just a few hours.

No fine-tuning required. No fancy tricks. Just train three end-to-end review networks and do an ensemble.

  • Feature extraction: 2 hours in parallel
  • Single model training: 6 hours
  • Ensemble model training: 30 mins
  • Beam search for caption generation: 3 hours in parallel

Below is a comparison with other state-of-the-art systems (with according published papers) on the MSCOCO evaluation server:

Model BLEU-4 METEOR ROUGE-L CIDEr Fine-tuned Task specific features
Attention 0.537 0.322 0.654 0.893 No No
MS Research 0.567 0.331 0.662 0.925 No Yes
Google NIC 0.587 0.346 0.682 0.946 Yes No
Semantic Attention 0.599 0.335 0.682 0.958 No Yes
Review Net 0.597 0.347 0.686 0.969 No No

In the diretcory image_caption_online, you can use the code therein to reproduce our evaluation server results.

In the directory image_caption_offline, you can rerun experiments in our paper using offline evaluation.

Code Captioning

Predicting comments for a piece of source code is another interesting task. In the repo we also release a dataset with train/dev/test splits, along with the code of a review network.

Check out the directory code_caption.

Below is a comparison with baselines on the code captioning dataset:

Model LLH CS-1 CS-2 CS-3 CS-4 CS-5
LSTM Language Model -5.34 0.2340 0.2763 0.3000 0.3153 0.3290
Encoder-Decoder -5.25 0.2535 0.2976 0.3201 0.3367 0.3507
Encoder-Decoder (Bidir) -5.19 0.2632 0.3068 0.3290 0.3442 0.3570
Attentive Encoder-Decoder (Bidir) -5.14 0.2716 0.3152 0.3364 0.3523 0.3651
Review Net -5.06 0.2889 0.3361 0.3579 0.3731 0.3840


This repo contains the code and data used in the following paper:

Review Networks for Caption Generation

Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W. Cohen

NIPS 2016

You can’t perform that action at this time.