Skip to content
Implementation of "Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning"
Branch: master
Clone or download
Latest commit 2618080 Jul 25, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
coco-caption First commit Mar 13, 2017
data First commit Mar 13, 2017
demo add figure Mar 17, 2017
image_model adding image model download script Mar 13, 2017
misc fix gradient for BN layer Mar 19, 2017
prepro First commit Mar 13, 2017
visu add figure Mar 17, 2017
LICENCE.md Create LICENCE.md Mar 13, 2017
README.md Update README.md Jul 24, 2017
demo.ipynb update demo Mar 17, 2017
eval_visulization.lua update demo Mar 17, 2017
train.lua Update train.lua May 2, 2017

README.md

AdaptiveAttention

Implementation of "Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning"

teaser results

Requirements

To train the model require GPU with 12GB Memory, if you do not have GPU, you can directly use the pretrained model for inference.

This code is written in Lua and requires Torch. The preprocssinng code is in Python, and you need to install NLTK if you want to use NLTK to tokenize the caption.

You also need to install the following package in order to sucessfully run the code.

Pretrained Model

The pre-trained model for COCO can be download here. The pre-trained model for Flickr30K can be download here.

Vocabulary File

Download the corresponding Vocabulary file for COCO and Flickr30k

Download Dataset

The first thing you need to do is to download the data and do some preprocessing. Head over to the data/ folder and run the correspodning ipython script. It will download, preprocess and generate coco_raw.json.

Download COCO and Flickr30k image dataset, extract the image and put under somewhere.

training a new model on MS COCO

First train the Language model without finetune the image.

th train.lua -batch_size 20 

When finetune the CNN, load the saved model and train for another 15~20 epoch.

th train.lua -batch_size 16 -startEpoch 21 -start_from 'model_id1_20.t7'

More Result about spatial attention and visual sentinel

teaser results

teaser results

For more visualization result, you can visit here (it will load more than 1000 image and their result...)

Reference

If you use this code as part of any published research, please acknowledge the following paper

@misc{Lu2017Adaptive,
author = {Lu, Jiasen and Xiong, Caiming and Parikh, Devi and Socher, Richard},
title = {Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning},
journal = {CVPR},
year = {2017}
}

Acknowledgement

This code is developed based on NeuralTalk2.

Thanks Torch team and Facebook ResNet implementation.

License

BSD 3-Clause License

You can’t perform that action at this time.