Skip to content
Image Captions Generation with Spatial and Channel-wise Attention
Branch: master
Clone or download
1
Latest commit 1675c5f Apr 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cnn
data init commit Jan 13, 2017
exp init commit Jan 13, 2017
utils init commit Jan 13, 2017
.gitignore init commit Jan 13, 2017
README.md Update README.md Apr 4, 2018
requirements.txt init commit Jan 13, 2017
sca_resnet_branch2b.py init commit Jan 13, 2017

README.md

SCA-CNN

Source code for the paper: SCA-CNN: Spatial and Channel-wise Attention in Convolution Networks for Imgae Captioning

This code is based on arctic-captions and arctic-capgen-vid.

This code is only for two-layered attention model in ResNet-152 Network for MS COCO dataset. Other networks (VGG-19) or datasets (Flickr30k/Flickr8k) can also be used with minor modifications.

Dependencies

  • A python library: Theano.

  • Other python package dependencies like numpy/scipy, skimage, opencv, sklearn, hdf5 which can be installed by pip, or simply run

    $ pip install -r requirements.txt
    
  • Caffe for image CNN feature extraction. You should install caffe and building the pycaffe interface to extract the image CNN feature.

  • The official coco evaluation scrpits coco-caption for results evaluation. Install it by simply adding it into $PYTHONPATH.

Getting Started

  1. Get the code $ git clone the repo and install the dependencies

  2. Save the pretrained CNN weights Save the ResNet-152 weights pretrained on ImageNet. Before running the code, set the variable deploy and model in save_resnet_weight.py to your own path. Then run:

$ cd cnn
$ python save_resnet_weight.py
  1. Preprocessing the dataset For the preprocessing of captioning, we directly use the processed JSON blob from neuraltalk. Similar to step 2, set the PATH in cnn_until.py and make_coco.py to your own install path. Then run:
$ cd data
$ python make_coco.py
  1. Training The results are saved in the directory exp.
$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python sca_resnet_branch2b.py

Citation

If you find this code useful, please cite the following paper:

@inproceedings{chen2016sca,
  title={SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning},
  author={Chen, Long and Zhang, Hanwang and Xiao, Jun and Nie, Liqiang and Shao, Jian and Liu, Wei and Chua, Tat-Seng},
  booktitle={CVPR},
  year={2017}
}
You can’t perform that action at this time.