Skip to content
Source code for "Recurrent Fusion Network for Image Captioning".
Branch: master
Clone or download
Latest commit 3bf74f5 Nov 24, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cider first commit Oct 23, 2018
coco-caption first commit Oct 23, 2018
data Update readme.md Nov 24, 2018
fig add fig Oct 23, 2018
misc first commit Oct 23, 2018
scripts
.gitattributes first commit Oct 23, 2018
.gitignore first commit Oct 23, 2018
dataloader.py first commit Oct 23, 2018
eval.py first commit Oct 23, 2018
eval_ensemble.py first commit Oct 23, 2018
eval_ensemble.sh review script Oct 23, 2018
eval_single.sh review script Oct 23, 2018
eval_utils.py first commit Oct 23, 2018
feat_array.py first commit Oct 23, 2018
get_rewards.py first commit Oct 23, 2018
main.py first commit Oct 23, 2018
main_rl.py first commit Oct 23, 2018
models.py first commit Oct 23, 2018
my_optim.py first commit Oct 23, 2018
opts.py first commit Oct 23, 2018
readme.md revised fig link Oct 23, 2018
train.py first commit Oct 23, 2018
train_recurrent_fusion_model.sh review script Oct 23, 2018
train_recurrent_fusion_model_rl.sh review script Oct 23, 2018
train_review_net.sh review script Oct 23, 2018
train_review_net_rl.sh review script Oct 23, 2018
train_rl.py first commit Oct 23, 2018
train_single_recurrent_fusion_model.sh review script Oct 23, 2018
train_single_recurrent_fusion_model_rl.sh review script Oct 23, 2018

readme.md

Recurrent Fusion Network for Image Captioning

framework This repository includes the implementations for Recurrent Fusion Network for Image Captioning.

Requirements

  • Python 3.6
  • PyTorch 0.3.1
  • Java

Training

0. Feature extraction

All scripts for feature extraction are included in data/feature_extraction. Please generate flipped and cropped images to perform data augmentation, download pre-trained models and extract features with the scripts. All extracted featuers should be put in the data directory.

1. Train with cross entropy loss

bash train_recurrent_fusion_model.sh

2. Training with reinforcement learning

bash train_recurrent_fusion_model_rl.sh

Evaluation

Evaluate with eval_single.sh and eval_ensemble.sh to obtain metric scores for single model and ensemble of multiple models, respectively.

Reference

If you find this repo useful, please consider citing:

@InProceedings{Jiang_2018_ECCV,
author = {Jiang, Wenhao and Ma, Lin and Jiang, Yu-Gang and Liu, Wei and Zhang, Tong},
title = {Recurrent Fusion Network for Image Captioning},
booktitle = {The European Conference on Computer Vision (ECCV)},
month = {September},
year = {2018}
}

Acknowledgements

Our code is based on Ruotian Luo's implementation ans is reorganized by Zhiming Ma.

You can’t perform that action at this time.