Skip to content
The pytorch implementation on “Fine-Grained Image Captioning with Global-Local Discriminative Objective”
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
RL_TDA/CSGD/Result
cider/pyciderevalcap/ciderD
data
misc
models
scripts
.gitignore
CSGD_trainpro.py
LICENSE
MLE_opts.py
MLE_trainpro.py
README.md
RL_opts.py
coco_vocab.py
dataloader_test_random.py
evalpro.py
evalpro_recall_utils.py
evalpro_utils.py
generate_random_5000.py
knn_dataloader.py
self_retrieval.py
vse_model.py

README.md

Fine-Grained-Image-Captioning

The pytorch implementation for "Fine-Grained Image Captioning with Global-Local Discriminative Objective"

Requirements:

Download MSCOCO dataset

  • Download the coco images from http://cocodataset.org/#download. Download 2014 Train images and 2014 Val images, and put them into the train2014/ and val2014/ in the ./image. Download 2014 Test images, and put them into the test2014/

Download COCO captions and preprocess them

Pre-extract the image features

  • python scripts/prepro_feats.py --input_json data/dataset_coco.json --images_root image

Prepare for Reinforcement Learning

  • Download Cider from: https://github.com/vrama91/cider And put "ciderD_token.py" and "ciderD_scorer_token4.py" in the "cider/pyciderevalcap/ciderD/", then
  • python scripts/prepro_ngrams.py --input_json data/dataset_coco.json --dict_json data/cocotalk.json --output_pkl data/coco-train --split train

Prepare for training

Start training

Training using MLE criterion in the initial 20 epochs

  • python MLE_trainpro.py --id TDA --caption_model TDA --checkpoint_path RL_TDA

Training by Global-Local Discriminative Objective

Eval

  • python evalpro.py --caption_model TDA --checkpoint_path RL_TDA

Self-retrieval Experiment

  • python generate_random_5000.py --caption_model TDA --checkpoint_path RL_TDA
  • python self_retrieval.py --id TDA --caption_model TDA --checkpoint_path RL_TDA
You can’t perform that action at this time.