Skip to content
No description, website, or topics provided.
Python Shell
Branch: master
Clone or download
Latest commit 03f106f May 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md Update README.md May 3, 2018
dataset_utils.py Create dataset_utils.py Dec 13, 2017
eval_embedding_nn.py Create eval_embedding_nn.py Dec 13, 2017
retrieval_model.py Create retrieval_model.py Dec 13, 2017
run_two_branch.sh Create run_two_branch.sh Dec 13, 2017
train_embedding_nn.py Create train_embedding_nn.py Dec 13, 2017

README.md

Two-Branch Neural Networks

Usage:

  • Train a model from scratch: Set the path to the training dataset and checkpoint save directory in run_two_branch.sh. Run sh run_two_branch.sh --train.
  • Evaluate a model on the validation set while training: Set the path to validation dataset and checkpoint save directory. Additionally, adjust the time interval between each evaluation in eval_embedding.py. By default, the script will evaluate once per minute on the newest checkpoint in the given checkpoint directory. Run sh run_two_branch.sh --val.
  • Evaluate a model on a specific checkpoint: Set the path to the test dataset and checkpoint MetaGraph (.meta file). Run sh run_two_branch.sh --test.
  • Use a pre-trained model: Download checkpoints from the URLs below. Follow the instruction for evaluating model on a specific checkpoint.

Any questions, feel free to contact me, lwang97@illinois.edu

Dataset:

Due to the size of the features (~17G for MSCOCO and 7G for Flickr30K), only the test split is available for download.

Data set splits:

  • Train, and test splits (the left is used for validation), please check readme.txt download

To extract images and sentences features,

  • For image features, we use the VGG 19 network to extract features.
  • For sentence features, we are using the code from "B. Klein, G. Lev, G. Sadeh, L. Wolf. Associating Neural Word Embeddings With Deep Image Representations Using Fisher Vectors (CVPR 2015)". Their code can be downloaded here.

Pre-trained models:

If you find our code helpful, please cite our Two-Branch Network Papers:

References

@inproceedings{wang2016learning,
title={Learning deep structure-preserving image-text embeddings},
author={Wang, Liwei and Li, Yin and Lazebnik, Svetlana},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={5005--5013},
year={2016}
}

@article{wang2017learning,
title={Learning Two-Branch Neural Networks for Image-Text Matching Tasks},
author={Wang, Liwei and Li, Yin and Lazebnik, Svetlana},
journal={arXiv preprint arXiv:1704.03470},
year={2017}
}
You can’t perform that action at this time.