Dual-Path Convolutional Image-Text Embedding https://arxiv.org/abs/1711.05535
Switch branches/tags
Nothing to show
Clone or download
Latest commit 32105e4 Sep 21, 2018
Failed to load latest commit information.
data add train code Nov 23, 2017
dataset Update README.md Jun 6, 2018
examples add imagenet preprocess Dec 22, 2017
matlab Update Layer.m Jan 21, 2018
test Create README.md Dec 24, 2017
test_coco Update extract_pic_feature_word2_plus.m Nov 24, 2017
test_cuhk Update find_pic_feature_word2_plus.m Nov 24, 2017
word2vector_matlab Create README.md Nov 22, 2017
CUHK-show.jpg Add files via upload Sep 20, 2018
LICENSE Initial commit Nov 17, 2017
README.md Update README.md Sep 20, 2018
coco_word2_Rankloss.m add coco Nov 24, 2017
coco_word2_Rankloss_vgg19.m add coco Nov 23, 2017
coco_word2_pool.m Update coco_word2_pool.m Nov 24, 2017
coco_word2_pool_vgg19.m Update coco_word2_pool_vgg19.m Nov 24, 2017
concat_2net.m add train code Nov 23, 2017
cuhk_word2_Rankloss.m add train code Nov 23, 2017
cuhk_word2_Rankloss_vgg16.m add train code Nov 23, 2017
cuhk_word2_pool.m Update cuhk_word2_pool.m Nov 24, 2017
cuhk_word2_pool_vgg16.m Update cuhk_word2_pool_vgg16.m Nov 24, 2017
gpu_compile.m add train code Nov 23, 2017
rand_diff_class.m add train code Nov 23, 2017
rand_diff_class2.m add train code Nov 23, 2017
rand_diff_class3.m add train code Nov 23, 2017
rand_same_class.m add train code Nov 23, 2017
rand_same_class_coco.m add coco Nov 23, 2017
resnet52_new_hope_word2_pool.m Update resnet52_new_hope_word2_pool.m Nov 23, 2017
resnet52_new_hope_word2_pool_vgg19.m Update resnet52_new_hope_word2_pool_vgg19.m Nov 23, 2017
resnet52_new_hope_word_Rankloss.m add train code Nov 23, 2017
resnet52_new_hope_word_Rankloss_vgg19.m add train code Nov 23, 2017
train_coco_Rankloss_shift_hard.m Update train_coco_Rankloss_shift_hard.m Nov 24, 2017
train_coco_Rankloss_shift_hard_vgg19.m Update train_coco_Rankloss_shift_hard_vgg19.m Nov 24, 2017
train_coco_word2_1_pool.m Update train_coco_word2_1_pool.m Nov 24, 2017
train_coco_word2_1_pool_vgg19.m Update train_coco_word2_1_pool_vgg19.m Nov 24, 2017
train_cuhk_Rankloss_shift.m Update train_cuhk_Rankloss_shift.m Nov 24, 2017
train_cuhk_Rankloss_shift_vgg16.m Update train_cuhk_Rankloss_shift_vgg16.m Nov 24, 2017
train_cuhk_word2_1_pool.m Update train_cuhk_word2_1_pool.m Nov 24, 2017
train_cuhk_word2_1_pool_vgg16.m Update train_cuhk_word2_1_pool_vgg16.m Nov 24, 2017
train_flickr_word2_1_pool.m Update train_flickr_word2_1_pool.m Nov 24, 2017
train_flickr_word2_1_pool_vgg19.m Update train_flickr_word2_1_pool_vgg19.m Nov 24, 2017
train_flickr_word_Rankloss_shift_hard.m Update train_flickr_word_Rankloss_shift_hard.m Nov 24, 2017
train_flickr_word_Rankloss_shift_hard_vgg19.m Update train_flickr_word_Rankloss_shift_hard_vgg19.m Nov 24, 2017


Dual-Path Convolutional Image-Text Embedding

This repository contains the code for our paper Dual-Path Convolutional Image-Text Embedding. Thank you for your kindly attention.

What's New: We updated the paper to the second version, adding more illustration about the mechanism of the proposed instance loss.

Prepocess Datasets

  1. Extract wrod2vec weights. Follow the instruction in ./word2vector_matlab;

  2. Prepocess the dataset. Follow the instruction in ./dataset. You can choose one dataset to run. Three datasets need different prepocessing. I write the instruction for Flickr30k, MSCOCO and CUHK-PEDES.

  3. Download the model pre-trained on ImageNet. And put the model into './data'.

(bash) wget http://www.vlfeat.org/matconvnet/models/imagenet-resnet-50-dag.mat

Alternatively, you may try VGG16 or VGG19.

Trained Model

You may download the three trained models from GoogleDrive.


  • For Flickr30k, run train_flickr_word2_1_pool.m for Stage I training.

Run train_flickr_word_Rankloss_shift_hard for Stage II training.

  • For MSCOCO, run train_coco_word2_1_pool.m for Stage I training.

Run train_coco_Rankloss_shift_hard.m for Stage II training.

  • For CUHK-PEDES, run train_cuhk_word2_1_pool.m for Stage I training.

Run train_cuhk_word_Rankloss_shift for Stage II training.


Select one model and have fun!

  • For Flickr30k, run test/extract_pic_feature_word2_plus_52.m and to extract the feature from image and text. Note that you need to change the model path in the code.

  • For MSCOCO, run test_coco/extract_pic_feature_word2_plus.m and to extract the feature from image and text. Note that you need to change the model path in the code.

  • For CUHK-PEDES, run test_cuhk/extract_pic_feature_word2_plus_52.m and to extract the feature from image and text. Note that you need to change the model path in the code.


  • Get word2vec weight

  • Data Preparation (Flickr30k)

  • Train on Flickr30k

  • Test on Flickr30k

  • Data Preparation (MSCOCO)

  • Train on MSCOCO

  • Test on MSCOCO

  • Data Preparation (CUHK-PEDES)

  • Train on CUHK-PEDES

  • Test on CUHK-PEDES

  • Run the code on another machine