codes for ECCV 2016
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

#Structured Matching for Phrase Localization

Created by Mingzhe Wang at University of Michigan, Ann Arbor.

This is released code for:

Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, Jia Deng, Structured Matching for Phrase Localization, ECCV 2016. paper

To run this code, make sure the following are installed:

Getting Started

To test with our pretrained model, run:

cd workspace

Download our pretrained model:


Download features of test data:


To evaluate the model of bipartite matching:

cd ../src/lua

th test_matching.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching/. Open matlab and run evaluation code from Bryan A. Plummer et al:


To specify a test file, run with -file test-model. To evaluate the model of structured matching:

cd ../lua

th test_pc.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching_pc


In all experiments, we reported Recall@1 in our ECCV paper.

To train your own models, you need to extract features for both phrases and bounding boxes. You can also download our features (63G) with the following commands:

cd workspace/


To train a model with bipartite matching, in the src/lua folder, run:

th train_matching.lua

To train a model with structured matching, run:

th train_matching_pc.lua

Output models are stored in workspace/model. Please refer to the comments in train_matching_pc.lua to adjust learning parameters.

To generate your own features, you need to

  • Download Flickr30k dataset.
  • Download Flickr30kEntities dataset.
  • Generate bounding boxes.
  • Extract features for bounding boxes.
  • Extract features for phrases.

The following tools were used to extract features in our experiments:

Please let me know if you have any issues about our code.


Thanks to Geoff Leyland for providing an excellent implementation of simplex algorithms in lua.