Localize objects in images using referring expressions
Switch branches/tags
Nothing to show
Clone or download
Latest commit b138c0f Nov 1, 2016
Failed to load latest commit information.
Google_Refexp_toolbox @ 675983c Initial commit Aug 6, 2016
caffe @ 4a71e88 Initial commit Aug 6, 2016
coco @ 2934299 Initial commit Aug 6, 2016
lib Initial commit Aug 6, 2016
proto_files Initial commit Aug 6, 2016
.gitmodules Initial commit Aug 6, 2016
Readme.md Update Readme Nov 1, 2016


Localizing objects using referring expressions

This repository contains code for detecting objects in images mentioned by referring expressions. The code is an implementation of the technique presented in our paper. We have also included links to pretrained models, our split of the Google RefExp dataset and also the processed version of the UNC RefExp dataset.

  title={Modeling Context Between Objects for Referring Expression Understanding},
  author={Varun K. Nagaraja and Vlad I. Morariu and Larry S. Davis},

We have also implemented the baseline and max-margin techniques proposed by Mao et al. in their CVPR 2016 paper. If you use the Google RefExp dataset, please cite this paper

  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin},

If you use the UNC RefExp dataset, please cite the following paper

  title={Modeling Context in Referring Expressions},
  author={Licheng Yu and Patric Poirson and Shan Yang and Alexander C. Berg and Tamara L. Berg},


Clone this repository
git clone --recursive https://github.com/varun-nagaraja/referring-expressions.git

We will call the directory that you cloned this repository into as $RefExp_ROOT

Build external components
  • Build caffe and pycaffe. The instructions for installing caffe are here.

     cd $RefExp_ROOT/caffe
     make -j8 && make pycaffe
  • Download VGGnet

     mkdir $RefExp_ROOT/caffe/models/vggnet
     cd $RefExp_ROOT/caffe/models/vggnet
     wget https://gist.githubusercontent.com/ksimonyan/211839e770f7b538e2d8/raw/0067c9b32f60362c74f4c445a080beed06b07eb3/VGG_ILSVRC_16_layers_deploy.prototxt
     wget http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel
  • Build COCO toolbox

     cd $RefExp_ROOT/coco/PythonAPI
Download the datasets
  • The MSCOCO dataset can be download from here. You will need the training images, validation images and object instance annotations.

    Our code uses the following directory structure

     ├── annotations
     │   ├── instances_train2014.json
     │   └── instances_val2014.json
     ├── images
     │   ├── train2014
     │   └── val2014
     ├── google_refexp
     └── unc_refexp
  • Download Google RefExp dataset with our split and MCG region candidates

     cd $COCO_PATH
     wget https://obj.umiacs.umd.edu/referring-expressions/google_refexp_umd_split.tar.gz
     tar -xzf google_refexp_umd_split.tar.gz
     rm google_refexp_umd_split.tar.gz

    Note: If you want the original split of the Google RefExp dataset, follow the instructions at this link. Then move the dataset files to the appropriate folder as indicated above.

  • Download UNC RefExp dataset with MCG candidates

     cd $COCO_PATH
     wget https://obj.umiacs.umd.edu/referring-expressions/unc_refexp.tar.gz 
     tar -xzf unc_refexp.tar.gz
     rm unc_refexp.tar.gz


  • Create cache directories where we will store the model and vocabulary files

     python lib/experiment_settings.py --coco_path $COCO_PATH
  • Download pre-trained vocabulary and model files

     cd $COCO_PATH/cache_dir
     cd h5_data
     wget https://obj.umiacs.umd.edu/referring-expressions/Google_RefExp_vocabulary.txt
     wget https://obj.umiacs.umd.edu/referring-expressions/UNC_RefExp_vocabulary.txt
     cd ..
     cd models
     # baseline models trained on Google RefExp and UNC RefExp
     wget https://obj.umiacs.umd.edu/referring-expressions/baseline_models.tar.gz
     tar -xzf baseline_models.tar.gz
     rm baseline_models.tar.gz
     # max-margin models
     wget https://obj.umiacs.umd.edu/referring-expressions/max_margin_models.tar.gz
     tar -xzf max_margin_models.tar.gz
     rm max_margin_models.tar.gz
     # context models with negative bag margin
     wget https://obj.umiacs.umd.edu/referring-expressions/mil_context_withNegMargin_models.tar.gz
     tar -xzf mil_context_withNegMargin_models.tar.gz
     rm mil_context_withNegMargin_models.tar.gz
     # context models with positive bag margin
     wget https://obj.umiacs.umd.edu/referring-expressions/mil_context_withPosNegMargin_models.tar.gz
     tar -xzf mil_context_withPosNegMargin_models.tar.gz
     rm mil_context_withPosNegMargin_models.tar.gz

    Note: In the paper, for Google RefExp experiments, we report numbers from models trained on a subset of the training set since we use the remaining training set for validation. However, these pretrained models were trained on the entire training set and hence provide slighty better numbers than those reported in the paper.

  • Evaluate on a dataset split

     python lib/comprehension_experiments.py --coco_path $COCO_PATH --dataset Google_RefExp --exp_name baseline --split_name val --proposal_source gt

    Use --visualize option to pause at every image and display localization results


  • Create files for training

     python lib/process_dataset.py --coco_path $COCO_PATH --dataset Google_RefExp --exp_name baseline

    This will first extract region features for all images in the dataset and dump them in a format suitable for loading in caffe. It will require a lot of space on disk depending on the experiment type you want to run.

  • If you are working with Google RefExp, we will split the training data to create a validation partition of our own. The test set of the Google RefExp dataset is not yet released.

     cd $COCO_PATH/cache_dir/h5_data/buffer_16/Google_RefExp_baseline_20
     head -n 5038 hdf5_chunk_list.txt > hdf5_chunk_list_part1.txt
     tail -n 300 hdf5_chunk_list.txt > hdf5_chunk_list_part2.txt
  • Edit training prototxt file (Ex.: proto_files/google_refexp/google_refexp.baseline.prototxt) and set the correct source in hdf5_data_param. For example,

     hdf5_data_param {
         source: "$COCO_PATH/cache_dir/h5_data/buffer_16/Google_RefExp_train_baseline_20/hdf5_chunk_list_part1.txt"
         batch_size: 16
  • Edit solver.prototxt file (Ex.: proto_files/google_refexp/google_refexp.baseline.solver.prototxt) and set snapshot_prefix appropriately. For example,

     snapshot_prefix: "$COCO_PATH/cache_dir/models/Google_RefExp_baseline/google_refexp.baseline"

    Important Create the required model directory (Ex.: $COCO_PATH/cache_dir/models/Google_RefExp_baseline)

  • Edit training script to use the experiment name you are interested in. Then run the script from the $RefExp_ROOT directory.


    The script will print the log to the screen and also write to a file.

  • When the training is complete, choose the iteration number of the model snapshot with the lowest cross entropy loss on the validation set. Set this iteration number in lib/experiment_settings.py file to test the trained model. The following command will extract the lines which contain the cross entropy loss.

     grep "Testing net (#1)" -A 4 google_refexp.baseline.log