Skip to content


Repository files navigation

Deep semantic-visual embedding with localization

Training and evaluation code for the paper Finding beans in burgers: Deep semantic-visual embedding with localization

This code allow training of new model, reproduction of experiments, as well as features extraction for both images and texts.

Author and contact: Martin Engilberge

Main dependencies

This code is written in python. To use it you will need:

  • Python 3.7
  • Pytorch 1.0
  • SRU[cuda]
  • Numpy
  • Scipy
  • Torchvision
  • Ms Coco API (pycocotools)
  • Visual Genome API
  • NLTK
  • opencv

An environment file for conda is available in the repository (environment.yml).

Getting started

You will first need to set the paths to the datasets and word embedding in the file misc/ Commentaries in the config file contains links where you can download the data.

To train and run model you will need:

To reproduce experiments in the paper:

Once the required paths have been set in the config file you can start training models using the following command:


By default all the scripts use gpu, you can switch to cpu mode by uncommenting device = torch.device("cpu") at the beginning of the script.

Model evaluation

Models can be evaluated on three tasks:

  • cross modal retrieval:
python -p "path/to/model/model.pth.tar" -te
  • pointing game:
python -p "path/to/model/model.pth.tar"
  • semantic segmentation:
python -p "path/to/model/model.pth.tar"

Features extraction

The features space produced by the joint embedding manages to capture semantic property. Two scripts can be used to extract feature from that space for images and texts.

For images the script takes a folder as input and produces the embedding representation for all the jpeg images in the folder.

python -p "path/to/model/model.pth.tar" -d "path/to/image/folder/" -o "path/to/output/file"

For text the script takes a text file and produces the embedding representation for each line.

python -p "path/to/model/model.pth.tar" -d "path/to/text/file/" -o "path/to/output/file"


If you found this code useful, please cite the following paper:

  title={Finding beans in burgers: Deep semantic-visual embedding with localization},
  author={Engilberge, Martin and Chevallier, Louis and P{\'e}rez, Patrick and Cord, Matthieu},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},


by downloading this program, you commit to comply with the license as stated in the file.


Deep semantic-visual embedding with localization







No releases published


No packages published