Deep semantic-visual embedding with localization
Branch: master
Clone or download
Latest commit 5ac730b Dec 13, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data first commit Dec 13, 2018
logs first commit Dec 13, 2018
misc first commit Dec 13, 2018
weights first commit Dec 13, 2018
LICENSE.md first commit Dec 13, 2018
README.md first commit Dec 13, 2018
environment.yml first commit Dec 13, 2018
eval_retrieval.py first commit Dec 13, 2018
image_features_extraction.py first commit Dec 13, 2018
pointing_game.py first commit Dec 13, 2018
semantic_seg.py first commit Dec 13, 2018
text_features_extraction.py first commit Dec 13, 2018
train.py first commit Dec 13, 2018

README.md

Deep semantic-visual embedding with localization

Training and evalutation code for the paper Finding beans in burgers: Deep semantic-visual embedding with localization

This code allow training of new model, reproduction of experiments, as well as features extraction for both images and texts.

Author and contact: Martin Engilberge

Main dependencies

This code is written in python. To use it you will need:

  • Python 3.7
  • Pytorch 1.0
  • SRU[cuda]
  • Numpy
  • Scipy
  • Torchvision
  • Ms Coco API (pycocotools)
  • Visual Genome API
  • NLTK
  • opencv

An environment file for conda is available in the repository (environment.yml).

Getting started

You will first need to set the paths to the datasets and word embedding in the file misc/config.py Commentaries in the config file contains links where you can download the data.

To train and run model you will need:

To reproduce experiments in the paper:

Once the required paths have been set in the config file you can start training models using the following command:

python train.py

By default all the scripts use gpu, you can switch to cpu mode by uncommenting device = torch.device("cpu") at the beginning of the script.

Model evaluation

Models can be evaluated on three tasks:

  • cross modal retrieval:
python eval_retrieval.py -p "path/to/model/model.pth.tar" -te
  • pointing game:
python pointing_game.py -p "path/to/model/model.pth.tar"
  • semantic segmentation:
python semantic_seg.py -p "path/to/model/model.pth.tar"

Features extraction

The features space produced by the joint embedding manages to capture semantic property. Two scripts can be used to extract feature from that space for images and texts.

For images the script takes a folder as input and produces the embedding representation for all the jpeg images in the folder.

python image_features_extraction.py -p "path/to/model/model.pth.tar" -d "path/to/image/folder/" -o "path/to/output/file"

For text the script takes a text file and produces the embedding representation for each line.

python text_features_extraction.py -p "path/to/model/model.pth.tar" -d "path/to/text/file/" -o "path/to/output/file"

Reference

If you found this code useful, please cite the following paper:

@inproceedings{engilberge2018finding,
  title={Finding beans in burgers: Deep semantic-visual embedding with localization},
  author={Engilberge, Martin and Chevallier, Louis and P{\'e}rez, Patrick and Cord, Matthieu},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={3984--3993},
  year={2018}
}

License

by downloading this program, you commit to comply with the license as stated in the LICENSE.md file.