Implement Natural Language Object Retrieval in tensorflow
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
datasets/ReferIt
exp-referit
external
img
models
prototxt
util
Demo_retrieval.ipynb
README.md
initialize.py
retriever.py
test.py
train.py

README.md

Natural Language Object Retrieval in Tensorflow

Implement Natural Language Object Retrieval in Tensorflow. The original code is release here, written in Caffe.
note that some of the codes are from natural-language-object-retrieval

What is natural language object retrieval?

In real AI, you may refer something with its location, color, or other characteristic. The AI robot should know where the corresponding object is. Here's the illustration:
ill
The blue box represents the ground truth, while the yellow stands for positive recall, red for negative recall.

Requirement

Training

Here, all the experiments are done on ReferIt dataset. You can also train on other dataset in the same way.

Install Caffe

  1. Download caffe: ./external/download_caffe.sh
  2. modify Makefile.config
  3. make -j
  4. make pycaffe

Preprocess

  1. Download the pretrained models: ./models/download_trained_models.sh
  2. Download the ReferIt dataset: ./datasets/ReferIt/ReferitData/download_referit_dataset.sh and ./datasets/ReferIt/ImageCLEF/download_data.sh
  3. Download pre-extracted EdgeBox proposals: ./data/download_edgebox_proposals.sh
  4. Preprocess the ReferIt dataset to generate metadata needed for training and evaluation: python ./exp-referit/preprocess_dataset.py
  5. Cache the scene-level contextual features to disk: python ./exp-referit/cache_referit_context_features.py
  6. Cache the bbox features to disk: python ./exp-referit/cache_referit_local_features.py
  7. Build training batches: python ./exp-referit/cache_referit_training_batches.py

Start training

Once you prepare the data, you can run the training code python train.py
If you want to use pretrained model, you should convert the pretrained model into .npy and load the pretrained model when training.

python initialize.py

Testing

Before testing, I cache the proposal feature to disk due to the RAM contraints.

  1. cache the proposal feature: python ./exp-referit/cache_edgebox_feature.py
  2. choose which model you want to test by modifying this line.
  3. python test.py

Experiments result

I use R@10 to evaluate the performance and all the test is on ReferIt dataset.
the performance through different epoch [without pretrained]:

the performance through different epoch [with pretrained]:

And here is the loss thourgh every epoch:

Reference

Feel free to contact me whenever you encounter unexpected errors in this repo.