Natural Language Object Retrieval in Tensorflow
What is natural language object retrieval?
In real AI, you may refer something with its location, color, or other characteristic. The AI robot should know where the corresponding object is. Here's the illustration:
The blue box represents the ground truth, while the yellow stands for positive recall, red for negative recall.
Here, all the experiments are done on ReferIt dataset. You can also train on other dataset in the same way.
- Download caffe:
- modify Makefile.config
- Download the pretrained models:
- Download the ReferIt dataset:
- Download pre-extracted EdgeBox proposals:
- Preprocess the ReferIt dataset to generate metadata needed for training and evaluation:
- Cache the scene-level contextual features to disk:
- Cache the bbox features to disk:
- Build training batches:
Once you prepare the data, you can run the training code
If you want to use pretrained model, you should convert the pretrained model into
.npy and load the pretrained model when training.
Before testing, I cache the proposal feature to disk due to the RAM contraints.
- cache the proposal feature:
- choose which model you want to test by modifying this line.
I use R@10 to evaluate the performance and all the test is on ReferIt dataset.
the performance through different epoch [without pretrained]:
the performance through different epoch [with pretrained]:
And here is the loss thourgh every epoch:
- Natural language object retrieval, Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, Trevor Darrell, cvpr2016 oral
Feel free to contact me whenever you encounter unexpected errors in this repo.