Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
1 contributor

Users who have contributed to this file

229 lines (181 sloc) 8.56 KB

Detect-to-Retrieve instructions

These instructions can be used to reproduce the results from the Detect-to-Retrieve paper for the Revisited Oxford/Paris datasets.

Install DELF library

To be able to use this code, please follow these instructions to properly install the DELF library.

Download datasets

mkdir -p ~/detect_to_retrieve/data && cd ~/detect_to_retrieve/data

# Oxford dataset.
wget http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/oxbuild_images.tgz
mkdir oxford5k_images
tar -xvzf oxbuild_images.tgz -C oxford5k_images/

# Paris dataset. Download and move all images to same directory.
wget http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/paris_1.tgz
wget http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/paris_2.tgz
mkdir paris6k_images_tmp
tar -xvzf paris_1.tgz -C paris6k_images_tmp/
tar -xvzf paris_2.tgz -C paris6k_images_tmp/
mkdir paris6k_images
mv paris6k_images_tmp/paris/*/*.jpg paris6k_images/

# Revisited annotations.
wget http://cmp.felk.cvut.cz/revisitop/data/datasets/roxford5k/gnd_roxford5k.mat
wget http://cmp.felk.cvut.cz/revisitop/data/datasets/rparis6k/gnd_rparis6k.mat

Download models

These are necessary to reproduce the main paper results:

# From models/research/delf/delf/python/detect_to_retrieve
mkdir parameters && cd parameters

# DELF-GLD model.
wget http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz
tar -xvzf delf_gld_20190411.tar.gz

# Faster-RCNN detector model.
wget http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz
tar -xvzf d2r_frcnn_20190411.tar.gz

# Codebooks.
# Note: you should use codebook trained on rparis6k for roxford5k retrieval
# experiments, and vice-versa.
wget http://storage.googleapis.com/delf/rparis6k_codebook_65536.tar.gz
mkdir rparis6k_codebook_65536
tar -xvzf rparis6k_codebook_65536.tar.gz -C rparis6k_codebook_65536/
wget http://storage.googleapis.com/delf/roxford5k_codebook_65536.tar.gz
mkdir roxford5k_codebook_65536
tar -xvzf roxford5k_codebook_65536.tar.gz -C roxford5k_codebook_65536/

We also make available other models/parameters that can be used to reproduce more results from the paper:

Feature extraction

We present here commands for extraction on roxford5k. To extract on rparis6k instead, please edit the arguments accordingly (especially the dataset_file_path argument).

Query feature extraction

For query feature extraction, the cropped query image should be used to extract features, according to the Revisited Oxford/Paris experimental protocol. Note that this is done in the extract_query_features script.

Query feature extraction can be run as follows:

# From models/research/delf/delf/python/detect_to_retrieve
python extract_query_features.py \
  --delf_config_path delf_gld_config.pbtxt \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --images_dir ~/detect_to_retrieve/data/oxford5k_images \
  --output_features_dir ~/detect_to_retrieve/data/oxford5k_features/query

Index feature extraction and box detection

Index feature extraction / box detection can be run as follows:

# From models/research/delf/delf/python/detect_to_retrieve
python extract_index_boxes_and_features.py \
  --delf_config_path delf_gld_config.pbtxt \
  --detector_model_dir parameters/d2r_frcnn_20190411 \
  --detector_thresh 0.1 \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --images_dir ~/detect_to_retrieve/data/oxford5k_images \
  --output_boxes_dir ~/detect_to_retrieve/data/oxford5k_boxes/index \
  --output_features_dir ~/detect_to_retrieve/data/oxford5k_features/index_0.1 \
  --output_index_mapping ~/detect_to_retrieve/data/oxford5k_features/index_mapping_0.1.csv

R-ASMK* aggregation extraction

We present here commands for aggregation extraction on roxford5k. To extract on rparis6k instead, please edit the arguments accordingly. In particular, note that feature aggregation on roxford5k should use a codebook trained on rparis6k, and vice-versa (this can be edited in the query_aggregation_config.pbtxt and index_aggregation_config.pbtxt files.

Query

Run query feature aggregation as follows:

# From models/research/delf/delf/python/detect_to_retrieve
python extract_aggregation.py \
  --use_query_images True \
  --aggregation_config_path query_aggregation_config.pbtxt \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --features_dir ~/detect_to_retrieve/data/oxford5k_features/query \
  --output_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/query

Index

Run index feature aggregation as follows:

# From models/research/delf/delf/python/detect_to_retrieve
python extract_aggregation.py \
  --aggregation_config_path index_aggregation_config.pbtxt \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --features_dir ~/detect_to_retrieve/data/oxford5k_features/index_0.1 \
  --index_mapping_path ~/detect_to_retrieve/data/oxford5k_features/index_mapping_0.1.csv \
  --output_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/index_0.1

Perform retrieval

Currently, we support retrieval via brute-force comparison of aggregated features.

To run retrieval on roxford5k, the following command can be used:

# From models/research/delf/delf/python/detect_to_retrieve
python perform_retrieval.py \
  --index_aggregation_config_path index_aggregation_config.pbtxt \
  --query_aggregation_config_path query_aggregation_config.pbtxt \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --index_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/index_0.1 \
  --query_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/query \
  --output_dir ~/detect_to_retrieve/results/oxford5k

A file with named metrics.txt will be written to the path given in output_dir, with retrieval metrics for an experiment where geometric verification is not used. The contents should look approximately like:

hard
mAP=47.61
mP@k[ 1  5 10] [84.29 73.71 64.43]
mR@k[ 1  5 10] [18.84 29.44 36.82]
medium
mAP=73.3
mP@k[ 1  5 10] [97.14 94.57 90.14]
mR@k[ 1  5 10] [10.14 26.2  34.75]

which are the results presented in Table 2 of the paper (with small numerical precision differences).

If you want to run retrieval with geometric verification, set use_geometric_verification to True and the arguments index_features_dir/query_features_dir. It's much slower since (1) in this code example the re-ranking is loading DELF local features from disk, and (2) re-ranking needs to be performed separately for each dataset protocol, since the junk images from each protocol should be removed when re-ranking. Here is an example command:

# From models/research/delf/delf/python/detect_to_retrieve
python perform_retrieval.py \
  --index_aggregation_config_path index_aggregation_config.pbtxt \
  --query_aggregation_config_path query_aggregation_config.pbtxt \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_roxford5k.mat \
  --index_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/index_0.1 \
  --query_aggregation_dir ~/detect_to_retrieve/data/oxford5k_aggregation/query \
  --use_geometric_verification True \
  --index_features_dir ~/detect_to_retrieve/data/oxford5k_features/index_0.1 \
  --query_features_dir ~/detect_to_retrieve/data/oxford5k_features/query \
  --output_dir ~/detect_to_retrieve/results/oxford5k_with_gv

Clustering

In the code example above, we used a pre-trained DELF codebook. We also provide code for re-training the codebook if desired.

Note that for the time being this can only run on CPU, since the main ops in K-means are not registered for GPU usage in Tensorflow.

# From models/research/delf/delf/python/detect_to_retrieve
python cluster_delf_features.py \
  --dataset_file_path ~/detect_to_retrieve/data/gnd_rparis6k.mat \
  --features_dir ~/detect_to_retrieve/data/paris6k_features/index_0.1 \
  --num_clusters 1024 \
  --num_iterations 50 \
  --output_cluster_dir ~/detect_to_retrieve/data/paris6k_clusters_1024

Next steps

To make retrieval more scalable and handle larger datasets more smoothly, we are considering to provide code for inverted index building and retrieval. Please reach out if you would like to help doing that -- feel free submit a pull request.

You can’t perform that action at this time.