Skip to content


Repository files navigation

Hierarchical Localization

⚠️ ⚠️ For a clean and research-friendly implementation of Hierarchical Localization, please refer to our CVPR 2019 paper at ethz-asl/hfnet. ⚠️ ⚠️

This repository contains the training and deployment code used in our paper Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization presented at CoRL 2018. This work introduces MobileNetVLAD, a mobile-friendly image retrieval deep neural network that significantly improves the performance of classical 6-DoF visual localization through a hierarchical search.

The approach is described in details in our video (click to play).

We introduce here two main features:

  • The deployment code of MobileNetVLAD: global-loc, a C++ ROS/Catkin package that can
    • load any trained image retrieval model,
    • efficiently perform the inference on GPU or CPU,
    • index a given map and save it as a protobuf,
    • and retrieve keyframes given a query image;
  • The training code: retrievalnet, a modular Python+Tensorflow package that allows to
    • train the model on any target image domain,
    • using the supervision of any existing teacher network.

The modularity of our system allows to train a model and index a map on a powerful workstation while performing the retrieval on a mobile platform. Our code has thus been extensively tested on an NVIDIA Jetson TX2, widely used for robotics research.

Retrieval on our Zurich dataset: strong illumination and viewpoint changes.


The package relies on map primitives provided by maplab, but can be easily adapted to other SLAM frameworks. We thus do not release the code performing the local matching. The trained MobileNetVLAD is provided in global-loc/models/ and is loaded using tensorflow_catkin.


Both Ubuntu 14.04 and 16.04 are supported. First install the system packages required by maplab.

Then setup the Catkin workspace:

export ROS_VERSION=kinetic #(Ubuntu 16.04: kinetic, Ubuntu 14.04: indigo)
export CATKIN_WS=~/maplab_ws
mkdir -p $CATKIN_WS/src
catkin init
catkin config --merge-devel # Necessary for catkin_tools >= 0.4.
catkin config --extend /opt/ros/$ROS_VERSION
catkin config --cmake-args \
	-DCMAKE_CXX_FLAGS="-fext-numeric-literals -msse3 -msse4.1 -msse4.2" \
cd src

If you want to perform the inference on GPU (see the requirements of tensorflow_catkin), add:

catkin config --append-args --cmake-args -DUSE_GPU=ON

Finally clone the repository and build:

git clone --recursive
touch hierarchical_loc/catkin_dependencies/maplab_dependencies/3rd_party/eigen_catkin/CATKIN_IGNORE
touch hierarchical_loc/catkin_dependencies/maplab_dependencies/3rd_party/protobuf_catkin/CATKIN_IGNORE
cd $CATKIN_WS && catkin build global_loc

Run the test examples:



Given a VI map in global-loc/maps/, an index of global descriptors can be created in global-loc/data/:

./devel/lib/global_loc/build_index \
	--map_name <map_name> \
	--model_name mobilenetvlad_depth-0.35 \
	--proto_name <index_name.pb>

As an example, we provide the Zurich map used in our paper. Several indexing options are available in, such as subsampling or mission selection.


An example of query is provided in Descriptor indexes for the Zurich dataset are included in global-loc/data/ and can be used to time the queries:

./devel/lib/global_loc/time_query \
	--map_name <map_name> \
	--model_name mobilenetvlad_depth-0.35 \
	--proto_name lindenhof_afternoon_aligned_mobilenet-d0.35.pb \
	--query_mission f6837cac0168580aa8a66be7bbb20805 \
	--use_pca --pca_dims 512 --max_num_queries 100

Use the same indexes to evaluate and visualize the retrieval: install retrievalnet, generate the Python protobuf interface, and refer to tango_evaluation.ipynb and tango_visualize_retrieval.ipynb.


We use distillation to compress the original NetVLAD model into a smaller MobileNetVLAD with mobile real-time inference capability.


Python 3.5 is required. It is advised to run the following installation commands within a virtual environment. You will be prompted to provide the path to a data folder (subsequently referred as $DATA_PATH) containing the datasets and pre-trained models and to an experiment folder ($EXPER_PATH) containing the trained models, training logs, and exported descriptors for evaluation.

cd retrievalnet && make install

Exporting the target descriptors

If you wish to train MobileNetVLAD on the Google Landmarks dataset as done in our paper, you first need to download the index of images and then download the dataset itself with The weights of the original NetVLAD model are provided by netvlad_tf_open and should be extracted in $DATA_PATH/weights/.

Finally export the descriptors of Google Landmarks:

python config/netvlad_export_distill.yaml google_landmarks/descriptors --as_dataset

Training MobileNetVLAD

Extract the MobileNet encoder pre-trained on ImageNet in $DATA_PATH/weights/ and run:

python config/mobilenetvlad_train_distill.yaml mobilenetvlad

The training can be interrupted at any time using Ctrl+C and can be monitored with Tensorboard summaries saved in $EXPER_PATH/mobilenetvlad/. The weights are also saved there.

Exporting the model for deployment

python config/mobilenetvlad_train_distill.yaml mobilenetvlad

will export the model in $EXPER_PATH/saved_models/mobilenetvlad/.

Evaluating on the NCLT dataset

Download the NCLT sequences in $DATA_PATH/nclt/ along with the corresponding pose files (generated with nclt_generate_poses.ipynb). Export the NCLT descriptors, e.g. for MobileNetVLAD:

python configs/mobilenetvlad_export_nclt.yaml mobilenetvlad

These can be used to evaluate and visualize the retrieval (see nclt_evaluation.ipynb and nclt_visualize_retrieval.ipynb).


Please consider citing the corresponding publication if you use this work in an academic context:

  title={Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization},
  author={Sarlin, Paul-Edouard and Debraine, Frederic and Dymczyk, Marcin and Siegwart, Roland and Cadena, Cesar},
  booktitle={Conference on Robot Learning (CoRL)},