AlphaGamma keypoint descriptor

Prototype & evaluation framework

(C) 2015-2017, Rok Mandeljc

Summary

This project contains the prototype implementation of the AlphaGamma keypoint descriptor, presented in:

1. R. Mandeljc and J. Maver, AGs: Local descriptors dervied from the dependent effects model, Journal of Visual Communication and Image Representation, Volume 58, p. 503-514, January 2019. DOI: 10.1016/j.jvcir.2018.12.008

The code is provided as supplement to the journal submission [1], and provides reference/prototype implementation of the AlphaGamma descriptor, as well as the RADIAL keypoint detector. In addition, experimental framework and scripts for reproducing the experimental results from the paper are provided.

The sections below outline the installation and setup, basic use of the code, and steps needed to reproduce the experiments.

Note: the majority of instructions are linux-centric (i.e., most of the listed command-line steps are written for linux shell).

Prerequisites

The code was primarily developed on linux (Fedora 24/25) with Matlab R2016b, but has also been tested on Windows with Visual Studio 2015 and Matlab R2016a.

The code makes use of OpenCV via mexopencv. For the sake of consistency, all the requirements are bundled with the code by means of git submodules. For OpenCV, we build and locally install a checkout from a custom branch that contains couple of fixes needed for consistent descriptor evaluation.

Linux

Recent 64-bit linux distribution with basic compilation toolchain (git, gcc, CMake, make). In addition, dependencies for building the git checkout of OpenCV are required. On Fedora, the basic set of development libraries I use can be installed via:

sudo dnf install git cmake gcc-c++ \
    libtiff-devel libjpeg-devel libwebp-devel jasper-devel OpenEXR-devel \
    ffmpeg-devel \
    eigen3-devel tbb-devel openblas-devel

Recent Matlab with Image processing toolbox and working MEX compiler.

Windows

The code was tested using 64-bit Windows 8.1 and Visual Studio 2015.

A recent Matlab with Image processing toolbox and working MEX compiler is required. Make sure that Matlab executable is in PATH; i.e., that you can start it by running matlab from Windows command prompt (cmd).

In addition, you will need git and CMake. Make sure that the path to CMake executable is in PATH.

Ensure that MEX compiler is properly set up in Matlab, and that it points to the correct Visual Studio installation. You can check this by running the following inside Matlab:

mex -setup C++

and following its instructions for choosing the correct compiler.

Installation

Create a working directory and move inside it, e.g.:

mkdir alphagamma-descriptor
cd alphagamma-descriptor

This directory will contain the checkout of code, as well as datasets, if you wish to replicate the results from the paper. By default, the code makes certain assumptions about locations of the dataset images that are tied to the described structure of the working directory. Unless stated otherwise, the rest of instructions assumes that commands are run from this directory (both shell and Matlab).

Checkout the code from git repository into "code" subdirectory:

git clone https://github.com/rokm/alphagamma-descriptor code
cd code
git submodule update --init --recursive
cd ..

The above command should also pull in all external dependencies from their corresponding repositories.

If you wish to use our evaluation framework to replicate the results from paper, follow to the next subsection to install the datasets. If you wish to just use the keypoint detectors and descriptors, you can skip the following subsection and proceed to the installation instructions for your platform.

Installing datasets

The evaluation code assumes that the datasets are located inside datasets subfolder inside your working directory. Therefore, create the datasets directory, and move inside it.

mkdir datasets
cd datasets

Oxford Affine dataset

The Oxford dataset sequences are available here: http://www.robots.ox.ac.uk/~vgg/research/affine

Inside the datasets directory, create a directory called affine. Inside this directory, download the sequences and unpack them to their corresponding directories:

mkdir affine
cd affine

wget http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/bikes.tar.gz
mkdir bikes
tar xvzf bikes.tar.gz -C bikes

wget http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/trees.tar.gz
mkdir trees
tar xvzf trees.tar.gz -C trees

wget http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/graf.tar.gz
mkdir graffiti
tar xvzf graf.tar.gz -C graffiti

wget http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/wall.tar.gz
mkdir wall
tar xvzf wall.tar.gz -C wall

http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/bark.tar.gz
mkdir bark
tar xvzf bark.tar.gz -C bark

http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/boat.tar.gz
mkdir boat
tar xvzf boat.tar.gz -C boat

http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/leuven.tar.gz
mkdir leuven
tar xvzf leuven.tar.gz -C leuven

http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/ubc.tar.gz
mkdir ubc
tar xvzf ubc.tar.gz -C ubc

rm -f *.tar.gz
cd ..

DTU dataset

The DTU Point Feature Data Set is avaiable here: http://roboimagedata.compute.dtu.dk/?page_id=24

Inside the datasets directory, create a directory called dtu_robot, into which we will unpack the required parts of the dataset.

mkdir dtu_robot
cd dtu_robot

First, download calibration file and rename it to Calib_Results_11.mat:

wget http://roboimagedata.imm.dtu.dk/data/calibration/calibrationFile.mat
mv calibrationFile.mat Calib_Results_11.mat

Download 3-D reconstruction data and unpack it into CleanRecon_2009_11_16 directory:

wget http://roboimagedata.imm.dtu.dk/data/3D_reconstructions/reconstructions.zip
unzip reconstructions.zip -d CleanRecon_2009_11_16

The above two steps can be replaced by downloading the whole evaluation code package from http://roboimagedata.imm.dtu.dk/code/RobotEvalCode.tar.gz and extracting the afore-mentioned Calib_Results_11.mat file and CleanRecon_2009_11_16 directory into current (dtu_robot directory).

Afterwards, download and extract the sequence data (we are using the half-sized images), and extract them directly into current directory. Even the half-sized sequences are quite large, and may take a while to download.

wget http://roboimagedata.imm.dtu.dk/data/tar600x800/SET007_12.tar.gz
tar xzf SET007_12.tar.gz

wget http://roboimagedata.imm.dtu.dk/data/tar600x800/SET019_24.tar.gz
tar xzf SET019_24.tar.gz

wget http://roboimagedata.imm.dtu.dk/data/tar600x800/SET043_48.tar.gz
tar xzf SET043_48.tar.gz

wget http://roboimagedata.imm.dtu.dk/data/tar600x800/SET049_54.tar.gz
tar xzf SET049_54.tar.gz

Finally, move back into datasets directory:

cd ..

WebCam dataset

WebCam dataset is available here: http://cvlab.epfl.ch/research/tilde

Download the dataset and extract it into webcam subdirectory (if you are extracting manually, move the contents of WebcamRelease into webcam):

mkdir webcam
cd webcam
wget https://documents.epfl.ch/groups/c/cv/cvlab-unit/www/data/keypoints/WebcamRelease.tar.gz
tar xzf /home/rok/Downloads/WebcamRelease.tar.gz  --strip=1
cd ..

Sanity check

Finally, move out of the datasets directory (back into your working directory):

cd ..

Following all the steps up so far, you should end up with the following directory structure inside your working directory, which is what the experimental framework will expect to find:

code: build_all.bat build_all.sh compile_code.m LICENSE README.md start_matlab.sh startup.m
 - external: lapjv  mexopencv  opencv  opencv_contrib  tight_subplot
 - paper2017: bargraph.tmpl.tex ... jasna_visualizations_webcam.m
 - +vicos: +descriptor +experiment +keypoint_detector +utils

datasets:
 - affine:
    - bark: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - bikes: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - boat: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - graffiti: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - leuven: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - trees: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - ucb: H1to2p ... H1to6p img1.ppm ... img6.ppm
    - wall: H1to2p ... H1to6p img1.ppm ... img6.ppm
 - dtu_robot: Calib_Results_11.mat
    - CleanRecon_2009_11_16: Clean_Reconstruction_01.mat ... Clean_Reconstruction_60.mat
    - SET007: Img001_01.bmp ... Img119_19.bmp
    - SET008: Img001_01.bmp ... Img119_19.bmp
    - SET009: Img001_01.bmp ... Img119_19.bmp
    - SET010: Img001_01.bmp ... Img119_19.bmp
    - SET011: Img001_01.bmp ... Img119_19.bmp
    - SET012: Img001_01.bmp ... Img119_19.bmp
    - SET019: Img001_01.bmp ... Img119_19.bmp
    - SET020: Img001_01.bmp ... Img119_19.bmp
    - SET021: Img001_01.bmp ... Img119_19.bmp
    - SET022: Img001_01.bmp ... Img119_19.bmp
    - SET023: Img001_01.bmp ... Img119_19.bmp
    - SET024: Img001_01.bmp ... Img119_19.bmp
    - SET043: Img001_01.bmp ... Img119_19.bmp
    - SET044: Img001_01.bmp ... Img119_19.bmp
    - SET045: Img001_01.bmp ... Img119_19.bmp
    - SET046: Img001_01.bmp ... Img119_19.bmp
    - SET047: Img001_01.bmp ... Img119_19.bmp
    - SET048: Img001_01.bmp ... Img119_19.bmp
    - SET049: Img001_01.bmp ... Img119_19.bmp
    - SET050: Img001_01.bmp ... Img119_19.bmp
    - SET051: Img001_01.bmp ... Img119_19.bmp
    - SET052: Img001_01.bmp ... Img119_19.bmp
    - SET053: Img001_01.bmp ... Img119_19.bmp
    - SET054: Img001_01.bmp ... Img119_19.bmp
 - webcam: README.txt
    - Chamonix: test train
    - Courbevoie: test train
    - Frankfurt: test train
    - Mexico: test train
    - Panorama: test train
    - StLouis: test train

Linux

To simplify the build process, use the provided build_all.sh script.

First, export the path to your matlab installation:

export MATLABDIR=/usr/local/MATLAB/R2016b

Then, run the build script from your working directory:

./code/build_all.sh

The shell script will attempt to:

• build OpenCV and install it inside the pre-determined sub-directory inside the code directory
• build mexopencv (using make)
• run Matlab-side build script compile_code.m, which builds additional MEX files

If no errors occurred, the script will print a message about successfully finishing the build process.

On linux, the OpenCV shared libraries (required by mexopencv) need to be in your LD_LIBRARY_PATH before Matlab is started. For convenience, a startup script is provided which takes care of that for your. Therefore, to start Matlab, run the following script from your working directory:

./code/start_matlab.sh

It will set up LD_LIBRARY_PATH and run the startup.m Matlab script to properly set up paths to external dependencies.

Windows

Similarly to linux, a build batch script is available. Open the Windows Prompt (cmd), and move inside the working directory.

Make sure that Matlab and CMake are in the PATH.

Then, set the Visual Studio version and architecture (required for CMake), and run the build script from the working directory:

set "DEFAULT_CMAKE_GENERATOR=Visual Studio 14"
set "DEFAULT_CMAKE_ARCH=x64"

code\build_all.bat

The script will attempt to:

• build OpenCV and install it inside the pre-determined sub-directory inside the code directory
• run Matlab and build mexopencv
• run Matlab-side build script compile_code.m, which builds additional MEX files

If no errors occurred, the script will print a message about successfully finishing the build process.

On Windows (in contrast to linux; see above), Matlab can be started normally - but make sure that the startup.m is executed before using the functions and objects from this project.

Basic use

This package provides reference implementation for AlphaGamma descriptor and RADIAL keypoint detector, as well as wrappers for several OpenCV-provided detectoes and the descriptors.

Keypoint detector and descriptor extractor classes are located inside vicos.keypoint_detector and vicos.descriptor namespace, respectively.

All detectors inherit the base class vicos.keypoint_detector.Detector, with method vicos.keypoint_detector.Detector.detect() that is used for detecting keypoints in the image.

Similarly, descriptor extractors inherit the base class vicos.descriptor.Descriptor, with two main methods: vicos.descriptor.Descriptor.compute() is used to compute descriptors from given keypoints and image, while vicos.descriptor.Descriptor.compute_pairwise_distances() is used to compute the distance matrix between two sets of exracted descriptors.

The following example illustrates the use of RADIAL keypoints with AG and AGS descriptors. The static methods vicos.descriptor.AlphaGamma.create_ag_float() and vicos.descriptor.AlphaGamma.create_ag_short() provide the default parametrization of AG and AGS descriptor from the paper [1].

% Load images (assuming datasets were installed and that we are inside
% the working directory)
I1 = imread('datasets/affine/graffiti/img1.ppm');
I2 = imread('datasets/affine/graffiti/img2.ppm');


%% Keypoint detection
% Create RADIAL keypoint detector
detector = vicos.keypoint_detector.FeatureRadial();

% Detect keypoints
kpts1 = detector.detect(I1);
kpts2 = detector.detect(I2);


%% AG descriptor
% Create floating-point AlphaGamma;
ag_float = vicos.descriptor.AlphaGamma.create_ag_float('base_keypoint_size', 8.25);

% Compute descriptors
[ desc1, kpts1 ] = ag_float.compute(I1, kpts1);
[ desc2, kpts2 ] = ag_float.compute(I2, kpts2);

% Compute distance matrix
M = ag_float.compute_pairwise_distances(desc1, desc2);


%% AGS descriptor
% Create binarized AlphaGamma
ag_short = vicos.descriptor.AlphaGamma.create_ag_short('base_keypoint_size', 8.0);

% Compute descriptors
[ desc1, kpts1 ] = ag_short.compute(I1, kpts1);
[ desc2, kpts2 ] = ag_short.compute(I2, kpts2);

% Compute distance matrix
M = ag_short.compute_pairwise_distances(desc1, desc2);

An example with SIFT keypoints:

% Load images (assuming datasets were installed and that we are inside
% the working directory)
I1 = imread('datasets/affine/graffiti/img1.ppm');
I2 = imread('datasets/affine/graffiti/img2.ppm');


%% Keypoint detection
% Create SIFT keypoint detector
detector = vicos.keypoint_detector.SIFT();

% Detect keypoints
kpts1 = detector.detect(I1);
kpts2 = detector.detect(I2);


%% SIFT descriptor
sift = vicos.descriptor.SIFT();

% Compute descriptors
[ desc1, kpts1 ] = sift.compute(I1, kpts1);
[ desc2, kpts2 ] = sift.compute(I2, kpts2);

% Compute distance matrix
M = sift.compute_pairwise_distances(desc1, desc2);


%% AG descriptor
% Create floating-point AlphaGamma; note that in general, different
% keypoint types require different base_keypoint_size parameter.
ag_float = vicos.descriptor.AlphaGamma.create_ag_float('base_keypoint_size', 3.25);

% Compute descriptors
[ desc1, kpts1 ] = ag_float.compute(I1, kpts1);
[ desc2, kpts2 ] = ag_float.compute(I2, kpts2);

% Compute distance matrix
M = ag_float.compute_pairwise_distances(desc1, desc2);

Reproduction of experimental results

The experimental results from the paper [1] were all obtained using scripts that are located in the paper2017 subfolder inside the code folder.

By default, all experiment functions cache their intermediate and final results; when run subsequently, they will attempt to load cached results before running the experiment. Therefore, the longer experiments can be left to run unattended, and be later re-run with additional options to enable visualization of results.

NOTE: it is assumed that the code snippets below are executed from the working directory as to avoid cluttering the code directory with cache directories and result files.

Synthetic deformations

To reproduce results with synthetic image rotation (Fig. 6a), use function jasna_experiment_rotation():

% Run all experiments (or load cached results).
jasna_experiment_rotation();

% Run all experiments (or load cached results), display Matlab figure,
% and export results to rotation.txt file
jasna_experiment_rotation('display_results', true, 'result_file', 'rotation-results.txt');

Similarly, results for scale (Fig. 6b) and shear (Fig. 6c) can be reproduced using:

jasna_experiment_scale('display_results', true, 'result_file', 'scale-results.txt');

jasna_experiment_shear('display_results', true, 'result_file', 'shear-results.txt');

Experiments on image sequences

The experiments on image sequences (Figs. 5, 7, 8, 9) can be reproduced by running the following functions:

% All experiments on sequences from Oxford dataset (by default, the
% following sequences are used: bikes, trees, leuven, boat, graffiti,
% and wall)
jasna_experiment_affine({ 'sift', 'surf', 'kaze', 'brisk', 'orb', 'radial' });

% All experiments on sequences from Frankfurt dataset (by default, the
% following sequence is used: Frankfurt)
jasna_experiment_webcam({ 'sift', 'surf', 'kaze', 'brisk', 'orb', 'radial' });

% All experiments on sequences from DTU dataset (by default, the following
% sequences are used: SET007, SET022, SET023, and SET049)
jasna_experiment_dtu({ 'sift', 'surf', 'kaze', 'brisk', 'orb', 'radial' });

The mandatory argument is cell array containing the pre-defined names of experiments. For each experiment id, the keypoints are tested with the native descriptor and both AG and AGS (i.e., SIFT, AG, and AGS on SIFT keypoints for 'sift' experiment id).

The experiments will take a while to run. A possible way to parallelize them is to run multiple Matlab instances, and execute the experiment functions with different subsets of above experiment IDs.

The intermediate and final results are cached in corresponding cache directories. To display the final results as shown in paper, use the jasna_display_results() function. Note that this function assumes that experiments have been run for all six experiment IDs that are shown above.

% Oxford sequences; show Matlab figures and export LaTeX code for graphs
jasna_display_results('_cache_affine-gray', { 'bikes', 'trees', 'leuven', 'boat', 'graffiti', 'wall' }, 'display_figure', true, 'output_dir', 'graphs-affine');

% Webcam sequence; show only Matlab figures (same as if 'display_figure' option was also omitted)
jasna_display_results('_cache_webcam-gray', { 'Frankfurt' }, 'display_figure', true);

% DTU sequences; do not display Matlab figures, but export the LaTeX code for graphs.
jasna_display_results('_cache_dtu-gray', { 'SET007', 'SET022', 'SET023', 'SET049' }, 'output_dir', 'graphs-dtu', 'display_figure', false);

The first argument is the name of cache directory that was created by the experiment script. The second argument is cell array with names of sequences (used as prefix for result files inside cache directory).

The 'display_figure' is an optional argument and controls whether Matlab figures with graphs should be created (enabled by default).

The 'output_dir' is an optional argument and controls, whether LaTeX code for graphs is exported (to the specified directory), or not (disabled by default). The specified output directory will contain several .tex files, one for each graph.

Correct match visualizations

Visualizations of correct matches in Fig. 10 were obtained using the following function:

jasna_visualizations_dtu();

The above function will run the relevant experiments. By default, it uses different cache directory than experiment scripts from the previous section. Inside this cache directory, called _visualization_dtu-gray, it will create several folders containing data and LaTeX code for visualizations. For example, folder _visualization_dtu-gray/SET010_Img025_08_Img119_08_SIFT_SIFT will contain LaTeX code and data for visualization of correct matches of SIFT descriptor on SIFT keypoints, when used on images 25 and 119 from DTU SET010.

A similar function can be used to generate some visualizations for sequences from the WebCam dataset:

jasna_visualizations_webcam();

The resulting images were excluded from the paper due to page limit.

LIFT keypoint detector/descriptor support

This repository incorporates support for the LIFT keypoint detector and descriptor from: https://github.com/cvlab-epfl/LIFT

As with other components, the code from above repository is incorporated as git submodule and wrapper classes (vicos.keypoint_detector.LIFT and vicos.descriptor.LIFT). Due to CUDA and python dependencies (Theano, lasagne, etc.) and limited testing, the support is limited to linux-based operating systems for now.

The instructions below are written for Fedora 25/26, where integration was developed and tested.

Dependencies

LIFT code requires a CUDA-compatible GPU.

For CUDA and CuDNN, Fedora packages are available in Negativo17 repository: https://negativo17.org/nvidia-driver

After setting it up, install cuda and cudnn (version 5):

sudo dnf install cuda-devel cuda-cudnn5.1-devel

On Fedora 25/26, the dependencies can be install directly from official repositories:

sudo dnf install python2-opencv python2-h5py python2-flufl-lock python2-parse python2-scipy python2-theano

We also need python2-lasagne, but Fedora (at the time of writing) provides incompatible (outdated) 0.1 version, which does not work with Theano/LIFT. Hence, you need to install the development version on your own, or grab it from the following Copr repository: (TBA)

Building C++ part

The LIFT code also includes some C++ code that needs to be compiled; this is handled by the master build script (from beginning of this README):

./code/build_all.sh

Setting up Theano

Fedora 26 comes with gcc compiler version that is incompatible with CUDA 8. Installing cuda packages from Negativo17 repository should also pull in the compat-gcc-53 packages. However, we need to instruct Theano to use it when compiling generated code. Create ~/.theanorc file with the following content:

[nvcc]
flags=-D_FORCE_INLINES -ccbin=/usr/bin/g++53

[global]
floatX = float32
device = cuda0

The floatX and device settings are necessary to avoid messages about using old gpu back-end and floatX=64.

Using LIFT wrapper

LIFT detector/descriptor can be used in same way as other wrapped detectors and descriptors, e.g.:

% Load images (assuming datasets were installed and that we are inside
% the working directory)
I1 = imread('datasets/affine/graffiti/img1.ppm');
I2 = imread('datasets/affine/graffiti/img2.ppm');


%% Keypoint detection
% Create LIFT keypoint detector
detector = vicos.keypoint_detector.LIFT();

% Detect keypoints
kpts1 = detector.detect(I1);
kpts2 = detector.detect(I2);


%% LIFT descriptor
lift = vicos.descriptor.LIFT();

% Compute descriptors
[ desc1, kpts1 ] = lift.compute(I1, kpts1);
[ desc2, kpts2 ] = lift.compute(I2, kpts2);

% Compute distance matrix
M = lift.compute_pairwise_distances(desc1, desc2);

The first run may take a while as it, behind the scenes, performs compilation of auto-generated code.

Using LIFT in experiments code

To activate LIFT experiments, use 'lift' experiment name when calling jasna_experiment_affine(), jasna_experiment_webcam(), and jasna_experiment_dtu().