Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences

Axel Barroso-Laguna · Sowmya Munukutla · Victor Adrian Prisacariu · Eric Brachmann

CVPR 2024 (Oral)

Project Page | Paper | arXiv | Supplemental

This is the reference implementation of the paper "Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences" presented at CVPR 2024.

The paper introduces Metric Keypoints (MicKey), a feature detection pipeline that regresses keypoint positions in camera space. MicKey presents a differentiable approach to establish metric correspondences via descriptor matching. From the metric correspondences, MicKey recovers metric relative poses. MicKey is trained in an end-to-end fashion using differentiable pose optimization and requires only image pairs and their ground truth relative poses for supervision.

Setup

Assuming a fresh Anaconda distribution, you can install dependencies with:

conda env create -f resources/environment.yml
conda activate mickey

We ran our experiments with PyTorch 2.0.1, CUDA 11.6, Python 3.8.17 and Debian GNU/Linux 11.

Evaluating MicKey

MicKey aims at addressing the problem of instant Augmented Reality (AR) introduced in the Map-free benchmark. In the Map-free set up, instead of building 3D maps from hundreds of images and scale calibrations, they propose to use only one photo of a scene as the map. The Map-free benchmark then evaluates how accurate is the estimated metric relative pose between the reference image (the map) and the query image (the user).

Download Map-free dataset

You can find the Map-free dataset in their project page. Extract the test.zip file into data/mapfree. Optionally, if you want to train MicKey, also download train and val zip files.

Pre-trained Models

We provide two MicKey models.

• mickey.ckpt: These are the default weights for MicKey, without using the overlapping scores provides in Map-free dataset and following the curriculum learning strategy described in the paper.
• mickey_sc.ckpt: These are the weights when training MicKey using the min and max overlapping scores defined in Map-free.

Extract mickey_weights.zip into weights/. In the zip file, we also provide the default configuration needed to run the evaluation.

Run the submission script

Similar to Map-free code base, we provide a submission script to generate submission files:

python submission.py --config path/to/config --checkpoint path/to/checkpoint --o results/your_method

The resulting file results/your_method/submission.zip can be uploaded to the Map-free online benchmark website and compared against existing methods in the leaderboard.

Run the local evaluation

The Map-free benchmark does not provide ground-truth poses for the test set. But we can still evaluate our method locally on the validation set.

python submission.py --config path/to/config --checkpoint path/to/checkpoint --o results/your_method --split val

and evaluate it as:

python -m benchmark.mapfree --submission_path results/your_method/submission.zip --split val

Running MicKey in custom images

We provide a demo script to run the relative pose estimation pipeline on custom image pairs. As an example, we store in data/toy_example two images with their respective intrinsics. The script computes their metric relative pose and saves the corresponding depth and keypoint score maps in the image folder. Run the demo script as:

python demo_inference.py --im_path_ref data/toy_example/im0.jpg \
                         --im_path_dst data/toy_example/im1.jpg \
                         --intrinsics data/toy_example/intrinsics.txt \
                         --checkpoint path/to/checkpoint \
                         --config path/to/config

Training MicKey

Besides the test scripts, we also provide the training code to train MicKey.

We provide two default configurations in config/MicKey/:

• curriculum_learning.yaml: This configuration follows the curriculum learning approach detailed in the paper. It hence does not use any image overlapping information but only relative ground truth poses during training.
• overlap_score.yaml: This configuration relies on the image overlapping information to only choose solvable image pairs during training.

To train MicKey default model, use:

python train.py --config config/MicKey/curriculum_learning.yaml \
                --dataset_config config/datasets/mapfree.yaml \
                --experiment experiment_name \
                --path_weights path/to/checkpoint/folder

Resume training from a checkpoint by adding --resume {path_to_checkpoint}.

The top models, according to the validation loss, the VCRE metric, and the pose AUC score, are saved during training. Tensorboard results and checkpoints are saved into the folder dir/to/weights/experiment_name.

Note that by default, the configuration is set to use 4 GPUs. You can reduce the expected number of GPUs in the config file (e.g., NUM_GPUS: 1).

BibTeX

If you use this code in your research, please consider citing our paper:

@inproceedings{barroso2024mickey,
  title={Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences},
  author={Barroso-Laguna, Axel and Munukutla, Sowmya and Prisacariu, Victor and Brachmann, Eric},
  booktitle={CVPR},
  year={2024}
}

License

Copyright © Niantic, Inc. 2024. Patent Pending. All rights reserved. This code is for non-commercial use. Please see the license file for terms.

Acknowledgements

We use part of the code from different repositories. We thank the authors and maintainers of the following repositories.

• Map-free
• RoMa
• DINOv2
• LoFTR
• DPT
• ExtremeRotation