Skip to content
PyTorch pre-trained model for real-time interest point detection, description, and sparse tracking (https://arxiv.org/abs/1712.07629)
Branch: master
Clone or download
ddetone
Latest commit 4698c7f Jul 31, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets Add README link. Jun 19, 2018
LICENSE Initial commit. Jun 15, 2018
README.md Fix year from 2017->2018 Jul 30, 2018
demo_superpoint.py Warn about opencv version. Jun 16, 2018
superpoint_v1.pth Initial commit. Jun 15, 2018

README.md

Research @ Magic Leap

SuperPoint Weights File and Demo Script

Introduction

This repo contains the pretrained SuperPoint network, as implemented by the originating authors. SuperPoint is a research project at Magic Leap. The SuperPoint network is a fully convolutional deep neural network trained to detect interest points and compute their accompanying descriptors. The detected points and descriptors can thus be used for various image-to-image matching tasks. For more details please see

This demo showcases a simple sparse optical flow point tracker that uses SuperPoint to detect points and match them across video sequences. The repo contains two core files (1) a PyTorch weights file and (2) a python deployment script that defines the network, loads images and runs the pytorch weights file on them, creating a sparse optical flow visualization. Here are videos of the demo running on various publically available datsets:

Freiburg RGBD:

KITTI:

Microsoft 7 Scenes:

MonoVO:

Dependencies

This repo depends on a few standard pythonic modules, plus OpenCV and PyTorch. These commands usually work (tested on Mac and Ubuntu) for installing the two libraries:

pip install opencv-python
pip install torch

Running the Demo

This demo will run the SuperPoint network on an image sequence and compute points and descriptors from the images, using a helper class called SuperPointFrontend. The tracks are formed by the PointTracker class which finds sequential pair-wise nearest neighbors using two-way matching of the points' descriptors. The demo script uses a helper class called VideoStreamer which can process inputs from three different input streams:

  1. A directory of images, such as .png or .jpg
  2. A video file, such as .mp4 or .avi
  3. A USB Webcam

Run the demo on provided directory of images in CPU-mode:

./demo_superpoint.py assets/icl_snippet/

You should see the following output from the ICL-NUIM sequence snippet:

Run the demo on provided .mp4 file in GPU-mode:

./demo_superpoint.py assets/nyu_snippet.mp4 --cuda

You should see the following output from the NYU sequence snippet:

Run a live demo via webcam (id #1) in CPU-mode:

./demo_superpoint.py camera --camid=1

Run the demo on a remote GPU (no display) on 640x480 images and write the output to myoutput/

./demo_superpoint.py assets/icl_snippet/ --W=640 --H=480 --no_display --write --write_dir=myoutput/

Additional useful command line parameters

  • Use --H to change the input image height (default: 120).
  • Use --W to change the input image width (default: 160).
  • Use --display_scale to scale the output visualization image height and width (default: 2).
  • Use --cuda flag to enable the GPU.
  • Use --img_glob to change the image file extension (default: *.png).
  • Use --min_length to change the minimum track length (default: 2).
  • Use --max_length to change the maximum track length (default: 5).
  • Use --conf_thresh to change the point confidence threshold (default: 0.015).
  • Use --nn_thresh to change the descriptor matching distance threshold (default: 0.7).
  • Use --show_extra to show more computer vision outputs.
  • Press the q key to quit.

BibTeX Citation

@inproceedings{detone18superpoint,
  author    = {Daniel DeTone and
               Tomasz Malisiewicz and
               Andrew Rabinovich},
  title     = {SuperPoint: Self-Supervised Interest Point Detection and Description},
  booktitle = {CVPR Deep Learning for Visual SLAM Workshop},
  year      = {2018},
  url       = {http://arxiv.org/abs/1712.07629}
}

Additional Notes

  • We do not intend to release the SuperPoint training or evaluation code, please do not email us to ask for it.
  • We do not intend to release the Synthetic Shapes dataset used to bootstrap the SuperPoint training, please do not email us to ask for it.
  • We use bi-linear interpolation rather than the bi-cubic interpolation described in the paper to sample the descriptor as it is faster and gave us similar results.
You can’t perform that action at this time.