Skip to content

Learned Features with ONNX

matlabbe edited this page Nov 3, 2026 · 5 revisions

Experimental, since v0.24.3

RTAB-Map can use learned local features (SuperPoint, ALIKED, DISK, XFeat...) and learned matchers (LightGlue, SuperGlue) exported to ONNX files, run in C++ by ONNX Runtime on the CPU or on a GPU (CUDA, TensorRT, OpenVINO). Unlike the Python-based PyDetector/PyMatcher, no Python or PyTorch is needed at runtime: a model is selected only with its file path.

Component Parameter to select it Model parameter Example models
Feature detector/descriptor Kp/DetectorStrategy=17 (loop closure), Vis/FeatureType=17 (odometry, registration) OnnxDetector/ModelPath SuperPoint, ALIKED, DISK, XFeat
Feature matcher Vis/CorNNType=10 OnnxMatcher/ModelPath LightGlue, SuperGlue

The detector and the matcher can be used independently: a learned detector with the usual nearest neighbor matching, or a learned matcher (e.g. LightGlue for SIFT) with a built-in detector. The developer documentation (model interfaces, export scripts details, implementation) is in corelib/src/onnx/README.md.

Installation

RTAB-Map has to be built from source with ONNX Runtime support (option WITH_ONNXRUNTIME, OFF by default).

  1. Optional, for NVIDIA GPUs: install CUDA and cuDNN in the versions required by the ONNX Runtime release (see its CUDA provider requirements), and TensorRT for the fastest inference:
    sudo apt install tensorrt
  2. Download an ONNX Runtime release archive (onnxruntime-linux-x64-gpu_cuda13-<version>.tgz for CUDA/TensorRT, onnxruntime-linux-x64-<version>.tgz for CPU only) and make its libraries findable:
    curl -LO https://github.com/microsoft/onnxruntime/releases/download/v1.31.1/onnxruntime-linux-x64-gpu_cuda13-1.31.1.tgz
    tar xzf onnxruntime-linux-x64-gpu_cuda13-1.31.1.tgz
    sudo mv onnxruntime-linux-x64-gpu_cuda13-1.31.1 /opt/
    sudo ln -sfn /opt/onnxruntime-linux-x64-gpu_cuda13-1.31.1 /opt/onnxruntime
    echo "/opt/onnxruntime/lib" | sudo tee /etc/ld.so.conf.d/onnxruntime.conf
    sudo ldconfig
  3. Optional, for Intel CPUs, GPUs and NPUs: get the OpenVINO execution provider plugin (no ONNX Runtime rebuild needed), then set Onnx/OpenVINOLibraryPath to /opt/onnxruntime-ep-openvino/libonnxruntime_providers_openvino_plugin.so. Don't add that folder to ld.so.conf: it has its own copies of TBB and hwloc.
    pip install onnxruntime-ep-openvino
    SRC=$(python3 -c "import onnxruntime_ep_openvino, os; print(os.path.dirname(onnxruntime_ep_openvino.__file__))")
    sudo mkdir -p /opt/onnxruntime-ep-openvino
    sudo cp -a $SRC/*.so* $SRC/cache.json $SRC/third-party-programs.txt /opt/onnxruntime-ep-openvino/
  4. Build RTAB-Map:
    cd rtabmap/build
    cmake -DWITH_ONNXRUNTIME=ON -DONNXRUNTIME_ROOT=/opt/onnxruntime ..
    make -j4
    sudo make install
    The CMake summary should show With ONNX Runtime = YES, and rtabmap --version shows With ONNX Runtime: true.

For ROS, rebuild rtabmap_ros from source against this RTAB-Map. In that case, don't build RTAB-Map with Python or libtorch (-DWITH_PYTHON=OFF -DWITH_TORCH=OFF): export the models in a separate terminal with a Python virtual environment (see Exporting models).

Getting models

The quickest start is the pre-exported models of the LightGlue-ONNX releases. Use the separate extractor and matcher files, not the end2end/pipeline ones (RTAB-Map extracts the features of each image once and matches them later against many images):

# SuperPoint + LightGlue (SuperPoint weights from Magic Leap: non-commercial use only)
wget https://github.com/fabio-sim/LightGlue-ONNX/releases/download/v1.0.0/superpoint.onnx
wget https://github.com/fabio-sim/LightGlue-ONNX/releases/download/v0.1.3/superpoint_lightglue.onnx

# DISK + LightGlue (Apache-2.0)
wget https://github.com/fabio-sim/LightGlue-ONNX/releases/download/v1.0.0/disk.onnx
wget https://github.com/fabio-sim/LightGlue-ONNX/releases/download/v0.1.0/disk_lightglue.onnx

A matcher must be used with the descriptors it was trained with. The working combinations:

Detector (OnnxDetector/ModelPath) Matcher License of the weights
superpoint.onnx (Magic Leap SuperPoint) superpoint_lightglue.onnx, or SuperGlue non-commercial
disk.onnx disk_lightglue.onnx Apache-2.0
ALIKED (exported) aliked_lightglue.onnx (exported) BSD-3-Clause
SIFT (built-in, Vis/FeatureType=1) sift_lightglue.onnx (exported) Apache-2.0
XFeat (exported) nearest neighbor (Vis/CorNNType 0-5) Apache-2.0
SuperPoint from rpautrat (exported) nearest neighbor (Vis/CorNNType 0-5) MIT

The two SuperPoint weights are different trainings: the published LightGlue for SuperPoint was trained with the Magic Leap ones, and matches poorly with the rpautrat (MIT) ones. To check a detector/matcher pair, match an image with a shifted copy of itself with rtabmap-matcher: the ratio of inliers over matches should be close to 1.

Exporting models

The other models are exported from their original PyTorch implementations with the scripts of corelib/src/onnx (one per network: SuperPoint, DISK, ALIKED, XFeat, LightGlue, SuperGlue). Each script checks the exported model against the original code. They need PyTorch >= 2.9, in a virtual environment:

python3 -m venv ~/venvs/torch
source ~/venvs/torch/bin/activate
pip install torch onnx onnxscript onnxruntime   # see https://pytorch.org/get-started/locally/ for a CUDA build

cd rtabmap/corelib/src/onnx
pip install lightglue
python3 export_aliked_onnx.py --output aliked.onnx
python3 export_lightglue_onnx.py --features aliked --output aliked_lightglue.onnx
python3 export_lightglue_onnx.py --features sift --output sift_lightglue.onnx

See the README for the requirements and options of each script (detection threshold, maximum keypoints, fixed input size...).

Usage

rtabmap-matcher

rtabmap-matcher shows the features and matches between two images, a quick way to try a model:

rtabmap-matcher --Vis/FeatureType 17 --OnnxDetector/ModelPath superpoint.onnx \
    --Vis/CorNNType 10 --OnnxMatcher/ModelPath superpoint_lightglue.onnx \
    --calibration data/stereo_rect/stereo_left.yaml data/stereo_rect/left/50.jpg data/stereo_rect/left/60.jpg
Time feature extraction (both images): 0.008097s
Time matching:                         0.007500s
Features: from=668 to=676
Matches: 442
Inliers: 333 (Vis/MinInliers=20)

rtabmap (standalone)

Set the parameters on the command line, or in the GUI (Preferences, Advanced view, RTAB-Map Settings):

  • OnnxDetector can be selected as the feature type in Memory > Vocabulary (the bag-of-words dictionary, for loop closure detection) and in Motion Estimation > Visual Registration (visual odometry and registration of the loop closures). Its model path is in Feature > OnnxDetector. The execution provider and its TensorRT/OpenVINO options, shared with the matcher, are in the ONNX panel.
  • OnnxMatcher is selected as the correspondence matching approach in Motion Estimation > Visual Registration, where its model path is set. When a registration has a motion guess (Frame-to-Frame odometry, or proximity detection with RGBD/ProximityOdomGuess=true), set the correspondence window (Vis/CorGuessWinSize) to 0 there: otherwise, the features are matched by projection in a window around their predicted position instead of with the matcher. Global loop closures and the default proximity detections have no guess and use the matcher. When the correspondence window is kept (> 0), also disable Reg/RepeatOnce (true by default): it repeats a registration done without a guess with its result as guess, which would replace the matcher's correspondences by the ones found by projection in the window. With a window of 0, the registration can't use a guess and doesn't repeat.
  • In odometry, OnnxMatcher is only supported by Frame-to-Frame odometry (Odom/Strategy=1); the default Frame-to-Map odometry matches against its local map without it.

With Frame-to-Frame odometry, LightGlue matches the features of the odometry, of the loop closures and of the proximity detections:

rtabmap --Kp/DetectorStrategy 17 --Vis/FeatureType 17 --OnnxDetector/ModelPath superpoint.onnx \
        --Vis/CorNNType 10 --OnnxMatcher/ModelPath superpoint_lightglue.onnx --Vis/CorGuessWinSize 0 \
        --Odom/Strategy 1

With the default Frame-to-Map odometry, keep the default correspondence window: the odometry falls back to its usual matching against its local map, and LightGlue is used for the global loop closures and the proximity detections, which are registered without a motion guess (RGBD/ProximityOdomGuess=false, the default):

rtabmap --Kp/DetectorStrategy 17 --Vis/FeatureType 17 --OnnxDetector/ModelPath superpoint.onnx \
        --Vis/CorNNType 10 --OnnxMatcher/ModelPath superpoint_lightglue.onnx --Reg/RepeatOnce false

The usual feature parameters apply: Kp/MaxFeatures/Vis/MaxFeatures keep the strongest keypoints (or Kp/SSC for a uniform distribution), and the ROI, grid and depth filtering parameters work as with the other detectors. A model is loaded once (and its TensorRT engine built once) even when several RTAB-Map objects use it (odometry, loop closure, registration).

ROS

With rtabmap_ros rebuilt against this RTAB-Map, pass the same parameters, e.g. with a RealSense D435i. args goes to both the odometry and rtabmap nodes, odom_args overrides it for the odometry only. With Frame-to-Frame odometry:

ros2 launch rtabmap_examples realsense_d435i_infra.launch.py \
  args:="--Vis/FeatureType 17 --Kp/DetectorStrategy 17 \
         --OnnxDetector/ModelPath $HOME/superpoint.onnx \
         --Vis/CorNNType 10 --OnnxMatcher/ModelPath $HOME/superpoint_lightglue.onnx \
         --Vis/CorGuessWinSize 0 --Odom/Strategy 1 --Onnx/Provider 1"

With the default Frame-to-Map odometry, LightGlue is used by rtabmap for the global loop closures and the proximity detections:

ros2 launch rtabmap_examples realsense_d435i_infra.launch.py \
  args:="--Vis/FeatureType 17 --Kp/DetectorStrategy 17 \
         --OnnxDetector/ModelPath $HOME/superpoint.onnx \
         --Vis/CorNNType 10 --OnnxMatcher/ModelPath $HOME/superpoint_lightglue.onnx \
         --Reg/RepeatOnce false --Onnx/Provider 1"

Choosing the execution provider

Onnx/Provider selects where the models run: 0=CPU, 1=CUDA (default), 2=TensorRT, 3=OpenVINO. A provider that the ONNX Runtime library doesn't have, or that fails to load (missing cuDNN, no GPU...), falls back to the CPU with a warning in the log.

  • CPU: works everywhere, but learned matchers are slow on CPU (hundreds of ms per pair). On CPU, the default kd-tree matching with the ratio test remains the sensible choice.
  • CUDA (NVIDIA): the simplest GPU option, a few ms per image.
  • TensorRT (NVIDIA): about twice as fast as CUDA. TensorRT compiles each model into an engine for the GPU at the first start, which takes seconds to minutes (a warning is logged). The engines are cached in Onnx/TensorRTCachePath (default: the tensorrt_cache folder of the RTAB-Map working directory), so the next starts are fast. The matcher's engine is built for 1 to Onnx/TensorRTMaxKeypoints (4096) keypoints per image; above, only the strongest keypoints are matched. Onnx/TensorRTFp16 halves the time again, but changes the results slightly and breaks some models (SuperGlue finds almost no match in FP16): compare with and without it on your data before enabling it.
  • OpenVINO (Intel CPUs, integrated and Arc GPUs, NPUs): set the device with Onnx/OpenVINODevice (CPU, GPU, NPU, AUTO). Even on CPU, it is often faster than the default CPU provider on Intel processors. GPU and NPU compile the models at load (cached in Onnx/OpenVINOCachePath) and compute in FP16 by default (Onnx/OpenVINOPrecision=1 for FP32, to get the CPU results).

Typical timings (GeForce RTX 3080, 512x384 images, up to 1000 keypoints, engines already cached):

CPU CUDA TensorRT TensorRT FP16 OpenVINO CPU
SuperPoint (detection, per image) 54 ms 4.8 ms 2.6 ms 1.5 ms 43 ms
LightGlue (matching, per pair) 471 ms 16 ms 9.7 ms 4.7 ms 229 ms
SuperGlue (matching, per pair) 267 ms 21 ms 10 ms broken 120 ms

For comparison, the default FLANN kd-tree matching with the ratio test takes about 15 ms per pair on CPU, for about half the inliers of LightGlue.

Troubleshooting

  • Failed to load ... libonnxruntime_providers_tensorrt.so ... libnvinfer.so.10: cannot open shared object file: the installed TensorRT has another major version than the one ONNX Runtime was built with (e.g. TensorRT 11 on Ubuntu 26.04, ONNX Runtime built with TensorRT 10). Install the TensorRT 10 libraries next to it, e.g. from NVIDIA's Python wheel (adapt sm86 to your GPU architecture):
    cd /tmp && pip download --no-deps "tensorrt-cu13-libs==10.16.1.11" --index-url https://pypi.nvidia.com -d trt10
    cd trt10 && python3 -m zipfile -e tensorrt_cu13_libs-*.whl x
    sudo mkdir -p /opt/tensorrt-10.16
    cd x/tensorrt_libs && sudo cp libnvinfer.so.10 libnvinfer_plugin.so.10 libnvonnxparser.so.10 \
        libnvinfer_builder_resource_sm86.so.10.16.1 libnvinfer_builder_resource_ptx.so.10.16.1 /opt/tensorrt-10.16/
    echo /opt/tensorrt-10.16 | sudo tee /etc/ld.so.conf.d/tensorrt10.conf && sudo ldconfig
    rm -rf /tmp/trt10
  • The provider falls back to CPU: the log tells why (library not found, no device...). Check that the CUDA and cuDNN versions match the ONNX Runtime release.
  • The first frames freeze for a long time with TensorRT: the engines are being built, see the warning in the log. They are cached for the next runs.
  • Few or no matches with LightGlue: check that the matcher was trained with the descriptors of the detector used (see the table in Getting models). With TensorRT, a LightGlue exported with an RTAB-Map script older than #1816 gives wrong matches: export it again.
  • The SuperPoint-LightGlue end-to-end models of LightGlue-ONNX don't load: they take two images and can't be used, use the separate extractor and matcher files.

Clone this wiki locally