Develop v2.7

This has been one of the larger releases with significant changes to the propagation to support IMU intrinsics and analytical integration. The documentation has been split into the original discrete derivations and analytical with IMU intrinsics to hopefully allow for comparison and learning between these two.

The IMU intrinsics code was released in relation to the following publication:

Yang, Yulin, Patrick Geneva, Xingxing Zuo, and Guoquan Huang. "Online Self-Calibration for Visual-Inertial Navigation: Models, Analysis and Degeneracy." IEEE Transactions on Robotics, 2023.
https://pgeneva.com/downloads/preprints/Yang2023TRO.pdf

Bug Fixes

• Fix #333 in ov_eval utility
• Stereo tracking improvements
  • Fix #332 chance reject of features out of bounds
  • Fix bad masking logic, so features should be detected more reliably
• Fix documentation #340 of feat init optimization Jacobian

Key Changes

• IMU intrinsic support has been added
  • Support corrections to IMU readings using intrinsics
  • Online calibration of parameters
  • Simulator support to verify performance
• Updated documentation
  • Discrete and analytical IMU propagation docs have been separated out
  • FEJ docs has been expanded
  • A few other improvements throughout the codebase

Example fixed masks in stereo tracking:

openvins_stereo_masks.mp4

Develop v2.6.3

In general a bunch of smaller changes and bug fixes. Thanks to all the contributors that opened issues and PRs to address the problems. We also have recently released a monocular plane-aided VINS, termed ov_plane, which leverages the OpenVINS project. We hope that those interested can check it out.

Bug Fixes

• fix: incorrect zero-velocity bias evolution code in 3861010
• fix: link errors by @FinleyPan in #272
• fix: display problem if the track length of feature is very long by @CanCanZeng in #267
• fix: bug in get_oldest_time from FeatureDatabase by @wang-lq in #287
• fix: Segmentation fault in State.h by @pritzvac in #306
• fix: newer matplotlib version in 0429390
• fix: pointcloud publishing extra (0,0,0) features in 70b0ae3
• fix: gram-schmidt bug if gravity was along the e1 vector in b397178
• fix: using sample config info on KAIST (see #300 and d171a03)

What's Changed

• release mode is no longer the default CMAKE mode to allow for debugging using native asserts
• cleanup some ROS-free install location logic by @goldbattle in #318
• cache our fast propagate values so we don't reintegrate by @goldbattle in #321
• incremential feature triangulation for active features in 8600cee
• use measurement compression for zero velocity update by @ccchu0816 in 01fb6e9
• example T265 and D455 realsense configuration files in 4ed7927
• support for the Indoor AR Table Visual-Inertial Datasets in cd781f9 (see documentation website)

New Contributors

• @FinleyPan made their first contribution in #272
• @CanCanZeng made their first contribution in #267
• @wang-lq made their first contribution in #287
• @pritzvac made their first contribution in #306

Full Changelog: v2.6.2...v2.6.3

Develop v2.6.2

• Fixes covariance init bug #262
• Do not use global publish namespace, use the specified one (both ROS1 and ROS2)
• Fix ROS2 IMU subscribe not being sensor QoS rclcpp::SensorDataQoS()
• NEW: Allow specifying the number of threads for parallelization / OpenCV (removes the multi_threading config value)
• NEW: Histogram timing plot to allow better insight

Develop v2.6.1

• Bug fix accidental deletion of IMU when time tracking was lost (fixes #246)
• Fix ROS2 param reading in visualizer (fixes #249)
• Fix bug that the first ever IMU reading is not past into the VioManger
• Dynamic initialization is now disabled by default on most datasets
• Now use predeclared classes and move headers to cpp files to improve compile time
• Rename ov_init simulator to SimulatorInit to prevent name conflicts
• Faster feature extraction logic by only extracting in sub-grids which need features (150hz for mono, 60hz for stereo)
• New simplified serial reader which doesn't use iterators
• Publishing of feature track information (image) is now in a separate thread and at 20hz. This visualization can take a while depending on image size and number of features. This means more multi-threading mutexes were added to the tracker classes.
• Configuration and scripts were updated for those which use OpenVINS as a baseline

Develop v2.6 - Dynamic Initialization and Async Publishing

Key Changes

• Dynamic initialization has been added alongside the current static one.
  • New dependency requirement of ceres-solver to support the MLE refinement
  • Recovered covariance still requires some inflation (likely due to being close to degenerate motion)
  • Tested on existing datasets, seems to work reasonable well, but please let us know
  • Closes #204 and #222
• The frontend now has two threads. The IMU will always publish the state odometry at its rate, along with recording the IMU readings. This should help deployment on real robots were the latency is key.
• Odometry publish is now in the local frame to match the ROS spec (fixes #224)

Dynamic initialization is an implementation based on:

Dong-Si, Tue-Cuong, and Anastasios I. Mourikis. "Estimator initialization in vision-aided inertial navigation with unknown camera-IMU calibration." 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.

• https://ieeexplore.ieee.org/abstract/document/6386235
• https://tdongsi.github.io/download/pubs/2011_VIO_Init_TR.pdf
• https://pgeneva.com/downloads/reports/tr_init.pdf

Demo Youtube Video:
https://www.youtube.com/watch?v=eSQLWcNrx_I

demo.mp4

v2.5 - New Config System, ROS2 support, and more.

Key Changes

• Yaml-based configuration system has been added to support easy ROS1, ROS2 and ROS-free version. The default values can be overridden using ROS parameters allowing for easy integration.
• Yaml sensor files are now directly read (e.g. from Kalibr) to facility faster integration with new sensors.
• Support for ROS2 building and publishing with live sensors (subscribe). Documentation has been updated to include instructions on how to build and run datasets.
• Build system will now link to Python3 for plotting utilities in a more robust manor.
• Static initialization now checks disparity to ensure the system is stationary.
• Internal logging levels with print statements have been added #197
• Docker files for ROS1 and ROS2 for Ubuntu 16.04, 18.04, and 20.04
• NOTE: CMake version needs to be >3.12, which can be installed using the following:
• NOTE: https://docs.openvins.com/gs-installing.html#gs-install-cmake

Other Changes

• Store camera width and height in the camera objects
• Allow for tracking of dynamic number of features. This is leveraged during initialization to reduce load.
• Default config files now have a larger grid to allow for more uniform feature extraction.
• SO(3) and SE(3) log functions have been changed to try to be more accurate
• Image transport now used so that CompressedImages can be subscribed to see images over wifi
• Fixed aruco feature publishing ids
• Simulator now uses the camera classes when generating measurements
• Other documentation fixes (e.g. #206)

v2.4 - Camera classes, Masks, Formatting and Smaller Fixes

Key Changes

• Camera class objects have been added which consolidate both the distortion / undistortion / Jacobian functionality
• The frontend has been reworked to take in this new camera object and the "CameraData" message objects directly
• Support for masked areas of the image has been added to the tracking (closes #159)
• Docker use and documentation has been added (originally mentioned in #167)
• Fix radtan norm intinsic jacob (fixes #169)
• Add live alignment of groundtruth if we have it (for better dev'ing)
• Dataset support for KAIST VIO mav dataset
• Fixed bug in serial reader for stereo were dropped frames would cause de-sync of left and right timestamps
• Remove "general" camera support, for more than 1 stereo pair you will need to implement your own logic, binocular should still support arbitrary if they are all sync'ed

v2.3 - Memory Management, Active Feature Publishing, Small Fixes

Key Changes

• Switch to using shared_ptr in the majority of places (fixed a lot of memory leaks), see #96
• Small fix to propagator for edge case where IMU measurements are lacked (should hopefully address #87, #89, #90)
• Support for periods in folder names when using ov_eval error_comparison
• Fix ov_eval RPE trajectory selection for low frequency groundtruth files
• Added option to do 1d triangulation, update triangulation documentation
• Update propagation doc to most closely follow Trawny's report
• Added docs on profiling the memory and complexity of the codebase
• Publish "active tracks" which are all features which are currently seen from the cam0
  • Can be used for loop closure and other XR applications.
  • All possible features are re-triangulated at each timestep to generate this active pointcloud / sparse depthmap.
• Option to limit number of MSCKF feature updates
  • Addresses large "spikes" in computation when a bunch of features are lost
  • Allows for tracking of more points then what are used in estimation
• More arbitrary camera subscriber logic
  • Enables multi-camera subscriptions with mixes of stereo pairs
  • All cameras are in same clock frame (might change in the future to per-camera time offset)
• Camera images are now processed ASAP to reduce latency (thanks @nmerrill67 )
• Continuous integration building to enable support for ubuntu 16.04, 18.04, and 20.04 and opencv 3-4 (thanks @nmerrill67 )
• On startup 4 dof gauge freedoms now fixed to zero uncertainty on startup
• Option to run in true single-threaded mode with no parallelization in the frontend stereo tracking.
• Visualization of the active SLAM points as green squares on the image now to help show where SLAM features are being tracked.

Validation

I ran a few runs on the ETH eurocmav dataset to compare performance vs previous releases. While there is no guarantee, I believe we don't have any major regressions in accuracy. Default levels of computational cost have gone up due to tracking of more points to have denser pointclouds. This can be tuned as necessary for any specific platform.

Vicon2gt Utility

We have also released a package to facilitate the generation of groundtruth trajectories.

This utility was created to generate groundtruth trajectories using a motion capture system (e.g. Vicon or OptiTrack) for use in evaluating visual-inertial estimation systems. Specifically we want to calculate the inertial IMU state (full 15 dof) at camera frequency rate and generate a groundtruth trajectory similar to those provided by the EurocMav datasets.

Please check it out here: https://github.com/rpng/vicon2gt

v2.2 - Zero Velocity Update Support

• Zero velocity update logic detection and update based on chi2 with the current IMU readings
• Updated documentation pages for zero velocity update
• Standstill initialization if zero velocity update is enabled (great for if you need a pose estimate as soon as possible)
• Support for using stereo on the KAIST Urban Dataset
• See PR #79 for more details and example runs
• Some matlab utilities for converting from their groundtruth format
• Some additional collected datasets of driving with the sensor mounted on a car

v2.1 - Example Secondary Pose Graph

Bug Fixes:

• Properly set FEJ value after we initialize the state, see #53
• State augmentation logic for time offset which caused under-confident covariance, see #69
• Fixed how RPE evaluator selects distance segments to ensure we select the maximum possible set. It is now based on a fixed length instead of varying as the segment length increases, and will warn if distance between groundtruth poses is too far, see commit 8ae1cb6.
• Since depth segmentation fault, see commit 9e3faeb
• Fixed RosVisualizer not being built into the shared library if we are building with ROS, see commit 18116ca

Key Changes / New Features:

• Publish marginalized pose and feature track information for ov_secondary
• Details on secondary pose graph example works can be seen in #66
• Make publishing TF optional, see #55
• Support for downsampling cameras by half if you have a high resolution camera input or need to save compute, see commit 18bd54f

Minor changes:

• Some documentation updates
• Update uzh-fpv launch file