You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
ILLIXR v4.0 (2025-03-28)
Note: This release contains bugs in the hand tracking and zed plugins. These have been addressed in v4.0.1.
Release Notes
This release of ILLIXR brings many new features and capabilities. A good potion of the code base has been refactored/reorganized to both standardize the code and group similar parts together. There were a total of 16 pull requests merged which introduced 10 new plugins, overhauled several others, and addressed 13 issues. The new plugins include remote rendering, hand tracking, and ORB_SLAM. Vulkan is now the default display back-end, and due to this, some of the OpenGL based plugins may not work on every system. See below for more detailed notes on the changes.
Plugins
Introduces a new ORB_SLAM3 based plugin. By @hungdche in PR #407
Deployed four new plugins: hand_tracking, webcam, hand_tracking.viewer, and zed.data_injection, reworked the zed plugin, and a zed_capture executable.
hand_tracking: detects and reports the positions of joints and fingertips; if depth information is available, or distances can be calculated by parallax, the reported coordinates are real-world in millimeters, otherwise they are in pixel coordinates
webcam: captures images from a webcam and publishes them to the webcam topic
hand_tracking.viewer: gui based interface for visualizing the output of the hand_tracker in real time
zed.data_injection: reads in data from disk and publishes it as if it were the zed plugin, good for debugging and testing
zed: now publishes information about the camera (pixels, field of view, etc.) which is required for some modes of depth sensing; also now publishes the pose corresponding to the current images; these are in addition to the images and IMU information it has always published
zed_capture: a standalone executable used to capture the images and poses from the zed camera and write them to disk (currently does not capture depth information); the products of this executable are intended to be used by the zed.data_injection plugin
By @astro-friedel in PR #428
The hand tracking plugin now can be used as an OpenXR API Layer, where it will intercept calls to get hand tracking data. It has been tested with Monado, but should be generally compatible. Adheres to OpenXR-1.0 standards. By @astro-friedel in PR #436
Deployed the following plugins:
tcp_network_backend: a backend for Switchboard using TCP to transmit topics between two connected ILLIXR instances
offload_rendering_server: encodes OpenXR frames with FFMPEG and transmits them across the Switchboard network backend
offload_rendering_client: decodes frames transmitted from an offload_rendering_server instance and adds decoded frames to a buffer pool
openwarp_vk: an implementation of OpenWarp, 6-DoF asynchronous reprojection, in ILLIXR
lighthouse: uses libsurvive to connect with Valve Index lighthouses for head tracking native_renderer has been reworked so that external applications can run asynchronously with reprojection.
By @qinjunj, @Jebbly, @shg8 in PR #437
Infrastructure
File loading is used by several plugins, each with their own code. The file loading code has been centralized into data_loading.hpp. By @astro-friedel in PR #416
Fixed an issue where plugins needed to be specified in the correct order, or they would throw an exception. By @astro-friedel in PR #417
Fixed an issue when changing a profile name and re-running CMake. By @astro-friedel in PR #418
The Runge-Kutta code was replicated several times in the code base. This code has now been centralized and rewritten to make it compatible with the current ILLIXR license. By @astro-friedel in PR #421
Updated the tcp socket code so that it can be distributed under the current ILLIXR license. By @williamsentosa95 in PR #424
The hand tracking plugin introduced a large number of breaking and non-breaking changes to the base structures used by ILLIXR.
breaking: the cam_type structures have been completely overhauled
all now inherit from a cam_base_type base struct, allowing for the cam_base_type to be used as a function argument type, letting a single function handle several different types of image input
all cam_base_type based structs are now defined in the common headers (under include/illixr) rather than in some plugins
there are now the following types: binocular_cam_type, monocular_cam_type, rgb_depth_type, depth_type, and cam_type_zed
breaking: the header data_format.hpp was getting too large and complex; it has been broken up into multiple headers under the data_format directory, with all struts and functions inside the ILLIXR::data_format namespace
camera_data.hpp: structures used to describe the physical characteristics of a camera (# pixels, field of view, etc.)
coordinate.hpp: enums for labelling the coordinate reference axes (e.g. LEFT_HANDED_Y_UP, RIGHT_HANDED_Y_UP, RIGHT_HANDED_Z_UP_X_FWD)
frame.hpp: the rendered_frame struct
hand_tracking_data: structs for holding positional information of points of a hand
imu.hpp: all imu related structs
misc.hpp: definitions that didn't fit with anything else
opencv_data_types.hpp: most cam_base_type structs
point.hpp: various structs for holding information about a point in 3D space
pose.hpp: all pose related structs
shape.hpp: structs related to shapes (e.g. rectangle)
template.hpp: common templated functions
unit.hpp: enums and functions related to units of measure
zed_cam.hpp: the cam_type_zed struct; this is kept separate from the other cam_base_type due to potential dependencies on ZED headers, which not all systems have
The pose structures were expanded to hold additional information used by the hand_tracking code, these additions do not affect current uses of these structures
By @astro-friedel in PR #428, #440
Environment variable handling has been introduced to the switchboard. Calls to std::getenv have been replaced with calls to swithchboard->get_env, switchbaord->get_env_char, or switchbaord->get_env_bool and calls to std::setenv have been replaced with calls to switchboard->set_env. This change allows the switchboard to act as a broker for all environment variables. Additionally, environment variables can now be specified on the command line or in a yaml file, and will be made available to all plugins via the switchboard. By @astro-friedel in PR #430
Misc
Breaking: Policies for contributing to the ILLIXR code base, as well as a general coding style guide have been introduced. The entire code base was overhauled to bring it into compliance with the style guide. By @astro-friedel in PR #419
Updated CMakeLists.txt to cleanly handle newer FindBoost capabilities, updated tags on some external packages, cleaned up CMake output, added some include lines required by newer gcc compiler versions, and added all.yaml profile for testing that the code fully builds. By @astro-friedel in PR #431
Updated several external packages ILLIXR builds from source to newer versions. Official support for CentOS, Fedora, and Ubuntu 20 and below has been dropped, support for Ubuntu 24 has been added. By @astro-friedel in PR #439
