Release Notes — Isaac Teleop 1.4
Everything that landed in release/1.4.x since release/1.3.
Streaming & CloudXR
The CloudXR runtime launches in-process, so examples no longer need one started for them
#700, #703, #748, #749, #750, #751, #695, #776
CloudXRLauncher grew argparse helpers and a launch_context(args) entry point, and every shipped example now uses them: python example.py brings up the runtime and WSS proxy itself instead of requiring a pre-started runtime or a sourced environment. --no-launch-cloudxr-runtime attaches to an existing one, and --cloudxr-install-dir / --cloudxr-device-profile / --cloudxr-env-config / --accept-eula are available uniformly. The device profile defaults to Quest3. If you embed Isaac Teleop, this is the pattern to copy — one with CloudXRLauncher.launch_context(args) around your session.
CloudXR moves to 6.3.0, with link quality visible to the operator
Both the Runtime SDK and the Web SDK are on 6.3.0 (from 6.2.0). The bump is API-compatible — everything new is additive — and the client adopts the diagnostics it brings: a 4-bar session-quality indicator on the in-XR HUD, and a Pose Send FPS card that measures the uplink command rate. That second number is what separates "the video is choppy" from "the robot is not receiving my commands", which previously could not be told apart from inside the headset.
The experimental runtime ships alongside the stable one as cloudxr_exp
Wheels now carry a second runtime, isaacteleop.cloudxr_exp, next to the stable one, selected with ISAAC_TELEOP_CLOUDXR_EXP. On Tegra T234 the experimental runtime is preferred automatically because it is the one that performs there; ISAAC_TELEOP_CLOUDXR_EXP=0 forces stable. Source builds still default the bundle off, since the experimental tarball lives on private NGC until a version goes GA.
Ctrl+C tears down the runtime and OpenXR session in the right order
Shutting down a CloudXR-backed session used to produce a flood of Broken pipe and XR_ERROR_RUNTIME_FAILURE errors and could leave a container unusable: the runtime process died before the OpenXR handles it served were released. The signal handler now propagates to the prior handler first so with-block teardown unwinds inside-out, and OpenXRSession releases its handles deterministically at context-manager exit rather than at garbage collection. SIGTERM, SIGINT and atexit all stop a session-scoped runtime, and main-thread join is selected automatically on Orin.
The WSS proxy can be turned off
If you already run your own secure WebSocket proxy, the launcher's built-in one can now be disabled instead of failing with a port-in-use error when an example starts. Useful for any deployment that terminates TLS ahead of Isaac Teleop.
Devices, Plugins & Trackers
reBot DevArm leader arm plugin, with Damiao and RobStride backends
The Seeed reBot DevArm (6-DOF plus gripper) streams as JointStateOutput on the same generic joint-space path the SO-101 leader plugin uses, so existing joint-space consumers pick it up unchanged. Both hardware builds are supported and inferred from the device argument: a path selects the Damiao dm-serial bus, a bare SocketCAN interface name such as can0 selects RobStride. Neither backend needs a vendor SDK, and a synthetic fallback lets the plugin run headless in CI.
OGLO tactile glove device
A complete new device: BLE client, packet parser, plugin, tracker, schema and example. It follows the standard plugin layout, so it is also a reasonable reference if you are adding a BLE glove of your own.
Vendor-neutral haptic feedback, with Manus gloves as the first consumer
Device output now has a transport, not just device input. A generic HapticCommand schema moves over XR_NVX1_push_tensor via TensorPushTracker / HapticCommandReaderTracker, so any out-of-process haptic device — glove, exoskeleton — reuses one wire format instead of adding a per-vendor tracker and FlatBuffer. Manus glove haptics is the first device wired through it.
Generic SE3 tracker device type
Any 6-DoF pose source can now be a first-class device without a bespoke schema. Se3TrackerPose is built on core.Pose, with units, Hamilton (x, y, z, w) quaternion order, reference-frame rules and two-level validity documented normatively in the schema itself. The tracker stack mirrors the joint-state pattern (live and replay implementations, recording traits, Python bindings), and an example controller plugin shows the producer side.
Full-body tracking is vendor-neutral, with the vendor chosen at session level
The full-body schema and tracker dropped their Pico-specific names, and the vendor is now selected per source on the session rather than baked into the type you instantiate. Recordings are unaffected — the wire format and enum values are identical. See Breaking Changes for the rename and for the new replay-mode restriction.
The Manus plugin exposes raw sensor data
Alongside the retargeted skeleton, the plugin can surface the glove's raw sensor values, which is what you want for calibration work, glove-side debugging, or training on unprocessed signals. Naming was standardized on "MANUS" across code and docs at the same time.
Rendering & Visualization
ProjectionLayer brings full-view RGBD content to Televiz
Televiz can now display full-view RGBD content — gsplat, nvblox, neural reconstruction — through a new ProjectionLayer, with color and depth composited so quad layers Z-composite correctly against it. The frame loop was reworked in the same change: VizSession::begin_frame now returns the real per-eye XR poses from xrLocateViews instead of placeholder identity views, so a renderer that renders against info.views is pose-correct by construction. Renderers that relied on the old placeholder behavior were rendering against the wrong pose and should be re-checked.
QuadLayer can submit native OpenXR quad layers
Opt in with QuadLayer::Config::use_openxr_quad_layer and the quad is submitted as a real XrCompositionLayerQuad instead of being composited into the shared render target. When every visible layer is a native quad, the shared projection layer is dropped entirely, which lets the runtime engage its quad fast path and client-side reconstruction — the difference shows up as sharper text and lower latency on streamed UI panels. Window and offscreen modes ignore the flag.
Televiz builds by default and installs from PyPI
BUILD_VIZ now auto-detects: on when Vulkan, the CUDA Toolkit and glslangValidator are present, off otherwise, so a core-only source build still configures. More importantly, camera_viz setup installs isaacteleop from PyPI — the published wheel bundles the viz module — so trying Televiz no longer requires a source build at all. Source builds become an opt-in development path.
camera_viz gains video-file replay, an XR-first default, and opt-in RTP
A new type: video frame source plays a recording through the normal FrameSource contract, so the whole camera-streaming path can be exercised with no camera attached — including the NVENC/RTP chain — and stereo: true splits side-by-side recordings per eye on the GPU. Display now defaults to XR rather than a desktop window, and RTP became opt-in via --with-rtp so direct-mode setup no longer pays for the GStreamer and PyGObject build (both are behavior changes; see Breaking Changes). A Dell 4MP webcam V4L2 config ships for desktop testing.
Client & Web UI
Record and replay XR input from the browser client
The web client can capture a scoped XR input trace and play it back, with a trace visualization and a recorder UI. Replay is time-paced by default, with the older frame-paced mode still available, so a recording made on one headset replays at real-world speed on a machine with a different frame rate. Loading a recording now reports success or a specific reason for rejection instead of surfacing a raw SyntaxError, and traces recorded on runtimes that omit XRFrame.predictedDisplayTime (PICO among them) carry finite timestamps rather than silently recording nulls.
Host workstation notices reach the operator in-headset
The client already ran a receive loop on the teleop_command channel, but the handler was an empty stub, so anything the host sent was dropped. Typed messages are now dispatched, and a system_notice renders as a dismissible banner in XR and mirrors to the 2D status box. Isaac Lab uses this to tell the operator their workstation is below the recommended spec — previously that warning only reached a terminal nobody can see while wearing a headset, leaving a low frame rate unexplained.
Debug settings are grouped, resettable, and documented
The flat list of roughly sixteen controls is now Device Profile up front with the rest folded into six collapsed groups under an Advanced divider. A "Reset to defaults" button clears global and per-project localStorage plus form-backed URL params, which is the fastest way out of a client wedged by a bad saved profile. Accepted URL query params are documented in the UI instead of only in source; element IDs are unchanged.
The out-of-box hub can start without adb
Set TELEOP_OOB_HUB_ONLY and --setup-oob starts the hub while skipping every adb step and the CDP automation, for containers, CI and wireless-only setups where adb is not available. The operator opens the teleop URL on the headset manually with oobEnable=1&serverIP=...&port=.... Combining it with --usb-local is rejected outright, since that path needs adb reverse.
ROS 2 Integration
teleop_ros2 is the supported ROS 2 reference publisher
Docs and package metadata now present teleop_ros2 as a reference integration consumed by Isaac ROS Teleop, not a disposable sample you are expected to rewrite — its path is unchanged, but you can build on it. The example also serves the CloudXR web client itself, so a ROS 2 bring-up no longer needs a separate process to get a headset connected.
Hand and end-effector poses publish as named pose arrays
xr_teleop/hand and xr_teleop/ee_poses moved from geometry_msgs/PoseArray to a new teleop_ros2_interfaces/msg/NamedPoseArray. Entries are addressed by name and the arrays have a fixed shape — 25 left_ joints then 25 right_, including WRIST and omitting PALM; left and right for end effectors — with invalid entries carried as zero poses flagged is_valid=false rather than being omitted. Subscribers no longer have to infer which index is which side from publication order, or guess why an index vanished. This is a breaking change; see below.
Runtime profiles are centralized and message building is split from publishing
Mode-specific configuration moved into one teleop_profiles module, and root-command, msgpack and TF construction moved out of TeleopRos2Node so publisher methods only perform ROS side effects. If you fork the node to add a mode or change a message, this is the seam to work against, and message construction is now unit-testable without a ROS graph.
Build, Packaging & Platform
isaacteleop.rig launches a whole teleop rig from one YAML file
A rig — the CloudXR runtime plus producer plugins plus consumer apps — comes up in a tmux session from a single declarative file, replacing per-demo shell scripts. The rig file is the only source of configuration: parameters live in its params: block and are substituted into every command, with no CLI override knobs by design, so the file you ran is the run you can reproduce. Unknown keys are hard errors, and a lint warns when a Python consumer would self-launch a second CloudXR runtime. Shipped rigs cover the SE3 tracker and full-body examples.
The teleop_ros2 image drops its nlopt source build
PyPI now publishes Linux aarch64 wheels for nlopt, so the Docker image's nlopt_wheel stage and its find-links plumbing are gone. Image builds on aarch64 get noticeably shorter, and the aarch64 source-build instructions are removed from the docs.
Examples, Rigs & Docs
C++ full-body tracking examples and a full-body rig
full_body_printer reads the 24 body joints directly, and record_full_body.cpp is a C++ counterpart to the Python recorder producing files that replay_full_body.py reads unchanged — so the full-body path is now demonstrated in C++ end to end, not only Python. The full_body rig launches the whole session with one command; it is the first consumer-only rig, with no producer plugin. Rig recordings land in the shared, gitignored recordings directory that the replay workflow already searches, and the recorder's output argument accepts a directory, writing a fresh timestamped file per run.
Live pose visualization for DeviceIO and MCAP streams
live_hand.py, live_controller.py and live_full_body.py show XR pose data in viser in real time with no MCAP round-trip, and a dedicated DeviceIO live-view example covers the device path. The shared HandViz / ControllerViz / FullBodyViz classes moved into common.py, so the replay scripts and the live ones draw from one implementation.
Vive teleop MuJoCo visualization examples
Two MuJoCo passive-viewer samples consume the HeadTracker / ControllerTracker / HandTracker streams: one draws the HMD and controllers as mocap bodies, the other draws both hands as 26-joint skeletons. Both do the OpenXR-to-MuJoCo conversion you will otherwise write yourself — Y-up to Z-up, and quaternion reorder from xyzw to wxyz.
LeRobot and SO-101 getting-started guides
A getting-started guide covers LeRobot and the SO-101 arm, the data-collection guide is synced with upstream and uses the hf CLI for login and dataset ids, and the SO-101 leader sim workflow is documented. The exact LeRobot checkout the real-data examples require is now written down, which was previously the most common way to get stuck.
Notable Fixes
The runtime crashed the instant a headset connected
The runtime worker imports asyncio, which loads the system OpenSSL before the native streaming stack dlopens its bundled libssl_nvst. Two OpenSSL builds in one process segfaulted during the client's WebRTC DTLS handshake — so idle waiting looked fine and connecting always died. The worker is now spawned with LD_PRELOAD pointing at the bundled libraries.
Se3 retargeters crashed when a controller lost tracking
A connected controller resting on a table reports a valid input group with grip_is_valid=False and a zero-norm orientation, which reached Rotation.from_quat and raised. In pipelined mode this killed the async retarget worker permanently, forcing a session teardown and restart. The controller branch now holds the last pose, matching how the hand branch already handled invalid joints.
Left-hand glove haptics never fired
The reader collapsed every drained HapticCommand into a single latest slot, so with a left and right endpoint pushed each frame only the last one survived — the CLI showed both being sent and one hand stayed silent. Samples are now bucketed per endpoint, with a new get_data(session, endpoint) accessor; the existing single-argument form is unchanged for single-endpoint consumers.
Quest profiles shipped a frame rate and bitrate the link could not hold
Stock Quest profiles asked for 90 FPS and 150 Mbps, above the 100 Mbps ceiling StreamSDK guidance warns about for a wireless link. Defaults are now 72 FPS / 25 Mbps for Quest 3 (inherited by 3S and 2), and 25/50 Mbps options were added to the bitrate select, since a value with no matching option silently fell back to the built-in default. The configured frame rate is also applied and confirmed before the CloudXR session is created, so the rate advertised to CloudXR is the one the browser actually granted.
The out-of-box static client failed to load its lazy chunks
The OOB static client did not sync or serve webpack's lazy chunks, so the bundle could not finish loading in exactly the setup used for first-run bring-up.
Examples resolved a stale 1.3.9rc1 wheel
The mcap and DeviceIO examples pinned isaacteleop[cloudxr]==1.3.9rc1, a frozen pre-release from another series, so uv sync && uv run died with an ImportError before the script started. Examples now track their own release line, and camera_viz resolves the series from the checkout's VERSION rather than an open-ended floor that could drift onto a newer mainline RC.
camera_viz kept running after its managed runtime died
camera_viz now stops when the CloudXR runtime or WSS proxy it started exits unexpectedly, instead of continuing against a dead stream. The same change makes XR-mode setup install its WSS dependencies for both the PyPI and local-wheel paths.
WSS certificate generation failed with a misleading error
A certificate directory created by root but not writable by the running user produced "No such file or directory" from a swallowed openssl subprocess. Certificates are now generated in-process with the cryptography package — which also sidesteps dynamic-linker conflicts from the caller's LD_LIBRARY_PATH — and an explicit writability check reports the real problem.
accept_eula=False was ignored
launch_context always fell back to args.accept_eula, so an explicit False could not force-disable acceptance. It is now honored.
Breaking Changes
ROS 2 hand and EE pose topics changed message type
xr_teleop/hand and xr_teleop/ee_poses publish teleop_ros2_interfaces/msg/NamedPoseArray instead of geometry_msgs/PoseArray. Add a dependency on the new teleop_ros2_interfaces package, rebuild subscribers against it, and read entries by name instead of by index — poses[0]/poses[1] for left and right end effectors no longer applies. Invalid entries are now present with is_valid=false rather than absent, so drop any logic that inferred validity from array length.
Full-body schema types dropped their Pico suffix
FullBodyTrackerPico is FullBodyTracker, FullBodyPosePicoT is FullBodyPoseT, BodyJointPico_* are BodyJoint_*, and deviceio_trackers/full_body_tracker_pico.hpp is gone. Rename to the generic identifiers; to stage the migration, include <schema/full_body_compat.hpp>, which aliases every old name with a [[deprecated]] warning. The wire format and enum values are unchanged, so existing recordings round-trip either way. Separately, setting a vendor on a source is now rejected in SessionMode.REPLAY — replay reads the recorded channel regardless of vendor, so remove the vendor selection from replay paths.
camera_viz RTP is opt-in via --with-rtp
The flag flipped from --no-rtp to --with-rtp, matching --with-zed, because only split mode needs the GStreamer stack, the PyGObject source build and the native codec. Add --with-rtp to any setup that uses split mode or a source: rtp config; --sender-only implies it. Direct-mode setups need no change and get faster.
camera_viz and Televiz default to XR display
display.mode defaults to XR when a config omits it, and the shipped configs set XR explicitly. If you were relying on the desktop window — headless CI, a machine with no XR runtime — pass --mode window or set the mode in your config.
MCAP example launcher flags were renamed
The six mcap record/live scripts now take the shared launcher CLI: --install-dir is --cloudxr-install-dir, and --env-file is --cloudxr-env-config with no default. examples/mcap_record_replay/python/default.env was deleted — it only pinned NV_DEVICE_PROFILE=Quest3, which is already the --cloudxr-device-profile default. Update scripts and CI that pass the old flag names.
The experimental CloudXR runtime is bundled by default and auto-selected on Tegra
Builds package isaacteleop.cloudxr_exp unless you pass -DENABLE_CLOUDXR_EXP_BUNDLE=OFF, and on Tegra T234 the experimental runtime is chosen automatically and fails if missing. If you need the stable runtime on Tegra, set ISAAC_TELEOP_CLOUDXR_EXP=0.
Source builds enable Televiz automatically
BUILD_VIZ auto-detects instead of defaulting off: it turns on when Vulkan, the CUDA Toolkit and glslangValidator are all present. A build machine that has them will now build the viz module and take longer; pass -DBUILD_VIZ=OFF to keep the old behavior.
run_cloudxr_via_docker.sh was removed
The script had been deprecated with a removal warning, and scripts/check_cloudxr_eula.sh went with it since nothing else called it. Launch the runtime through the Python launcher, which handles the EULA itself — in most cases that means deleting the script invocation rather than replacing it, because examples now launch the runtime in-process.
examples/camera_streamer was removed
The Holoscan sample is superseded by the Televiz-based camera_viz sample. Port to camera_viz; its own camera_streamer.py sender is unrelated and still present.
Contributions require a DCO sign-off
A DCO workflow now gates pull requests, so every commit needs a Signed-off-by line. Use git commit -s, and git rebase --signoff to fix a branch that predates this.