Skip to content

v1.3.131

Choose a tag to compare

View all tags
@jiwenc-nv jiwenc-nv released this 30 Jun 20:09
· 255 commits to main since this release
Immutable release. Only release title and notes can be modified.
v1.3.131
7002ed6

This release expands Isaac Teleop from a CloudXR/WebXR teleoperation core into a multi-device, multi-robot platform: new dexterous-hand and arm retargeting, data gloves and exoskeletons, a ROS 2 reference integration, ARM64 support, a record/replay data pipeline, a preview Vulkan visualization stack, and a substantially streamlined CloudXR operations story.

Top 10 new features

1. SO-101 arm teleoperation — leader arm + XR controller (LeRobot ecosystem)

Two complementary ways to drive the low-cost SO-101 5-DOF arm:

  • Generic joint-space device + SO-101 leader arm (#660): a reusable device path (name-keyed JointStateOutput schema → JointStateSource → JointStateRetargeter) with joint (leader→follower mirror) and ee_pose (URDF forward-kinematics) modes — reusable for any joint-encoder device (leader arms, exoskeletons, gloves). The reference so101_leader plugin reads the 6 Feetech STS3215 servos over serial with no SDK dependency, wire-compatible with LeRobot's FeetechMotorsBus, and ships a calibrate subcommand that mirrors lerobot-calibrate (and can emit LeRobot-format calibration files).
  • SO-101 XR-controller retargeters (#651, hardened by #692): drive an SO-101 follower from an XR controller in Isaac Lab (Isaac-Stack-Cube-SO101-IK-Abs-v0) — clutch-rebased absolute EE pose (engaging never jerks the arm) + proportional analog-trigger gripper; #692 gates the clutch on grip-pose validity to prevent NaN/IK crashes on tracking loss.

Maturity: the leader / joint-space device is a complete reference implementation (plugin, retargeter, calibration tool, docs, example, MCAP-replay backend); the SO-101 XR-controller retargeters are experimental (tune-in-sim). No LeRobot package dependency — SO-101 is a LeRobot-ecosystem arm; a separate examples/lerobot dataset integration predates this release.

2. Sharpa Wave dexterous-hand retargeters

Map OpenXR hand tracking onto the Sharpa Wave hand via two interchangeable backends (#449, #474, #484, #490, #491): dexpilot (default for hand teleop) wrapping the open-source dex-retargeting DexPilot optimizer with SHA-pinned official URDFs, and pink_ik — a Pinocchio/Pink IK path ported from V2D (robotic_grounding). Bimanual combiners and ROS hand-mode integration with output joint-name aliasing.
Maturity: mixed — dexpilot runs on open-source deps; pink_ik requires V2D's robotic_grounding (bundled as a temporary bridge). dexpilot emits a fixed 22-joint set; pink_ik's DOF count is model-dependent.

3. Teleoperation with Isaac ROS (ROS 2 reference) — teleop_ros2

A single-node rclpy publisher (ROS 2 Humble) that streams controller, hand, head, end-effector, locomotion, and retargeted finger-joint data over standard topics. Four selectable modes (controller_teleop, hand_teleop, controller_raw, full_body); standard ROS remapping (#343); foot-pedal locomotion (#395); configurable TF frames, finger-joint names, and config/asset roots (#368, #507); in-process CloudXR via CloudXRLauncher (#609); head-pose publishing (#618); deterministic MCAP-fixture integration tests.
Maturity: preview. This ships under the Isaac ROS umbrella (the web-client picker has an isaacros entry; the quick start has a "Teleoperation with Isaac ROS" card pointing here) as a ROS 2 reference publisher; the broader Isaac ROS CUDA-accelerated framework remains tracked as "Planned."

4. Egocentric Hand Reconstruction (codename "Ego4Robo")

A post-processing pipeline (src/postprocessing/egocentric_hand_reconstruction, #363/#417/#672/#599) that recovers world-frame 4D hand + camera motion from a single egocentric video by chaining two containerized models — ViPE (camera pose/depth) → Dyn-HaMR (MANO hand mesh). Exports per-track MANO meshes/joint trajectories, "jumpy-track" quality control, and an OSMO cloud-batch workflow — turning head-cam video into demonstration data for imitation learning.
Maturity: experimental (v0.0.1; requires user-supplied licensed MANO/BMC assets). User-facing name is "Egocentric Hand Reconstruction"; "Ego4Robo" is the internal codename / CI branch prefix.

5. ARM64 / aarch64 support

Isaac Teleop now installs and runs on ARM64 hosts such as NVIDIA DGX Spark (#506, #516, #509, #513, #358, #589, #681): the ROS 2 reference container auto-builds the nlopt wheel (none on PyPI for aarch64), required-environments platform constraints, Manus-plugin arch-conditional SDK fixes, and a Python 3.13 + arm64 CI leg.
Maturity: first-class for the containerized ROS 2 / CI path; bare-metal pip install of the [retargeters] extra still needs a documented one-time nlopt wheel build. (Jetson Nano remains planned.)

6. Streamlined CloudXR operations

  • CloudXRLauncher (#353): programmatic, in-process start / health_check / stop of the CloudXR runtime + WSS TLS proxy (context-manager API; runs the runtime as an isolated subprocess to avoid CUDA-context clashes with Isaac Sim). Replaces the now-deprecated run_cloudxr_via_docker.sh (#440) and powers ROS 2 live teleop (#609).
  • Out-of-band (OOB) teleop setup (#388, #390, #415, #438, #461, #655): --setup-oob auto-opens the web client on a USB-tethered headset over ADB and clicks CONNECT for you via Chrome DevTools Protocol; optional --usb-local relays signalling/media over the USB cable with a local coturn TURN server; a side-channel metrics/control hub; configurable ports.
  • CloudXR runtime and web SDK bumped to 6.2.0 (#564, #528).

Maturity: mixed (Launcher GA; OOB automation newer). Note: OOB = out-of-band, not out-of-box — the control hub runs on a side channel beside the video stream. --usb-local still requires Wi-Fi associated (Chromium's WebRTC excludes loopback ICE candidates), though no teleop traffic crosses Wi-Fi.

7. Televiz (preview)

A new opt-in (BUILD_VIZ default OFF) Vulkan + OpenXR + CUDA compositor exposed to Python as isaacteleop.viz (#423, #427, #451, #466, #471, #502, #517, #535, #536, #542, #641). VizSession composites GPU camera frames into XR / window / offscreen targets via QuadLayer (mono or true stereo), with zero-copy CUDA↔Vulkan interop (DeviceImage). Ships a camera_viz example driving v4l2 / OAK-D / ZED / synthetic sources (plus a RealSense 435i config, #683), locally or over RTP H.264, and can share one OpenXR session with TeleopSession.
Maturity: preview — QuadLayer only today; ProjectionLayer / 3D rendered content explicitly not yet shipped. Product name: "Televiz" (capital T, lowercase v).

8. MCAP record & replay

Record any teleop session — head, hands, controllers, full-body (Pico), pedals, message channels — to an MCAP file (the open Foxglove standard, with FlatBuffer payloads), then replay it headless through the exact same retargeting pipeline (SessionMode.REPLAY), frame-aligned and deterministic, with no headset attached (#290, #330, #389, #396, #401–#404, #425, #508, #527, #554). Ideal for CI/regression, debugging, and dataset generation; includes viser browser replay examples for hand/controller/full-body.
Maturity: mixed — recording is GA-quality; replay has preview edges (no end-of-file signal; recordings made before #527 must be re-recorded for correct frame alignment).

9. Manus data-glove plugin (partner-authored)

Integrates MANUS Core SDK gloves for full hand-joint tracking (#431, #441, #510, #526, #513, #520, #525). The wrist anchor auto-selects between HMD optical hand tracking (XR_MNDX_xdev_space) and a Quest 3 controller fallback; per-finger poses are injected into the OpenXR hand layer so any retargeter consumes them transparently. Includes installer/udev scripts (SHA-256-verified SDK), a Vulkan "MANUS Data Visualizer", and a ROS controller-mode path feeding the Sharpa retargeters.
Maturity: mixed (Linux/x86_64 primary, ARM fixed; proprietary SDK not redistributed). Authored by Manus — see Community & partner contributions.

10. Web client overhaul

The browser teleop client (now simply the "web client", renamed from "WebXR client" in #671) becomes the default entry point (#338, #398, #420, #473/#476, #537, #556/#560, #561, #591, #629, #671, #685, #687):

  • Registry-driven application picker including Isaac Lab, Isaac Sim, ROS, and GEAR.
  • Every CloudXR setting is URL-settable via a param registry; settings persist per application.
  • Headless mode + window.xrDevice exposure for programmatic / AI-agent control.
  • Per-version hosting (/client/<slug>/, /client/stable/) and self-hosting via --host-client.
  • Robustness: old-browser warning, self-signed-cert acceptance link, stale-WebGL-error draining, exit-on-streaming-error, XR-panel drag fix, Server-IP clear button.
    Maturity: mixed.

Other notable improvements

  • OpenXR controller haptic feedback (rumble) (#566) — a vendor-neutral device-output path (HapticSink / IHapticDevice → xrApplyHapticFeedback) so sim contact force or a trigger pull can vibrate the operator's controller. GA path with example + tests; the broader force-feedback framework (EndEffectorForce) is foundation ahead of consumers.
  • Haptikos hands plugin (#486, #642) (partner — Magos) — an OpenXR plugin for the Haptikos hand exoskeleton (hand/finger pose injection only; Linux / Meta Quest). See Community section.
  • Pipelined retargeting (#467) — opt-in RetargetingExecutionMode.PIPELINED runs the retarget graph on a background worker so step() returns the latest finished result instead of blocking the control loop, with latency telemetry and deadline pacing.
  • Teleop state machine (#340, #391) — DefaultTeleopStateManager (stopped/paused/running + reset), fail-safe to STOPPED on input loss, with reset wired into stateful retargeters.
  • Camera pipeline (#323, #266, #349, #341, #203) — correct full-range BT.601 OAK-D color via a custom CUDA kernel, working stereo, graceful auto-reconnect, an OAK-D W center-RGB stream, and a standalone depthai 3.7.1 recording CLI with live preview.
  • Monotonic tracker API (#352, #315, #320, #351) — trackers now implement update(monotonic_time_ns) instead of XrTime, removing OpenXR from deviceio_base; schema-defined hand tracker and factory-based tracker creation make adding new devices easier.
  • Pico full-body tracking — new body-tracking docs (#397), graceful bypass when the runtime lacks body-tracking support (#287), and full-body demo videos (#573).
  • New XR controller "menu" button (menu_click) added to the controller schema and live tracker.
  • Python 3.13 support (#358, #405) and CMake structure standardization (#600 + conformance refactors).
  • Ecosystem / docs: Social links — Discord + WeChat (#667), a device roadmap (Haply, 3D Space Mouse), MCAP record/replay docs (#424), body-tracking docs, and a headset-firmware quick-start note (#643).

Community & partner contributions

Contributions from outside NVIDIA, verified by commit-author email and GitHub org membership:

Manus (manus-meta.com) — Manus data-glove integration

  • Tim van Wankum — authored the foundational Manus plugin: HMD hand tracking, installer, docs, and Vulkan visualizer (#431). (The PR was merged under an NVIDIA account, but the implementation is Manus's.)
  • Aniket Mazumder — Manus SDK root-path handling in CMake (#441).

Magos / Haptikos (themagos.com) — Haptikos hand-exoskeleton plugin

  • giorgosbabanis — Haptikos hands plugin (#486).
  • PKontrazis — Haptikos documentation (#642).

Community

  • Minh Vu (@fallintoplace) — documented Python environment activation in the quick start (#661).

Detailed PR List

New Contributors

Full Changelog: v1.1.8...v1.3.131

Contributors

life1ess, fallintoplace, and 20 other contributors
Assets 3
Loading