Add SO-101 support in Isaac Lab#5989
Conversation
jiwenc-nv
requested review from
ClemensSchwarke,
Mayankm96,
Toni-SM,
hhansen-bdai,
jtigue-bdai,
kellyguo11,
masoudmoghani,
matthewtrepte,
ooctipus,
ossamaAhmed,
pascal-roth,
peterd-NV and
xyao-nv
as code owners
github-actions
Bot
added
documentation
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 Code Review — PR #5989: Add SO-101 Teleop
Latest review: Commit df713e6 (rebased onto latest develop)
Summary
This PR adds support for the TheRobotStudio SO-101 5-DOF follower arm to Isaac Lab, including:
- Asset configuration (
SO101_CFG,SO101_HIGH_PD_CFG) - Cube-stacking task environments (joint-position and IK-absolute)
- Generalization of gripper MDP logic to support single-jaw grippers
- XR teleoperation support with IsaacTeleop
✅ All Previous Issues Addressed
- Yaw tuple typo fixed —
"yaw": (-1.0, 1, 0)→"yaw": (-1.0, 1.0)✓ - Test coverage added —
test_so101_teleop_pipeline.pytests the XR teleop retargeting pipeline
🟢 No New Issues
The PR was rebased onto the latest develop branch. The SO-101 implementation is unchanged from the previous review.
🟡 Non-Blocking Suggestions (for in-sim tuning)
-
Table/Cube geometry: Base
StackEnvCfgtable is atpos=[0.5, 0, 0]but SO-101 cubes atx ∈ [0.15, 0.30]. Verify in-sim. -
Gripper limit margin:
_SO101_GRIPPER_OPEN = 1.745is at USD limit (~100°). Consider small margin.
🟢 Positive Aspects
- Clean MDP generalization for single-jaw grippers
- 5-DOF limitation well-documented
- Position-only IK correctly handles under-actuated arm
- Proper test coverage for teleop pipeline
- Changelog fragments included
Verdict: Ready for human review — Implementation is solid and well-documented.
Update (commit 44e01de): New commits add the live EE home-latching feature (ee_frame_prim_path in IsaacTeleopCfg) and integrate it into the clutch retargeter so engaging teleop no longer snaps the arm. The _read_prim_world_matrix refactor in isaac_teleop_device.py is clean. No new issues introduced in the SO-101 changes. Non-SO-101 develop-branch changes (reach task renaming, hydra typed-selector validation, visualizer contacts/planes, version bumps) are out of scope for this review.
Update (commit 7edb2d1): Branch was rebased/squashed — all file contents are byte-identical to the previously reviewed 44e01de7. No new code changes, no new issues. Previous review findings still apply.
Update (commit 2c90531): New commits add changelog fragments (.rst files in changelog.d/ directories for isaaclab_assets, isaaclab_tasks, and isaaclab_teleop). These are documentation-only — no code changes. Previous inline comment (gripper tolerance) remains unaddressed but is non-blocking. No new issues.
jiwenc-nv
force-pushed
the
jiwenc/so-101
branch
2 times, most recently
from
a459b30 to
7d3d44c
Compare
jiwenc-nv
force-pushed
the
jiwenc/so-101
branch
3 times, most recently
from
44e01de to
7edb2d1
Compare
jiwenc-nv
changed the title
Greptile SummaryThis PR adds SO-101 (TheRobotStudio 5-DOF arm) support to Isaac Lab, introducing a robot asset config, two stacking task environments (joint-position and IK-Abs), three custom XR teleop retargeters (clutch EE-pose, analog gripper, wrist-roll), and corresponding teleop infrastructure extensions. The MDP observation and termination helpers are generalised from parallel-gripper-only to support single-jaw grippers.
Confidence Score: 3/5Safe to merge after fixing the import guard bug; all other concerns are tuning and tolerance questions that do not affect correctness for installations with isaacteleop present. The retargeter modules unconditionally import from isaacteleop at their own module top level, so importing stack_ik_abs_env_cfg without isaacteleop installed raises an unhandled ImportError despite the try/except block that is intended to make the module degrade gracefully. The _TELEOP_AVAILABLE = False branch is unreachable in that scenario. Additionally, the open-position success condition uses a sub-milliradian tolerance that may be unachievable with the SO-101's soft gripper actuator in the joint-position task, potentially making the stacking success signal never fire. stack_ik_abs_env_cfg.py needs the retargeter imports moved inside the try/except block. observations.py and terminations.py warrant a revisit of the atol/rtol=1e-4 gripper-open tolerance once the robot is validated in simulation. Important Files Changed
Sequence DiagramsequenceDiagram
participant XR as XR Controller
participant Dev as IsaacTeleopDevice
participant SL as TeleopSessionLifecycle
participant CR as SO101ClutchRetargeter
participant GR as SO101GripperRetargeter
participant RR as SO101RollRetargeter
participant TR as TensorReorderer
participant Env as IsaacLab Env
Dev->>Dev: _get_ee_frame_world_matrix()
Dev->>SL: step(anchor_fn, target_T_world, ee_world_matrix)
SL->>SL: _build_external_inputs() feeds world_T_anchor + robot_ee_pos
XR-->>CR: grip_pos + grip_ori (transformed)
SL-->>CR: robot_ee_pos (world_T_ee)
CR->>CR: "clutch latch on engage, home = live EE pos (base frame)"
CR->>TR: ee_pose 7D pos_xyz + quat_xyzw
XR-->>GR: trigger_value
GR->>TR: gripper_command closedness in 0-1
XR-->>RR: grip_orientation
RR->>TR: roll_command swing-twist angle rad
TR->>TR: output_order pos_x pos_y pos_z roll_value gripper_value
TR->>Env: action 5D
Env->>Env: DiffIK(pos_xyz) + JointPos(roll) + JointPos(gripper)
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| torch.isclose( | ||
| robot.data.joint_pos.torch[:, joint_id], | ||
| open_val, | ||
| atol=1e-4, | ||
| rtol=1e-4, | ||
| ), | ||
| stacked, | ||
| ) | ||
| else: | ||
| raise ValueError("No gripper_joint_names found in environment config") | ||
|
|
There was a problem hiding this comment.
object_stacked open-position tolerance may be too tight for the SO-101 soft actuator
atol=1e-4, rtol=1e-4 requires the gripper joint to be within ≈0.0003 rad (≈0.017°) of gripper_open_val = 1.745. For the joint-position config (SO101_CFG), the gripper actuator has stiffness 17.8, which is soft enough that gravity and contact residuals can leave the joint several tenths of a radian away from the commanded target. In practice this means object_stacked may never return True for the SO-101 joint-position task, since the gripper cannot reliably settle within the sub-milliradian band. The corresponding cubes_stacked termination in terminations.py has the same threshold. Consider widening the tolerance or using a dedicated gripper_threshold parameter for this check.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Add an SO-101 (5-DOF) cube-stacking environment driven by XR teleoperation through the IsaacTeleop retargeting pipeline. - Add the SO-101 robot asset and register it in isaaclab_assets. - Add Isaac-Stack-Cube-SO101 joint-position and IK-Abs task configs. The 5-DOF arm uses position-only IK over four joints, with the wrist roll driven directly and an analog gripper, flattened to a 5D action [pos_x, pos_y, pos_z, roll, gripper]. - Build the teleop pipeline from the SO-101 retargeters provided by Isaac Teleop (isaacteleop.retargeters): SO101ClutchRetargeter (relative-origin EE pose), SO101RollRetargeter (wrist-roll swing-twist), and SO101GripperRetargeter (analog gripper). The clutch retargeter (in Isaac Teleop) latches the controller origin on the first RUNNING frame, rotates the world-frame controller delta into the seated base frame (yawed +90 deg about +Z), and latches the robot's live end-effector pose as home on engage, so engaging teleop does not jerk the arm. This task supplies that live pose: isaaclab_teleop: add IsaacTeleopCfg.ee_frame_prim_path; the device reads that prim's world transform each frame and supplies it to the pipeline as the robot_ee_pos ValueInput leaf (passed through verbatim, not composed with the anchor/rebase transforms). Add sim-free tests for the Lab-side pipeline wiring (5D action width and element order) and pin the clutch's seat-derived constants in Isaac Teleop to this task's base seat pose. The retargeter math itself is unit tested in Isaac Teleop.
Backfill the isaaclab_assets, isaaclab_tasks, and isaaclab_teleop changelog.d entries for the SO-101 cube-stack work (one consolidated minor fragment per package), as required by the nightly changelog and version compilation.
1 participant
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Screenshots
Please attach before and after screenshots of the change if applicable.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there