strands-robots v0.5.0

Release date: 2026-08-03 · Previous release: v0.4.1 (2026-07-01)

804 commits · ~146 merged pull requests across five weeks. This is the largest
release since the project began: a new NVIDIA Isaac Sim backend, agent-facing
analytic motion primitives, a remote-inference client/server split, terrain
locomotion curricula, and a deep hardening pass across every numeric input the
agent and mesh surfaces accept.

strands-robots is the robot-control library for Strands Agents:
policy inference, teleoperation, calibration, recording, and simulation (MuJoCo,
Newton, and now Isaac) behind one agent-callable tool surface.

---

Highlights

• NVIDIA Isaac Sim backend - full IsaacSimulation with procedural scene loading,
  LeRobotDataset recording parity, a delta-EEF differential-IK controller for GR00T
  actuation, EEF-state grounding for LIBERO, and pip-isaacsim 6.0.x support.
• Analytic motion primitives - move_to / set_gripper / rotate_wrist, backed
  by one shared mink IK bridge, give an agent the staging/transport/release vocabulary
  to position a robot around a learned policy's competence region.
• stop_when= semantic early-return - a rollout terminates as soon as the world
  reaches a predicate state, with stopped_reason telemetry, turning a monolithic
  rollout into a retryable primitive.
• Remote policy inference - a client/server split so an edge robot can drive a
  policy running on a remote GPU.
• LeRobot 0.6 policy support - the [lerobot] / [molmoact2] extras now track
  lerobot>=0.6.0,<0.7.0, so MolmoAct2 resolves straight from PyPI (no from-source
  build) and the policy resolver covers every family lerobot 0.6.x ships.
• Terrain locomotion curricula - create_world(terrain=..., difficulty=...) for
  stairs / slope / rough / pyramid heightfields, a full locomotion reward/predicate DSL,
  and shipped go2 / g1 / t1 benchmarks.
• rosbridge transport - use_rosbridge + RosbridgeRobot drive ROS1 robots and
  any host with no ROS install (verified live against the NASA Curiosity Mars rover
  Gazebo sim).
• A repository-wide input-hardening pass - every numeric knob on the simulation,
  hardware, mesh, teleop, and recording surfaces now refuses a value it cannot honor
  instead of silently clamping, coercing, or reporting a false success.

The changelog assembly moved to per-PR news fragments (changelog.d/) this
cycle, so overlapping PRs no longer conflict on the shared [Unreleased] anchor.

---

New backend: NVIDIA Isaac Sim

Isaac joins MuJoCo and Newton as a first-class simulation backend behind the same
tool surface.

• IsaacSimulation - procedural scene building, robot/object/camera lifecycle,
  and physics stepping.
• LeRobotDataset recording parity (IsaacRecordingMixin) - Isaac rollouts record
  the same dataset format as MuJoCo and Newton.
• GR00T action actuation - an Isaac-side delta-EEF differential-IK controller
  converts GR00T's task-space {x, y, z, roll, pitch, yaw, gripper} deltas into joint
  targets (no robosuite dependency).
• EEF state grounding - IsaacSimulation.get_body_state plus LiberoAdapter
  EEF-pose/gripper state sources so language-conditioned policies get the
  state.x/y/z/... they require; this also unblocks the predicate DSL on Isaac.
• LIBERO init-state arm qpos - robosuite joint names are mapped onto the USD
  articulation so an episode starts from the LIBERO ready pose rather than the USD
  default.
• pip-isaacsim 6.0.x support - synchronous physics-view invalidation, camera
  warm-up under a stopped timeline, matrix-parser tolerance, and documented install
  collateral (the coverage>=7.6 + OMNI_KIT_ACCEPT_EULA caveats).
• URDF joint names - the importer's USD-mangled joint names are demangled back to
  the URDF's own names, so the same URDF reports one joint vocabulary on Isaac and MuJoCo.

---

Agent capabilities

Analytic motion primitives (move_to / set_gripper / rotate_wrist)

The Harness VLA staging vocabulary, backed by one shared mink DLS IK bridge
(strands_robots/simulation/ik.py, deduplicated from the cosmos3 and vera copies):

• move_to(robot_name, position, orientation=None, ...) - position-only when
  orientation=None (the right contract for 5-DOF arms), DLS IK to the world-frame
  target then a servo-drive loop; returns the IK residual on an unreachable target
  rather than hanging or raising.
• set_gripper(state="open"|"close") - ctrlrange-endpoint set-point with registry-driven
  open/close convention.
• rotate_wrist(target_yaw) - wrist-yaw set-point while every other actuator holds.

All three refuse to run while a policy drives the same robot, take the engine lock per
control tick so renders and stop_policy interleave, and are not collision-aware
(a future curobo backend can drop in). Registry gripper metadata (gripper block per
robot) resolves which actuators the primitives command, so an arm actuator whose name
collides with a gripper hint is no longer silently excluded from IK.

The figure under pixel-to-world grounding below is the end-to-end demo: an agent
names a pixel, get_world_point turns it into a world coordinate, and move_to
puts the jaw there.

get_world_point(camera, pixels) - pixel-to-world grounding

Instead of reading privileged object poses, an agent picks pixels on a target's visible
surface and this call unprojects each through the pixel-aligned metric depth buffer
(p_world = T_world_cam · depth · K⁻¹ · [u, v, 1]), with built-in multi-pixel median
and per-pixel validity. Works identically on hardware with an RGB-D camera, so grounding
built on it transfers.

Figure 1 - point at a pixel, the arm goes there (#1649 + #1654).
Left: the frame the agent saw, with the single pixel it named - no hand-written joint
targets. Middle: get_world_point unprojects that pixel and the result is compared
against the cube's privileged pose; the pixel resolves to the point on the visible
surface, which is what "looking at a frame" actually means. Right: move_to
drives the jaw to that coordinate and reports its IK residual. Rendered headless
(MUJOCO_GL=egl) against the tagged source.

stop_when= semantic early-return

run_policy(stop_when={"predicate": "grasped", ...}) terminates the rollout as soon as
the world reaches the predicate state (checked every control step, on both the
synchronous and async-RTC paths), and reports stopped_reason ∈
predicate | budget | cancelled | error plus steps_used. This turns a rollout into a
retryable primitive an agent can stage → invoke → inspect → re-invoke, and composes with
recording (the recorded episode's frame count equals steps_used).

---

Remote policy inference

A client/server split (strands_robots/inference/) so an edge robot can drive a policy
that runs on a remote GPU. lerobot_async gained a rename_map for remote
observation-key remapping and now forwards it to the PolicyServer, and a numeric
handshake (set_control_frequency / set_rtc_observed_delay) is validated at the call
rather than mid-rollout.

---

LeRobot 0.6 policies

The [lerobot] and [molmoact2] extras now require lerobot>=0.6.0,<0.7.0 (was
>=0.5.0,<0.6.0), and the 0.5.1-era torch/torchcodec overrides are dropped - lerobot
0.6 ships mature, platform-correct dependency pins of its own.

• MolmoAct2 installs from PyPI. MolmoAct2Policy ships in lerobot 0.6 (lerobot
  PR #3604), so pip install 'strands-robots[molmoact2]' resolves it directly - the old
  "install lerobot from source" step, the transformers==5.3.0 pin, and the
  lerobot[smolvla]==0.5.1 guidance are all gone.
• The policy resolver covers the full lerobot 0.6.x family - pi0, pi0.5,
  pi0_fast, SmolVLA, MolmoAct2, ACT, Diffusion, and the newer eo1 /
  vla_jepa - with a regression guard so a family a future lerobot release adds is
  picked up automatically. Both lerobot_async's server-supported set and the training
  expert_only set are sourced live from lerobot rather than a hard-coded list that
  drifts.
• expert_only fine-tuning (freeze the VLM, train only the action expert) is now
  gated per policy: a run is allowed iff that policy's lerobot config actually declares a
  train_expert_only field, instead of against a fixed {pi0, pi05, pi0_fast, smolvla}
  tuple (PI0FASTConfig, for one, exposes no such field).
• RTC seam documented honestly - run_policy(async_rtc=...) no longer claims
  SmolVLA/MolmoAct2 blend the chunk seam internally; the public lerobot/smolvla_base
  ships rtc_config=None and MolmoAct2 has none, so the library's own real-time-chunking
  path is what joins the seam.

---

Locomotion & terrain curricula

• Terrain heightfields - create_world(terrain="stairs"|"slope"|"rough"|"pyramid", difficulty=...) as a curriculum knob a locomotion trainer ramps across resets, plus
  get_ground_height(x, y) to query the local surface, floating-base seating on the
  terrain surface, and terrain-relative reward measurement.
• Reward/predicate DSL - base_velocity, base_height, base_orientation,
  base_lin_vel_z / base_ang_vel_xy, base_velocity_tracking reward terms; and
  base_beyond_x/y, base_yaw_beyond, base_below_z, base_tipped predicates.
• Shipped benchmarks - go2_walk_forward, go2_strafe_left, go2_turn_left,
  g1_walk_forward, t1_walk_forward via register_builtin_benchmarks().
• Declarative benchmark spec accepts a natural-language instruction field.

Figure 2 - locomotion ground: four terrain kinds and a difficulty curriculum.
unitree_go2 settled on each heightfield create_world(terrain=...) generates. The
quadruped's standing clearance is identical on every panel while the ground beneath
it changes - that is floating-base seating placing the robot on the local surface
rather than the nominal plane, which is what makes difficulty a curriculum knob a
trainer can ramp across resets instead of a reset that drops the robot into geometry.

---

rosbridge transport - ROS1 and ROS-less hosts

use_rosbridge (tool) + RosbridgeRobot (mesh) speak the rosbridge WebSocket JSON
protocol via pure-pip roslibpy (new [rosbridge] extra), reaching every ROS1 robot
and any machine without a ROS install - the two gaps use_ros (rclpy) and use_rtps
(DDS) leave open. RosbridgeRobot.from_curiosity() preconfigures the NASA Curiosity Mars
rover Gazebo sim; the full NL→agent→WebSocket→ROS1 pipeline was verified live driving the
rover across Mars terrain.

---

Recording, datasets & rendering

• DatasetRecorder.create(overwrite=...) for honest re-recording into an existing
  repo_id; sync_dataset_to_bucket(root, bucket) for lifecycle-independent bucket
  sync; eval_policy(video=...) / evaluate_benchmark(video=...) record a
  per-episode rollout MP4.
• verify-dataset flags truncated/partial-encode videos and now localises partial
  parquet corruption - one broken shard is named, and every other check still runs,
  instead of collapsing the whole report.
• Rendering pipeline (strands_robots/rendering/) - gaussian-splat and panorama
  backgrounds, a hybrid depth compositor, and an MJPEG live-stream + clip encoder, each
  refusing render options it cannot honor.
• Recording-rate integrity - a dataset can no longer be recorded, resumed, or driven
  at an fps that disagrees with the rollout's control_frequency (the recorder writes
  one un-decimated frame per control step, so a mismatch mislabels the episode timebase).
  A lost dataset frame now fails the rollout rather than silently producing a re-timestamped
  episode.

---

Reliability & safety hardening

This release's largest single theme: the agent and mesh surfaces now refuse a value
they cannot honor rather than clamping it, coercing it, or reporting a false success.
Many of these are subtle - a rollout that reported success while doing nothing, a
recording that reported success with no file on disk, an actuator command silently
rewritten to full travel. Each fix ships with a regression test that fails on the prior
code and a measured before/after artifact.

Safety-critical

• Emergency stop / lockout - four independent defects each defeated e-stop or lockout
  on some path (mesh resume-proof on the fallback publish path, cloud fan-out dropping the
  second e-stop as a replay, Reachy Mini e-stop no-op under an RPC allowlist, and
  broadcast() undercounting acks) are all fixed.

• E-stop honesty - an emergency stop now reports the peers that did not stop
  (peers_not_stopped) instead of a clean acknowledgement from a robot that never stopped.

  

• pose_tool emergency_stop actually de-energizes the arm (writes Torque_Enable=0
  to every motor) instead of returning a fabricated success while nothing touches the bus.

• Teleop slew bound - a teleop stream inside every other cap can no longer reverse a
  joint full-scale on every frame (~14× the servo's rated speed); a per-joint slew bound
  refuses commands a joint cannot physically travel that fast, and the bound is frame-shape
  independent.

  

• Teleop source scoping - a wildcard source_peer_id (* / **) is refused, so a
  receiver can no longer follow every peer on the mesh.

  

Hardware task lifecycle

• A stop pressed during bring-up now actually stops the task instead of letting the
  arm move once connection finishes.

  

• One rollout per bus - a second rollout is refused while another owns the motors bus,
  instead of two control loops driving one arm.

  

• cleanup() / stop() are now genuinely terminal: they close the motors bus and every
  camera, refuse later rollouts, and run teardown off the event loop; a failed connect
  closes every half-open device; a per-task Policy.reset() clears stale per-episode state.

• Hardware control_frequency, task duration, action_horizon, max_relative_target,
  and per-camera options are validated at the call, before the serial port opens.

Simulation input domains

Every numeric knob on the sim surface now honors or refuses - no silent clamp:

• add_object - mass (an accepted inf mass NaN'd the whole world), size, color,
  and the object's inertia is now integrated from its shape rather than a hardcoded
  constant.

  

• send_action - a boolean actuator command is refused rather than silently written as
  full-travel 1.0/0.0; an actuator-name key and its joint-name alias now write the
  same value; a non-finite action value is refused instead of teleporting the whole scene.

  

• run_policy(n_steps=N) executes exactly N steps at every control frequency (a float
  round-trip previously truncated 1 step at 49 Hz to 0), and duration / control_substeps
  / policy_config / stop_when are all validated up front.

• create_world (timestep/gravity), set_geom_properties, set_body_properties,
  randomize / set_obs_noise, raycast, patch_scene_mjcf, get_camera_params,
  get_world_point, and the whole vector-parameter family now read caller vectors by
  membership (accepting NumPy arrays) rather than truthiness.

• Physics setters (mass / size / gravity / timestep) survive the next scene recompile
  instead of being silently reverted; a resized geom re-derives its owning body's inertia;
  a physics checkpoint is refused once its model is swapped; a latched external wrench
  belongs to the body it was applied to.

  

• A robot is simulated under the solver settings its own model declares. MuJoCo's
  <option> is model-global and does not survive spec.attach(), so every solver
  setting a robot MJCF declared for itself was discarded when the robot was composed
  into a generated scene - 42 of the shipped registry robots declare one.

  

  Figure 3 - the declared integrator, honored (#1687). Under the integrator the
  scene silently fell back to, a stiff position servo never settles; under the one
  panda.xml declares, the same hold settles to zero residual joint velocity with a
  smaller tracking error. Same scene, same command - the only difference is whether the
  model's own declaration survived composition.

  

  The same fix in motion: with the declaration discarded the scripted pick pushes the
  cube on approach and squeezes through it on the lift; with it honored, the grasp holds.

• Adding a robot no longer rewinds the scene it joins. add_robot reset the whole
  world to give the newcomer a clean start, so one ordinary incremental edit discarded
  every pose, contact and servo set-point already established.

  

  Figure 4 - an incremental edit is no longer destructive (#1763). A parked arm and
  a crate settled on a pedestal, then a second robot is added. Before: every joint
  returns to zero and the crate teleports back to its spawn height - reported as a
  successful add. After: pose and contact state are preserved.

• move_to on the advertised [all]/[sim-mujoco] install now runs - the extra declares
  the QP solver its IK needs.

  

Policy & training correctness

• The local LeRobot policy state and action sides agree on what a hardware observation is
  (a mixed-dtype observation no longer silently mis-scaled a command).
• In-process LeRobot training - resume, LoRA use_peft, and base_model warm-start
  correctness; val_episodes now produces an actual validation loss.
• A short action vector no longer fabricates zero commands for the unmatched actuators
  (which would drive them to zero at servo speed); a chunk count the consumer cannot execute
  is refused; SO-101 real-hardware camera fourcc + sim-embodiment policies now work.
• PersistentPolicy async get_actions no longer deadlocks under two-thread contention.

Registry & robot factory

• A register_robot alias collision now fails closed rather than bricking every subsequent
  lookup; alias-less names resolve; vera://host:port URLs parse; a leader arm is refused
  by Robot() (it is a teleoperator, not a follower) instead of being driven as a position
  servo; every documented install hint names an extra that exists.

---

Migration notes

• Python 3.12+ remains required (lerobot>=0.6.0; the 0.5.1-era torch/torchcodec
  overrides are dropped).
• [sim-mujoco] and [all] now declare the mink + qpsolvers[daqp] IK stack, so
  move_to runs on the advertised install. Isaac Sim is still installed separately (pip
  isaacsim 6.0.x or the Omniverse launcher) - see the caveats in docs/simulation/isaac.md.
• Recording fps must match control_frequency. A rollout will now refuse to record,
  resume, or run against a dataset whose fps disagrees with the rollout's control rate.
  Align the two (record at the rollout rate, or drive at the dataset rate); the error
  message names both remedies.
• Numeric-input strictness. Calls that previously succeeded with an out-of-domain
  value (fps=0, mass=inf, duration=-1, a boolean gripper command, a wildcard teleop
  source, …) now return a structured error. This is intentional - the prior behaviour was a
  silent false success - but a caller relying on the old leniency will see the refusal.
• register_robot alias collisions now raise. A registration whose alias collides with
  an existing canonical name or another robot's alias is refused at write time rather than
  saved and then failing every later read.
• New extras: [rosbridge] (pure-pip roslibpy), [ollama] (local-LLM agent path).

---

Contributors

Thanks to everyone who contributed to this release:

• @cagataycali - maintainer; the input-hardening pass,
  motion primitives IK, recording integrity, and most of the release.
• @yinsong1986 - NVIDIA Isaac Sim backend, GR00T delta-EEF
  actuation, LIBERO EEF grounding, get_world_point, stop_when=, analytic motion
  primitives, and per-robot gripper metadata.
• @Vivek0712 - rosbridge transport (use_rosbridge +
  RosbridgeRobot, NASA Curiosity verified live) and ROS 2 action support in use_ros.
• @mertkrgl - hardware run_policy(policy_object=...)
  sim parity, the [ollama] extra, and teleop status/example fixes.
• @sundargthb - streaming-data-loop notebook and
  storage-bucket example fixes.
• @max-rattray-aws - HIL gate for
  lerobot_train pretrained_path.
• @CurrentlyAWey - HIL gate for the lerobot_train
extra_flags passthrough.

---

Full changelog

168 behavioural changelog sections across this release (43 additions, 114 fixes, 8 docs,
2 changed, 1 quality). See CHANGELOG.md (assembled from the changelog.d/ fragments) and
the full commit comparison.