Skip to content

feat: add experimental LeRobot policy runtime - #3315

Open
TomCC7 wants to merge 7 commits into
feat/galaxea-a1z-hardwarefrom
feat/experimental-lerobot-policy
Open

feat: add experimental LeRobot policy runtime#3315
TomCC7 wants to merge 7 commits into
feat/galaxea-a1z-hardwarefrom
feat/experimental-lerobot-policy

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jul 31, 2026

Copy link
Copy Markdown
Member

Contribution path

Problem

DimOS needs an experimental runtime for executing trained LeRobot policies against live robot observations without coupling generic learning utilities to A1Z hardware or promoting an unstable policy API into a core package.

LeRobot 0.6 also requires Python 3.12+, Transformers 5, Hugging Face Hub 1.x, and NumPy <2.3. Those requirements conflict with the existing DimOS perception and development environments, which use Transformers 4.53 and NumPy 2.3.

Solution

  • Add LeRobotPolicyModule under dimos.experimental.robot_policy with named, lazily loaded checkpoints, typed streams, RPC status/control, and background-skill support.
  • Convert camera and joint-state observations into LeRobot inference inputs and publish ordered joint targets through the coordinator boundary.
  • Reject stale or incomplete observations, invalid actions, non-finite values, and mismatched joint dimensions.
  • Pin the optional runtime to LeRobot 0.6.0 on Python 3.12+ with its required Transformers 5 stack.
  • Use uv conflict declarations to keep the LeRobot environment separate from the existing Transformers 4 perception, base, Unitree, and development environments. uv now rejects accidental mixed installs.
  • Keep LeRobot optional for repository-wide mypy while checking its imported API boundary when the extra is installed.
  • Keep A1Z teaching, collection, replay, dataset preparation, camera, and other learning utilities out of this PR for a separate follow-up.

How to Test

Install and smoke-test the actual runtime environment:

uv sync --extra lerobot --no-default-groups --frozen
python -c "import importlib.metadata as m; import dimos.experimental.robot_policy.lerobot; print(m.version('lerobot'), m.version('transformers'))"

Expected versions: LeRobot 0.6.0 and Transformers 5.5.x.

Run the policy tests and CI-equivalent type check:

uv sync --only-group tests --frozen
pytest dimos/experimental/robot_policy/test_lerobot.py -q
uv sync --only-group lint --frozen
UV_NO_SYNC=1 uv run mypy

Validated locally: 16 policy tests passed; mypy passed across 935 source files; changed-file pre-commit hooks and uv lock --check passed.

AI assistance

OpenAI Codex with GPT-5 implemented and validated the change under direct user guidance. The user reviewed and approved the scope, package placement, dependency strategy, and stacked-PR structure.

Checklist

  • I have read and approved the CLA.

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.15584% with 36 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/experimental/robot_policy/lerobot.py 89.00% 20 Missing and 13 partials ⚠️
dimos/experimental/robot_policy/test_lerobot.py 99.05% 2 Missing and 1 partial ⚠️
@@                      Coverage Diff                      @@
##           feat/galaxea-a1z-hardware    #3315      +/-   ##
=============================================================
+ Coverage                      75.18%   75.23%   +0.05%     
=============================================================
  Files                           1145     1146       +1     
  Lines                         110038   110632     +594     
  Branches                        9900     9936      +36     
=============================================================
+ Hits                           82728    83237     +509     
- Misses                         24532    24544      +12     
- Partials                        2778     2851      +73     
Flag Coverage Δ
OS-ubuntu-24.04-arm 69.11% <94.15%> (+0.14%) ⬆️
OS-ubuntu-latest 71.18% <94.15%> (+0.12%) ⬆️
Py-3.10 71.18% <94.15%> (+0.12%) ⬆️
Py-3.11 71.18% <94.15%> (+0.12%) ⬆️
Py-3.12 71.18% <94.15%> (+0.12%) ⬆️
Py-3.13 71.17% <94.15%> (+0.12%) ⬆️
Py-3.14 71.19% <94.15%> (+0.13%) ⬆️
Py-3.14t 71.18% <94.15%> (+0.12%) ⬆️
SelfHosted-Large 29.09% <32.14%> (+0.01%) ⬆️
SelfHosted-Linux 35.86% <32.14%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/robot/all_blueprints.py 100.00% <ø> (ø)
dimos/experimental/robot_policy/test_lerobot.py 99.05% <99.05%> (ø)
dimos/experimental/robot_policy/lerobot.py 89.00% <89.00%> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from a8ab2d8 to 486d959 Compare July 31, 2026 23:35
@TomCC7
TomCC7 changed the base branch from main to feat/galaxea-a1z-hardware July 31, 2026 23:35
@TomCC7
TomCC7 marked this pull request as ready for review July 31, 2026 23:58
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 1, 2026
@TomCC7 TomCC7 mentioned this pull request Aug 1, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant