[PRD] DimOS Manipulation SDK Preview

[TomCC7]

DimOS Manipulation SDK Preview

Field	Value
Owner	TBD
Project	DimOS Manipulation SDK Preview
Labels	manipulation, sdk
Status	Draft
Milestone	TBD
Target date	TBD

Problem: DimOS has the pieces for manipulation, but developers must assemble planning, robot configuration, recording, conversion, and policy execution themselves.

Approach: Expose a small set of stable DimOS workflows and RPCs. Let each robot integration bind those workflows to real hardware through named profiles.

Outcome: A developer can build a classical manipulation app or collect demonstrations and run a learned policy without rebuilding the integration stack.

1. Problem

Manipulation works in DimOS today, but not as a coherent product. Developers must choose a blueprint, find the right RPCs and streams, configure conversion, and match each policy to its robot.

We cannot solve this with one central run(policy, robot) abstraction. Robots differ in joints, grippers, cameras, controllers, calibration, observations, and policy actions. Hiding those differences would make the easy cases vague and the hard cases unsafe.

The SDK needs a consistent journey while keeping each robot integration as the source of truth.

Why now?

The upcoming hackathon gives us an internal product deadline and a real validation setting. OpenYAM will be the reference robot, but the product is a universal manipulation SDK for robots integrated with DimOS. Because Quest hardware will not be available, OpenYAM also needs kinesthetic teaching for demonstration collection.

Current state

We have	What is missing
General manipulation planning RPCs	A documented, validated developer path
Blueprints and Dimos.connect()	A consistent workflow across robot integrations
Raw recording and DataPrep	One-command collection and automatic profile reuse during conversion
Policy and pick/place building blocks	End-to-end OpenYAM validation

Custom scripts can bridge these gaps for one setup, but they do not create a reusable DimOS product.

2. Solution

Build manipulation apps against stable DimOS workflows. Integrate each robot once through profiles, then use the same planning, collection, training, and execution journey.

flowchart LR
    D["Application developer"]

    subgraph DimOS["Stable DimOS surface"]
        S["CLI workflows + typed RPCs"]
    end

    subgraph Integration["Hardware integration"]
        P["Named profile"] --> B["Blueprint + job-specific Modules"] --> H["Robot + cameras + controllers"]
    end

    D --> S --> P

Loading

DimOS owns the public commands, RPCs, workflow lifecycle, and error behavior. A named profile owns the robot, camera, control mode, calibration, observation schema, action schema, and readiness checks. Blueprints continue to compose and connect the required Modules.

This split keeps the SDK hardware-independent without pretending that hardware is uniform. New robot packages add profiles; the central CLI does not grow robot-specific branches. Each command launches only the Modules needed for that job.

The preview includes:

• the existing general planning interface, documented and validated as the public planning API;
• a dedicated object-driven pick/place RPC Module;
• profile discovery and profile-driven workflows;
• raw collection, later conversion, LeRobot training, and policy execution commands;
• OpenYAM profiles for kinesthetic collection and ACT execution with a wrist RGB-D camera.

The SDK ships inside the existing dimos Python package and CLI. It does not add an umbrella wrapper around Modules or move connection management out of Blueprints.

3. Usage scenarios

flowchart TB
    subgraph Classical["Classical application"]
        C1["Python app"] --> C2["Dimos.connect()"] --> C3["Planning + pick/place RPCs"] --> C4["Physical task"]
    end

    subgraph Learning["Learning workflow"]
        L1["dimos collect profile"] --> L2["Raw recording"] --> L3["dimos dataprep build"] --> L4["dimos policy train"] --> L5["dimos policy run profile checkpoint"] --> L6["Physical task"]
    end

Loading

Classical manipulation: An application developer connects a Python script to a running blueprint. The script uses the existing planning RPCs for motion and the dedicated pick/place RPCs for object-driven tasks.

Learning: An operator runs dimos collect openyam-teach --task pick-cube, guides OpenYAM by hand, and saves or discards each attempt. DimOS stores raw data. The operator converts it afterward, trains ACT through LeRobot, and runs the checkpoint with an OpenYAM execution profile.

Teach is one way to control the robot during collection, alongside teleoperation and future modalities. It is not a separate data API. The recording carries its profile and schema so conversion and execution can check compatibility without asking the user to select the robot again.

4. Success metrics

The reference task uses one OpenYAM, a calibrated wrist RGB-D camera, a rigid cube, and a visible destination tray.

Outcome	Measure	Verified by
Fast first use	A new developer completes the planning quickstart within 15 minutes and saves a first episode within 10 minutes.	A non-implementing engineer
Reliable planning	The documented planning sequence succeeds in five consecutive OpenYAM runs and reports unreachable or cancelled work clearly.	OpenYAM owner
Trustworthy collection	A 20-attempt session contains exactly 15 saved and 5 discarded attempts; interruption preserves prior saved work.	Feature implementer and OpenYAM owner
Complete learning path	Raw data converts to LeRobot, ACT training starts through DimOS, and a compatible checkpoint runs for ten episodes without integration failures.	Milestone owner and OpenYAM owner
Physical task success	ACT succeeds at least 7/10 times; object-driven pick/place succeeds at least 8/10 times.	Milestone owner and OpenYAM owner

5. Milestone issues

Each row becomes one Linear issue.

Issue	Outcome	Done when
1. Profile-driven workflows	Robot packages can publish discoverable collection and execution profiles.	OpenYAM profiles resolve by name and incompatible setups fail before motion.
2. General planning interface	Developers can use the existing planning RPCs as the public planning API.	The quickstart and five-run planning check pass on OpenYAM.
3. One-command collection	dimos collect launches a complete raw recording session.	Save, discard, shutdown, and interruption preserve the correct episodes.
4. OpenYAM kinesthetic teaching	An operator can guide OpenYAM and control its gripper without Quest.	The operator records the first valid episode within 10 minutes.
5. Profile-aware conversion	A raw recording converts without selecting the robot again.	The saved episodes produce a valid LeRobot dataset.
6. LeRobot training command	DimOS provides a transparent entry point to LeRobot training.	An ACT smoke run receives the original arguments, output, and exit status.
7. Profile-driven policy execution	A compatible checkpoint runs through a named hardware profile.	Preflight blocks incompatible inputs and ten compatible episodes run without integration failures.
8. Object-driven pick/place	A dedicated RPC Module picks and places named objects into semantic or exact destinations.	The reference task succeeds at least 8/10 times and failures do not hang.
9. Preview release	Developers can follow one guide through both product journeys.	A non-implementer completes the quickstarts and the owner signs off on all metrics.

Appendices

A1. Dependencies

• PR #3315 provides candidate recording, DataPrep, OpenYAM, and policy foundations.
• PR #3318 is a reference for kinesthetic teaching and episode control; A1Z and replay are not dependencies.
• The reference setup needs OpenYAM, a working gripper, a calibrated wrist RGB-D camera, and an NVIDIA-capable Linux runtime host.
• LeRobot provides the reference dataset, ACT training, and checkpoint interfaces.

A2. Open questions

• Owner: TBD
• Milestone: TBD
• Target date: TBD

A3. Decision log

Decision	Product reason
Use hardware-owned profiles behind generic workflows.	A robot name alone cannot define its control and data contract.
Keep the existing planning RPC interface.	Another wrapper would duplicate the DimOS Module model.
Keep collection separate from manipulation planning.	Recording should work with teaching, teleoperation, and future control sources.
Record raw data and convert later.	Raw sessions stay recoverable and conversion stays repeatable.
Validate on OpenYAM without naming an OpenYAM SDK.	OpenYAM is the reference, not the product boundary.
Launch job-specific Blueprints.	One universal Blueprint would start unnecessary or conflicting systems.

A4. Non-goals

• replay or a Quest dependency;
• multi-manipulator applications;
• cross-robot dataset or checkpoint portability;
• new policy, VLA, or VLM research;
• a new umbrella manipulation or perception API;
• one all-capabilities Blueprint;
• automatic conversion during collection;
• industrial safety certification or non-Python application APIs.

A5. Public surface

dimos profile list
dimos collect <profile> --task <label>
dimos dataprep build <recording.db>
dimos policy train <LeRobot arguments>
dimos policy run <profile> <checkpoint>

Python applications attach through Dimos.connect(). They use the existing general manipulation planning RPCs and the dedicated PickAndPlaceModule RPCs. Agent skills may adapt those RPCs, but they do not define a separate implementation.