Docs for OS 0.7: the on-robot brain, the skills API, and spatial memory - #24
Open
theo-michel wants to merge 11 commits into
Open
Docs for OS 0.7: the on-robot brain, the skills API, and spatial memory#24theo-michel wants to merge 11 commits into
theo-michel wants to merge 11 commits into
Conversation
Rewrite the skills pages for the new authoring surface from innate-os PR #542: 'from innate import ...' with bare type annotations for interfaces/state/cameras, docstring-as-guidelines, plain-string success returns, fail(), on_cancel()/check_cancelled(), feedback(), say(), storage, wait_for(). First example on each page spells out guidelines() explicitly; the shorthand defaults are introduced afterward. Tables now list the typed declarations (image: MainImage, odom: Odometry, ...) and the real Manipulation methods (move_to_joint_positions, open/close_gripper, torque_on/off). Composing-skills worked example synced with the current run_routine_demo.py; physical examples reference the new pick_any_object skill.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…ghting Brings the docs up to innate-os main (v0.7.0-rc1-dev). Speed modes (innate-os #576, controller-app #235). New "Drive speed modes" section on the control page, linked from the web app, controller app and configuration pages rather than repeated. Four modes, not the three the clients' fallback tables suggest -- the robot ships Slow/Med/Fast/Mad and Mad (2.0x) exceeds motion_control.max_speed, so that carries a warning. Also documents that recording holds the robot at Med and restores on exit, and that speed mode is robot state shared by every client. Skills and agents. The previous commit was already stale: #542 moved the API again. Cancellation is now the framework's job -- self.sleep() replaces check_cancelled() in loops and the on_cancel brake hooks are gone. Returns are a plain string, self.fail(), or SkillOutput; the (message, SkillResult) tuples are deprecated. guidelines() is kept explicit as the documented default with the class-docstring shortcut demoted to a tip, since the explicit form is easier to learn from. composing-skills is rewritten -- it documented `from innate.skills import` and walked through run_routine_demo.py, both deleted. Composition is now declaration-based (pick: PickSocks -> self.pick(...)), with self.skills.run() for dispatch by id. Agents move to brain_client.agents.types with typed SkillRef/InputRef class lists; brain_client.agent_types no longer exists. Same for inputs, where definition.mdx imported a brain_client.input_types module that was never there. The shipped-agent list named two chess agents that don't exist and mislabelled basic_agent as "No Prompt". Workspace docs follow the import-based package model, and extra_skill_dirs is replaced by symlinked skill packs. Syntax highlighting. Python's TextMate grammar leaves bare identifiers unscoped -- its #expression repository ends with an unscoped catch-all -- so type annotations and imported classes rendered as plain text and no theme could reach them. languages/python-typed.json vendors the grammar with a CamelCase type rule inserted before that catch-all, registered via styling.codeblocks.languages.custom. Strings, comments, True/False/None and ALL_CAPS constants are excluded. Mintlify ignores injectTo and won't let a custom grammar shadow the bundled "python", so the fences are ```innatepy. Screenshots are captured from a live robot at 2x and annotated: the teleop speed picker, and the web app's agent panel on a new "Using the web app" section of starting-an-agent. Verified: 156 imported symbols and every documented method/attribute resolve against the innate-os checkout, 49 full-module examples parse, and links, anchors and images all resolve.
New Manipulation surface (move_to, follow + Waypoint, move_joints, rest, gripper_open/close, pose, recover, clamp_reach) with the standing-grip- target contract, plus a deprecated-methods table for the 0.6.0 shims and a one-line migration map. Examples updated to the raising, blocking API.
The Innate Agent page described a cloud service at wss://agent-v1.innate.bot. 0.7 deleted that transport: the agent loop now runs on the robot against Gemini, reached through the Innate proxy, a direct GEMINI_API_KEY, or not at all. Rewrite it around the loop — look/think/act, one turn at a time, speech streaming as the model writes it, and a turn that hasn't spoken yet being abandoned and re-run when the user talks over it. Document the three tools every agent gets for free (wait, go_to_point_in_view, stop_current_skill) and what happens to the toolset while a skill runs. Agent pages move to `from innate import Agent, InputRef, SkillRef` — one namespace for skills and agents. The old brain_client.agents.types path still works, so this is a preference, not a break. Also fixes three one-liners that still called the brain cloud-hosted, on overview, ros2-core and advanced-development.
…lt type Tutorial examples leaned on the class docstring standing in for guidelines(). It's the shorthand shipped skills use, but it hides the fact that the text is a prompt going to a model — so every tutorial example now writes guidelines() out, and the docstring form moves into a collapsed accordion for once that's second nature. Eight examples changed; three had no guidelines at all. Skill results: SkillOutput is one type carrying message, .data and an evidence image, so a skill can show what it found rather than only describe it. Composing skills loses its pre-release warning — the feature shipped in 0.7 — and gains a note on the OS version it needs. Navigation gains stop(), and the odometry-closed drive() / rotate_by(), which arrive rather than commanding a speed and hoping. Adds what ships in workspace/innate_skills/, grouped by what you'd want to do: seven of those skills are new in 0.7 and nothing pointed at them. physical-skill-examples is retitled Full-Body Examples — "physical skill" now specifically means a trained policy, so the old title collided. Path unchanged. Policy-defined skills gains the physical_skills typed-ref call contract.
…non-blocking motion move_to's verification tolerances were documented as tol_xy / tol_z. The parameters are tolerance_xy / tolerance_z, so anyone copying that line got a TypeError. Fixed in the prose and the shared method table. Adds two capabilities that were missing entirely: - move_by() nudges from the arm's *measured* pose rather than its last commanded one, which is the shape a visual-servoing loop wants. - block=False with .moving and wait() drives the base while the arm travels. Worth stating plainly that such a motion is unverified until joined — wait() is what surfaces the failure. Replaces the ArmFailed/ArmUnhealthy prose with a table (command was impossible vs. hardware needs attention), notes the GRIPPER_MAX_STRENGTH cap, and points at innate.exceptions. Robot state gains the odom-vs-pose frame table — odom drifts but never jumps, pose corrects and therefore can, and picking the wrong one is the classic bug — plus the real per-feed grace windows (3 s cameras, 6 s battery, 2 s the rest) in place of "a short grace period". Verified against innate-os 1c994242: robot/manipulation.py, state/odometry.py, state/pose.py, and _feed_specs in skills/types.py. Co-authored-by: Karmanyaah Malhotra <karmanyaahm@users.noreply.github.com>
…obot memory: SpatialMemory was entirely undocumented. Covers the begin() + self.wait_for() shape and why it's built that way — a search can run for minutes, and waiting through wait_for is what keeps Stop responsive — the RecallVerdict fields, and the distinction that matters most: found=False is a successful search whose answer is "nowhere", which is not the same as error. Screenshots captured from mars-the-44th: the Nav page showing remembered views painted across the office map, and the Brain page mid-turn with real tool calls and think latencies.
…al robot naming The quickstart's first build step pointed at the Gmail/IMAP skill, which needs an app password before it does anything. It now opens with a victory_spin you can paste, save and watch, then hands that skill to the agent — so the first five minutes end in the robot moving. SSH examples showed a bare `goodbot` password and a placeholder hostname. Robots ship as MARS the <N>th, so the real pattern is mars-the-<N>th.local with goodbot<N>. Both carried over from #25. Co-authored-by: Karmanyaah Malhotra <karmanyaahm@users.noreply.github.com>
Version selector and the shared version badges move to 0.7.0, and the spatial-memory page joins the Code-Defined Skills nav. Drops the Cloud Agent badge: the component it versioned is what the on-robot brain replaced, so a number for it now points readers at something that isn't in the loop.
theo-michel
changed the title
Seven pages carried less than a page of unique content each. "Evaluate and iterate" is the same loop as deploying a trained skill, so it moves onto that page, replacing three weak bullets that already said the same thing. "Rollout evaluation" documented an unreleased feature across 106 lines — whose UI, dataset layout, and every default may still change — and becomes a note beside it. The two Inputs children merge into their parent, a stub whose only job was linking to them. The FAQ had two questions, which go to Troubleshooting and the hardware overview. The controller app's connection page was 32 lines and joins the overview. The early-units fix was already out of the nav and applies only to robots shipped before January that were never set up. Also trims the agent examples, which repeated identical class scaffolding four times when only the skills and prompt differ, and removes a section of policy-defined skills that documented calling a policy from code twice. Redirects added for every removed page.
Adds captures of the Logging, Collect, and Training pages, the simulator's web app, and a fresh Teleop hero — the old one was still v0.5.1-dev, with seven nav icons where the app now has twelve, no speed modes, and no skills selector. With the images in place, the prose describing them goes. Logging loses its column list, filter bullets, and three-way pivot list, keeping only the two behaviors a screenshot can't show: what Clean parses, and that Live pauses autoscroll on scroll-up. The simulator's setup step and the agent panel walk- through lose their narration the same way. Also drops the last two mentions of rollout evaluation, which describe a Profiling page that no released build serves. Corrects "the web app doesn't run skills", which contradicted the manual triggering page: /brain/available_skills lists learned and replay skills alongside code ones, and Teleop runs them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The brain moved onto the robot in 0.7 and the skills API was rewritten around typed declarations. The docs still described the 0.6 world, so a reader following them today writes against an interface that no longer exists — and, on the agent page, against a cloud service that was deleted.
Every API claim here was checked against innate-os at
1c994242(0.7.0-rc3).The agent
software/agent.mdxclaimed the agent runs in the cloud atwss://agent-v1.innate.bot. That whole transport — auth handshake, registration, the ready-for-image flow — was deleted in 0.7. The page is rewritten around the loop that replaced it: look → think → act, one turn at a time, speech streaming out as the model writes it, and a turn that hasn't started speaking being abandoned and re-run when you talk over it.Also documents the three tools every agent gets without listing them (
wait,go_to_point_in_view,stop_current_skill), what the toolset collapses to while a skill runs, and the three ways to connect a model — hosted key, your ownGEMINI_API_KEY, or none, which is a supported mode rather than a broken one.Agent pages move to
from innate import Agent, InputRef, SkillRef. The old path still works; this is a preference, not a break.Skills
Guidelines are written out explicitly. Tutorial examples had drifted to letting the class docstring stand in. It's the shorthand shipped skills use, but it hides that the text is a prompt going to a model — so it's now a collapsed accordion for later rather than the thing you meet first. Eight examples changed; three had no guidelines at all.
One real bug fixed:
move_to's tolerances were documented astol_xy/tol_z. They aretolerance_xy/tolerance_z— the documented call raisedTypeError.Two missing capabilities:
move_by(), which nudges from the arm's measured pose (what a servoing loop needs), andblock=False+.moving+wait(), with the sharp edge stated plainly — a non-blocking motion is unverified until you join it.Spatial memory is new (
memory: SpatialMemory) and was entirely undocumented. The page covers thebegin()+wait_for()shape and why it's built that way, and the distinction that catches people:found=Falseis a successful search whose answer is "nowhere", which is not an error.Plus:
SkillOutputwith.dataand an evidence image, composing losing its pre-release warning,stop()/drive()/rotate_by()on navigation, the odom-vs-pose frame table, real per-feed grace windows, and a table of what already ships inworkspace/innate_skills/— seven of those skills are new in 0.7 and nothing pointed at them.physical-skill-examplesis retitled Full-Body Examples; in 0.7 "physical skill" specifically means a trained policy, so the old title collided. Path unchanged, no redirect needed.Screenshots
Captured from
mars-the-44th: the Nav page with remembered views painted across the office map, and the Brain page mid-turn with real tool calls and think latencies.Relationship to #25
@karmanyaahm rewrote several of the same pages in parallel. Where that PR was better I took it and credited it in the commits — the tolerance fix,
move_by, non-blocking motion, theArmFailed/ArmUnhealthysplit, the odom-vs-pose table, thephysical_skillscall contract, the quickstart'svictory_spin, and the real SSH naming. Those examples are re-written here with explicitguidelines()and the repo'sinnatepyfences.#25 still holds work this branch doesn't: its own passes on external services, composing, and the shared interface tables. Worth reconciling rather than merging both blind.
Before merging
0.7.0 is still at rc3. This bumps the version selector to 0.7.0, so merging publishes docs for an API that hasn't tagged. Worth holding for the release.
Two carried-over tables remain unaudited against source, both 0.6-era behavioral claims:
HeadTiltAnglesTableandNavigationUseCasesTable.