Graph-as-Policy

🧪 BETA CODE RELEASE | Beta | July 1, 2026

The graph is the policy. GaP targets Variational Automation — tasks a robot must perform persistently and reliably across many varying instances (objects vary in geometry and pose), not just solve once. It compiles a natural-language task into a typed, verified computation graph of modular skills, self-improves it in simulation, and runs the graph — not a black-box policy — on simulators and real robots.

_{GaP running on real robots — graphs generated from natural-language tasks, refined in sim, executed on hardware. Left→right: grocery packing (Franka, 10×), popcorn (long-horizon stove manipulation, 16×), tool & charger packing into tagged bins, and USB-C insertion (UR5 with force feedback).}

Important

🧪 GaP Beta Code release (1 July 2026). GaP is under active development and now in beta testing. Please send comments and suggestions to kych@berkeley.edu — we plan to release an updated version by 1 Aug 2026. Expect rough edges: APIs, the workflow schema, and skill interfaces may change without notice between releases.

import gap

conn   = gap.connector.sim("libero", task="libero_object/0")
result = gap.execute("examples/libero_quickstart/graph", conn)
graph  = gap.agent.generate_sync("pick the soup can and put it in the basket")
print(graph)                  # the policy, as a graph
gap.viz.serve("outputs")      # browse the recorded trial

Skills live in the sibling open-robot-skills repo (Anthropic Agent Skills format, contributable) and are discovered by path — clone the two repos side by side and every command finds them.

Quickstart

Requirements: 1× NVIDIA RTX 4090 (≥24 GB VRAM, Linux + EGL), an LLM API key (OpenRouter / Vertex), and uv. First run downloads ~3.5 GB of model weights; the gated SAM3 weights are part of the default perception path, so in practice you also need HF_TOKEN (a free HuggingFace token with SAM3 access) before the first run.

git clone --recurse-submodules https://github.com/graph-robots/graph-as-policy.git
#   ^ submodules are required (vendored sim stack) — if you already cloned
#     plain, run: git submodule update --init --recursive
git clone https://github.com/graph-robots/open-robot-skills.git    # sibling, auto-discovered
cd graph-as-policy
uv sync                                  # engine + LIBERO sim (now baseline)
# `uv sync --extra vertex` instead if you'll use --provider vertex for codegen
uv run gap skills install --all          # per-bundle venvs (sam3, cuRobo, vlm, …)
#   --all includes the heavyweight learned-policy bundles; for one example,
#   `gap skills install --workflow <graph-dir>` installs just what it uses

export HF_TOKEN=...                      # for the gated SAM3 weights
uv run gap skills check --download       # weight prefetch (SAM3 + GDINO) + capability gate

# Pick one LLM provider for codegen + the in-graph VLM. openrouter is the
# default; for vertex, see `docs/source/authoring/llm-providers.md`.
export OPENROUTER_API_KEY=...
MUJOCO_GL=egl uv run gap run examples/libero_quickstart/graph \
  --sim libero_object_all_variance/0
uv run gap viz                           # browse the trial at localhost:9432

See the 15-minute tour for the full walkthrough (clone → run → generate → trace).

Examples

Example	What it shows
libero_quickstart	Hero: vision → OBB grasp → transport, ground-truth verified
grocery_packing	Pack every item with a loop: a graph with a real backward edge
generate_a_graph	Instruction → validated workflow dir; all LLM providers
build_a_graph	Full Python authoring with gap.builder: checkpoints, recovery
agent_quickstart	Claude Code + one skill: sentence → graph → sim success
grocery_fulfillment	Flagship acceptance family; graphs LLM-generated per task
benchmark	Grid harness: smoke → posvar → release gate
steered_policy	Perceive + hover, then hand to a learned VLA policy
collect_and_train	Graph as scripted expert → dataset → train → policy node
cable_ur	Real UR + ZED perception (motion-disabled)
real_franka_pick_place	Real Franka pick-place via robots_realtime

Full gallery with install needs and time estimates in examples/README.md. Real-robot examples — read docs/safety.md first.

Documentation

Full docs site: https://graph-robots.github.io/graph-as-policy/

• Quickstart (15-min tour) — clone → run → generate → trace
• Runtime & schema — workflow JSON, executor semantics, checkpoints
• Skill authoring — Agent Skills format, gap.requires: frontmatter
• LLM providers — openrouter / vertex
• Skill registries — --skills, $GAP_SKILLS_PATH, gap registry …
• CLI reference — every gap verb
• Safety — required reading before any real-robot example

Use with Claude Code & AI agents

GaP ships an agent skill (agent/) that teaches AI coding agents the workflows above:

claude plugin marketplace add graph-robots/graph-as-policy
claude plugin install gap@gap

Then ask: "what can this robot do right now?", "run the quickstart graph in sim", "generate a graph that packs the groceries", "why did this trial fail?". Real-robot commands are gated on explicit human confirmation. Other agents (Cursor, Codex, …) and the no-install path: agent/INSTALL.md.

License & attribution

graph-as-policy and open-robot-skills are Apache-2.0-licensed; they stand on third-party work — LIBERO/LIBERO-PRO, Variational-Automation-Benchmark, robosuite, robots_realtime, pyroki, SAM3, Grounding DINO, and (optionally) NVIDIA cuRobo. Full attribution table: NOTICE.md.