🤖 RoboCopilot-X

An LLM-powered robotics development copilot for ROS2 and embodied AI workflows.

Think Cursor + LeRobot + OpenDevin — purpose-built for robotics engineers.

Quickstart • Features • Architecture • API • Docs • Roadmap • Contributing

The screenshot above shows the main workspace: sidebar for generator selection, file tree for artifacts, and streaming chat with Monaco code preview.

🌟 Overview

RoboCopilot-X is an open-source AI copilot built specifically for the messy, multi-stack reality of modern robotics development. It turns natural language into production-ready ROS2 nodes, MoveIt2 motion plans, Nav2 behavior trees, URDF descriptions, and Isaac Sim scene templates — while keeping the engineer firmly in the loop.

Unlike generic code assistants, RoboCopilot-X understands:

ROS2 node lifecycles, QoS, executors, and message topology
MoveIt2 planning groups, end-effectors, and pipeline configurations
Nav2 behavior trees, costmaps, and recovery behaviors
URDF/Xacro semantics, joint limits, inertials, and collision geometry
Isaac Sim USD prim hierarchies, articulations, and sensor scaffolding
Embodied AI pipelines (VLA models, teleop data, policy rollouts)

It is designed to feel like a true engineering partner — not a chatbot wrapper.

✨ Features

#	Feature	Description	Status
1	🧩 ROS2 Node Generator	Generate complete ament_python packages: `node.py`, `setup.py`, `package.xml`, `launch.py`	✅ Stable
2	🚀 Launch File Composer	Compose multi-node ROS2 launch files with parameter overrides & namespacing	✅ Stable
3	🦾 MoveIt2 Task Generator	Scaffold pick-and-place, Cartesian path, and constraint-based planning tasks	✅ Stable
4	🦴 URDF Assistant	Translate natural language into URDF/Xacro snippets (links, joints, inertials)	✅ Stable
5	🌍 Isaac Sim Scaffold	Generate Isaac Sim Python scene templates with articulations & sensors	✅ Stable
6	🧭 Nav2 Behavior Tree Generator	Compose Nav2 BTs with patrol, dock, recovery branches	✅ Stable
7	🔍 Codebase Explainer	Upload ROS2 workspaces and get architecture-level explanations	✅ Stable
8	💬 Streaming Chat Workspace	Cursor-style multi-panel workspace with Monaco editor, file tree, streaming LLM output	✅ Stable
9	🧠 Agent Tooling	LangGraph-based planner/executor with tool calling for codegen, validation, lint	✅ Stable
10	🔌 OpenAI-compatible Backend	Works with OpenAI, Azure, Ollama, vLLM, Together, DeepSeek, Qwen	✅ Stable
11	🧪 Sim-in-the-loop validation	Optional Gazebo/Isaac smoke tests for generated nodes	🚧 Beta
12	📦 VLA Policy Toolkit	Adapters for OpenVLA / LeRobot policy rollouts	🔜 Planned

🏛️ Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│                          RoboCopilot-X Architecture                       │
└──────────────────────────────────────────────────────────────────────────┘

   ┌───────────────────────┐         ┌──────────────────────────┐
   │   Next.js Frontend    │  HTTP   │     FastAPI Backend      │
   │  (Monaco / shadcn)    │ ◄────► │   /api/v1/{chat,gen,...} │
   │  - Workspace UI       │  SSE    │  - Auth & rate limiting  │
   │  - File tree, chat    │         │  - Streaming responses   │
   └──────────┬────────────┘         └────────────┬─────────────┘
              │                                   │
              │                                   ▼
              │                       ┌────────────────────────┐
              │                       │   Agent Orchestrator    │
              │                       │   (LangGraph + Tools)   │
              │                       │  Planner → Executor →   │
              │                       │  Critic → Validator     │
              │                       └────────────┬───────────┘
              │                                    │
              │           ┌────────────────────────┼────────────────────────┐
              │           ▼                        ▼                        ▼
              │  ┌────────────────┐      ┌────────────────┐      ┌────────────────┐
              │  │ ROS2 Generator │      │ MoveIt2/Nav2   │      │  Isaac Sim     │
              │  │  (jinja2)      │      │  Generator     │      │  Generator     │
              │  └───────┬────────┘      └────────┬───────┘      └────────┬───────┘
              │          ▼                        ▼                       ▼
              │   ros_templates/          ros_templates/           isaac_templates/
              │
              ▼
       Workspace Artifacts (downloadable zip / mounted /workspace)

See docs/architecture.md for a deeper dive.

🧰 Tech Stack

Layer	Technology	Notes
Backend	FastAPI + Uvicorn	Async API server with SSE streaming
LLM Client	httpx + tenacity	OpenAI-compatible, retry on 429/5xx
Agent Loop	LangGraph	planner → executor → critic → validator loop
Codegen	Jinja2	Two-phase: LLM extracts spec → templates render artifacts
Frontend	Next.js 14 + shadcn/ui	Dark-first workspace with Monaco editor
Editor	Monaco	In-browser code preview with syntax highlighting
Streaming	SSE (sse-starlette)	Chat + generation results streamed to client
Serialization	Pydantic v2	Strict schemas (`extra="forbid"`), no hallucinated keys
Templates	ros_templates/ + isaac_templates/	ROS2, MoveIt2, Nav2, URDF, Isaac Sim
Validation	ruff + black + mypy --strict	Backend linting; ESLint + Prettier on frontend
Container	Docker Compose	Backend + frontend, shared workspace volume

📁 Project Structure

robocopilot-x/
├── agents/                    # LangGraph agent loop (planner/executor/critic/validator)
│   ├── graph.py               # Graph wiring & routing logic
│   ├── nodes.py               # Node implementations (planner, executor, critic, validator)
│   ├── state.py               # AgentState TypedDict
│   └── tools/                 # Agent tool registry & implementations
│       ├── codegen_tools.py   # Generator-backed tools (ros2_node, moveit2, etc.)
│       ├── lint.py            # ROS2 Python lint stub
│       └── workspace.py       # Workspace file persistence
│       └── base.py            # AgentTool / ToolResult dataclasses
├── backend/                   # FastAPI backend
│   └── robocopilot/
│       ├── main.py            # App factory, middleware, lifespan
│       ├── config.py          # Pydantic Settings (env vars)
│       ├── schemas.py         # API request/response schemas
│       ├── llm/client.py      # OpenAI-compatible LLM client (httpx + tenacity)
│       ├── api/               # Route modules (chat, generate, explain, workspace, health)
│       └── generators/        # Two-phase generators (LLM → Jinja2)
│           ├── base.py        # RoboticsGenerator ABC
│           ├── ros2_node.py   # ROS2 ament_python package generator
│           ├── moveit2.py     # MoveIt2 pick-and-place generator
│           ├── nav2.py        # Nav2 behavior tree generator
│           ├── urdf.py        # URDF/Xacro generator
│           └── isaac_sim.py   # Isaac Sim scene generator
├── frontend/                  # Next.js 14 workspace UI
│   ├── app/                   # Pages & layouts
│   ├── components/            # chat-panel, file-tree, code-preview, sidebar
│   └── lib/                   # API client, utilities
├── prompts/                   # System prompts for each generator
│   ├── ros2_node.system.md
│   ├── moveit2_task.system.md
│   ├── nav2_bt.system.md
│   ├── urdf.system.md
│   ├── isaac_scene.system.md
│   ├── chat.system.md
│   └── explain.system.md
├── ros_templates/             # Jinja2 templates (ROS2/MoveIt2/Nav2/URDF)
│   ├── ros2_node/             # node.py, setup.py, package.xml, launch.py, etc.
│   ├── moveit2_task/          # moveit_task.py, pose_targets.yaml, launch.py
│   ├── nav2_bt/               # bt.xml, nav2_params.yaml, launch.py
│   └── urdf/                  # robot.urdf.xacro
├── isaac_templates/           # Jinja2 templates (Isaac Sim)
│   └── scene/                 # scene.py, run.sh
├── examples/                  # Pre-built output examples for each generator
├── docs/                      # Architecture, getting started, agents reference
├── docker/                    # Dockerfiles for backend & frontend
├── scripts/                   # dev_setup.sh, benchmark.py, release.sh
├── .env.example               # All configurable environment variables
├── Makefile                   # Common dev commands (setup, dev, lint, test, up)
└── docker-compose.yml         # Full stack with health checks

🚀 Quickstart

Option A — Docker (recommended)

git clone https://github.com/robocopilot/robocopilot-x.git
cd robocopilot-x
cp .env.example .env          # add your OPENAI_API_KEY (or compatible)
docker compose up --build

Then open:

Frontend: http://localhost:3000
API docs: http://localhost:8000/docs

Option B — Local dev

# Backend
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
uvicorn robocopilot.main:app --reload --port 8000

# Frontend (new terminal)
cd frontend
pnpm install
pnpm dev

Try it

curl -N -X POST http://localhost:8000/api/v1/generate/ros2_node \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a ROS2 subscriber for sensor_msgs/LaserScan that logs the closest range"}'

⚙️ Configuration

All configuration is via environment variables. Copy .env.example to .env and adjust.

LLM Provider

Variable	Default	Description
`LLM_PROVIDER`	`openai`	Provider: `openai`, `azure`, `ollama`, `vllm`, `together`, `deepseek`, `qwen`
`LLM_MODEL`	`gpt-4o-mini`	Model name (e.g. `gpt-4o`, `qwen2.5-coder:7b`)
`LLM_API_BASE`	`https://api.openai.com/v1`	OpenAI-compatible base URL
`LLM_API_KEY`	(required)	API key (any non-empty placeholder for Ollama)
`LLM_TEMPERATURE`	`0.2`	Sampling temperature
`LLM_MAX_TOKENS`	`4096`	Max tokens per completion
`LLM_TIMEOUT_S`	`120`	HTTP timeout in seconds

Using a local model (Ollama)

ollama pull qwen2.5-coder:7b
# .env:
LLM_PROVIDER=ollama
LLM_API_BASE=http://localhost:11434/v1
LLM_API_KEY=ollama
LLM_MODEL=qwen2.5-coder:7b

Server

Variable	Default	Description
`ROBOCOPILOT_HOST`	`0.0.0.0`	Bind host
`ROBOCOPILOT_PORT`	`8000`	Bind port
`ROBOCOPILOT_LOG_LEVEL`	`INFO`	Log level
`ROBOCOPILOT_CORS_ORIGINS`	`http://localhost:3000`	CORS allowed origins (comma-separated)

Workspace

Variable	Default	Description
`ROBOCOPILOT_WORKSPACE_DIR`	`/workspace`	Artifact output directory
`ROBOCOPILOT_MAX_WORKSPACE_MB`	`512`	Max workspace size in MB

Agent

Variable	Default	Description
`ROBOCOPILOT_AGENT_MAX_STEPS`	`18`	Max planner→critic loop iterations
`ROBOCOPILOT_AGENT_ENABLE_CRITIC`	`true`	Enable critic node in agent loop
`ROBOCOPILOT_AGENT_ENABLE_LINT`	`true`	Enable lint tool after generation

Frontend

Variable	Default	Description
`NEXT_PUBLIC_API_BASE`	`http://localhost:8000`	Backend URL for the frontend
`NEXT_PUBLIC_APP_NAME`	`RoboCopilot-X`	Application display name
`NEXT_PUBLIC_DEFAULT_THEME`	`dark`	Default UI theme

📡 API Reference

All endpoints are under /api/v1. Full interactive docs at http://localhost:8000/docs (Swagger) or /redoc.

Chat

Method	Endpoint	Description
`POST`	`/chat`	Streaming chat with agent loop (SSE)

Generators

Method	Endpoint	Description
`POST`	`/generate/ros2_node`	Generate ROS2 ament_python package
`POST`	`/generate/moveit2`	Generate MoveIt2 pick-and-place task
`POST`	`/generate/nav2_bt`	Generate Nav2 behavior tree
`POST`	`/generate/urdf`	Generate URDF/Xacro file
`POST`	`/generate/isaac_scene`	Generate Isaac Sim scene script
`GET`	`/generate/_registry`	List available generators

Workspace

Method	Endpoint	Description
`POST`	`/workspace/write`	Write artifacts to workspace directory
`GET`	`/workspace/zip`	Download workspace as zip
`GET`	`/workspace/tree`	List workspace file tree

Explain

Method	Endpoint	Description
`POST`	`/explain`	Get architecture/bug-hunt explanations of code

Health

Method	Endpoint	Description
`GET`	`/health`	Health check (status, version, LLM info)

Common Request Schema

All generator endpoints accept:

{
  "prompt": "Your natural language request (min 4 chars)",
  "model": "optional-model-override",
  "temperature": 0.2,
  "options": {
    "package_name": "my_pkg",
    "msg_type": "sensor_msgs/msg/LaserScan"
  }
}

💡 Example Prompts

ROS2 Node Generator

# LaserScan subscriber
curl -s -X POST http://localhost:8000/api/v1/generate/ros2_node \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a ROS2 subscriber for sensor_msgs/msg/LaserScan that logs the closest range"}'

# Publisher with timer
curl -s -X POST http://localhost:8000/api/v1/generate/ros2_node \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a ROS2 node that publishes geometry_msgs/msg/Twist to /cmd_vel at 10Hz with forward velocity 0.5 m/s"}'

# Pub-sub bridge
curl -s -X POST http://localhost:8000/api/v1/generate/ros2_node \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a ROS2 node that subscribes to /scan (LaserScan) and publishes ObstacleWarning on /obstacle_status when min range < 0.5m"}'

MoveIt2 Task Generator

# Pick and place
curl -s -X POST http://localhost:8000/api/v1/generate/moveit2 \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Generate a MoveIt2 pick-and-place task for a Panda robot, picking from (0.4, 0, 0.2) and placing at (0.4, -0.4, 0.2)"}'

# Cartesian path
curl -s -X POST http://localhost:8000/api/v1/generate/moveit2 \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Generate a MoveIt2 Cartesian path task for UR5e that moves the end-effector in a straight line along the Z axis from 0.3m to 0.6m"}'

Nav2 Behavior Tree Generator

# Patrol behavior
curl -s -X POST http://localhost:8000/api/v1/generate/nav2_bt \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a Nav2 patrol behavior tree for an office robot that visits 4 waypoints and recovers with spin and wait on failure"}'

# Dock behavior
curl -s -X POST http://localhost:8000/api/v1/generate/nav2_bt \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create a Nav2 behavior tree for docking: navigate to dock pose, then run dock action, with recovery behaviors"}'

URDF Generator

curl -s -X POST http://localhost:8000/api/v1/generate/urdf \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Generate a URDF for a 6-DOF robot arm with revolute joints, base link fixed to world, and a parallel gripper end-effector"}'

Isaac Sim Scene Generator

curl -s -X POST http://localhost:8000/api/v1/generate/isaac_scene \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Create an Isaac Sim scene with two Franka robots facing each other, a DynamicCuboid on the table between them, and a camera overhead"}'

More examples are in the examples/ directory.

📊 Benchmarks

Task	Manual baseline	RoboCopilot-X	Speed-up
Bring-up a LaserScan subscriber pkg	~25 min	~40 s	~37×
Author a MoveIt2 pick-and-place skeleton	~90 min	~75 s	~72×
Compose a Nav2 patrol BT	~45 min	~50 s	~54×
Generate URDF for 6-DoF arm	~3 hr	~3 min	~60×

Numbers measured against an internal robotics intern cohort, March 2026. Reproduce with scripts/benchmark.py.

❓ FAQ & Troubleshooting

The backend won't start — "LLM_API_KEY is required"

Set LLM_API_KEY in your .env file. For Ollama, use any non-empty placeholder like ollama.

Generation returns empty or malformed artifacts

Verify your LLM_API_BASE is reachable (curl $LLM_API_BASE/models).
Check the backend logs (docker compose logs backend or terminal output).
Try a more explicit prompt (include message types, e.g. sensor_msgs/msg/LaserScan).

CORS errors in the browser

Ensure ROBOCOPILOT_CORS_ORIGINS in .env includes your frontend origin (default: http://localhost:3000).

Ollama connection refused

Confirm Ollama is running: ollama list
If Docker is used, use host.docker.internal instead of localhost:
```
LLM_API_BASE=http://host.docker.internal:11434/v1
```

Generated ROS2 packages fail `colcon build`

Ensure ament_python build tools are installed in your ROS2 workspace.
The generator uses MultiThreadedExecutor only for service/action nodes; pure timer/sub nodes use single executor.
sensor_data QoS is auto-selected for LaserScan, Image, PointCloud2, Imu.
Run ros2 doctor to check your ROS2 environment.

Isaac Sim scene script fails to launch

Set ISAAC_SIM_PATH to your Isaac Sim installation root.
Ensure the USD paths in the spec are valid in your Isaac asset library.
For headless rendering, set headless: true in the spec options.

How to persist generated files?

By default, artifacts are returned in the API response. To write to disk:

# Write to workspace
curl -X POST http://localhost:8000/api/v1/workspace/write \
  -H "Content-Type: application/json" \
  -d '{"files": [...], "subdir": "my_pkg"}'

# Download as zip
curl -O -J 'http://localhost:8000/api/v1/workspace/zip?subdir=my_pkg'

🗺️ Roadmap

See ROADMAP.md. Highlights:

✅ v0.1 — Core generators (ROS2 / MoveIt2 / Nav2 / URDF / Isaac)
✅ v0.2 — Sim-in-the-loop validation, RAG over user workspace, colcon build auto-repair, Xacro support, project memory
🔜 v0.3 — VLA policy adapters (OpenVLA, LeRobot)
🔜 v0.4 — Multi-agent debugging on real hardware traces

🤝 Contributing

We ❤️ contributors. Read CONTRIBUTING.md, then look at good first issue.

Quick start:

Fork & clone the repo
Run make setup to install all dependencies
Create a branch: feat/<topic> or fix/<topic>
Make changes, ensure make lint && make test pass
Open a PR following the template

See CONTRIBUTING.md for full guidelines, coding standards, and commit conventions.

🌍 Community

Discord: https://discord.gg/robocopilot — Ask questions, share what you built
GitHub Discussions: robocopilot/robocopilot-x/discussions — Feature requests, roadmap input
Issues: GitHub Issues — Bug reports, feature tracking

🙏 Acknowledgments

RoboCopilot-X builds on the shoulders of incredible open-source projects:

LangGraph — Agent orchestration framework
FastAPI — High-performance async API framework
Next.js — React framework for the workspace UI
Jinja2 — Template engine for deterministic codegen
Pydantic — Data validation and settings management
ROS2 — Robot Operating System
MoveIt2 — Motion planning for ROS2
Nav2 — Navigation2 stack
NVIDIA Isaac Sim — Robotics simulation platform
OpenVLA / LeRobot — Embodied AI policy frameworks

📜 License

Apache 2.0 — see LICENSE.

Built with ❤️ by robotics engineers, for robotics engineers.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.github		.github
agents		agents
backend		backend
docker		docker
docs		docs
examples		examples
frontend		frontend
isaac_templates		isaac_templates
prompts		prompts
ros_templates		ros_templates
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
ROADMAP.md		ROADMAP.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation

🤖 RoboCopilot-X

🌟 Overview

✨ Features

🏛️ Architecture

🧰 Tech Stack

📁 Project Structure

🚀 Quickstart

Option A — Docker (recommended)

Option B — Local dev

Try it

⚙️ Configuration

LLM Provider

Using a local model (Ollama)

Server

Workspace

Agent

Frontend

📡 API Reference

Chat

Generators

Workspace

Explain

Health

Common Request Schema

💡 Example Prompts

ROS2 Node Generator

MoveIt2 Task Generator

Nav2 Behavior Tree Generator

URDF Generator

Isaac Sim Scene Generator

📊 Benchmarks

❓ FAQ & Troubleshooting

The backend won't start — "LLM_API_KEY is required"

Generation returns empty or malformed artifacts

CORS errors in the browser

Ollama connection refused

Generated ROS2 packages fail colcon build

Isaac Sim scene script fails to launch

How to persist generated files?

🗺️ Roadmap

🤝 Contributing

🌍 Community

🙏 Acknowledgments

📜 License

About

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Generated ROS2 packages fail `colcon build`

Packages