v0.1.0 — Initial Release

CUA v0.1.0 — Initial Release

LLM-powered browser automation for internal dashboard operations. Define known workflows as deterministic YAML playbooks; for everything else, the LLM agent drives the browser autonomously.

Highlights

Multi-Provider LLM Support

• Works with Anthropic, OpenAI, and Google Gemini via PydanticAI
• Switch providers by changing a model string — no code changes needed
• Recommended: gemini-3-flash-preview (fastest), claude-sonnet-4-6 (most precise), gpt-5.4 (strong quality)

Playbook-First Architecture

• Deterministic YAML playbooks for known workflows — zero LLM calls, 1-5s execution
• Selector fallback chains for resilience against UI changes
• Parameter injection, step verification, and data extraction
• Automatic LLM handoff when a playbook step fails — remaining steps passed to the full agent

Browser Agent

• Single browser_dom tool with 9 actions (goto, click, extract, scroll, key_press, wait_for, screenshot, get_dom, execute_sequence)
• execute_sequence batches multiple actions in one tool call — fills a 5-field form in one round-trip
• DOM-first approach (~200-500 tokens) instead of screenshot-first (~1-2K tokens)
• Streaming execution — tool calls execute as they arrive from the API stream
• Adaptive thinking budget and aggressive context pruning

Cognitive Blinders (Safety)

• Proactive observation filtering: the agent only sees DOM elements relevant to its task
• Task scope classification (read/navigate/interact/fill_form) controls visible elements and allowed actions
• Prompt injection detection and redaction in DOM content
• Multi-layer pre-execution checks: deterministic rules → regex fast-path → LLM validation

Modal Deployment

• One-command deploy: modal deploy api/server.py::modal_app
• Each session runs in an isolated sandbox with its own browser, desktop (Xvfb + Openbox), and agent runtime
• Bearer token authentication via CUA_API_KEY
• REST API: create runs, check status, stream events (SSE), stop runs
• Session recordings persisted to Modal Volume

Session Recording

• Every session recorded via Playwright tracing
• Replay in Playwright Trace Viewer with DOM snapshots, screenshots, network requests
• Per-action screenshots saved as JPEGs
• Recordings accessible via API on Modal deploys

Observability

• Built-in OpenTelemetry instrumentation
• Distributed tracing across API → sandbox → agent → browser actions
• GenAI semantic convention attributes (input/output tokens, model, etc.)
• Works with Jaeger, Grafana, Datadog, or any OTLP-compatible collector

API Quick Start

# 1. Create secret
modal secret create llm-secret \
  GOOGLE_API_KEY=... \
  CUA_API_KEY=your-secret-key \
  ENVIRONMENT=production

# 2. Deploy
modal deploy api/server.py::modal_app

# 3. Create a run
curl -X POST https://<workspace>--cua-serve.modal.run/runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{"directive": "Go to example.com and tell me the page title"}'

Local Quick Start

pip install -e ".[dev]" && patchright install chromium

# Playbook (deterministic, no LLM)
python scripts/run_local.py --directive "Cancel order #12345" --playbook cancel_order

# LLM agent (autonomous)
python scripts/run_local.py --directive "Find the latest order on dashboard.internal"

Requirements

• Python 3.13+
• At least one LLM API key (Anthropic, OpenAI, or Google)
• Modal account (for cloud deployment)