webpilot

Browser automation that thinks — an LLM agent that plans, acts, and recovers instead of following brittle scripts.

The problem

Traditional RPA scripts break the moment a website changes a button, renames a field, or shows an unexpected popup. Maintaining selector-based automation is endless.

What this does

Give webpilot a goal, not a script:

"Take the data from samples/invoice.txt and submit it in the vendor form"

The agent:

1. Reads the source document (LLM extraction into a strict invoice schema)
2. Opens the page and looks at it (URL + title + simplified DOM → LLM)
3. Plans the next action as strict JSON: click / type / select / scroll / navigate / upload / wait
4. Executes via Playwright, observes the result, re-plans
5. Recovers from failures — every error is fed back into the next planning round instead of crashing

Every run captures per-step screenshots to runs/<timestamp>/step-N.png and ends with a report: done, failed, budget_exhausted, or error.

Architecture

flowchart LR
    G[Goal + source doc] --> PL[Planner<br/>LLM]
    PL --> AC[Action executor<br/>Playwright]
    AC --> OB[Observer<br/>URL + DOM summary]
    OB --> PL
    AC --> DONE[Goal check /<br/>result report]

Loading

The observe → plan → act loop continues until the goal is met or a step budget (default 20) is exhausted.

The LLM must answer with exactly one JSON action from a strict schema — hand-validated, unknown fields rejected. Invalid output gets one repair round with the precise rejection reason; code fences and stray prose are stripped automatically. Example:

{ "action": "type", "selector": "#vendor-name", "text": "Northwind Traders Ltd.", "reason": "fill the vendor field from the extracted invoice" }

Quick start

git clone https://github.com/Adi40709/webpilot
cd webpilot
cp .env.example .env      # LLM key, or point it at a local llm-gateway / Ollama

# local browsers (one-time), or skip this and use docker compose instead
npm install playwright && npx playwright install chromium

node run.js --goal "Fill the demo vendor form with the extracted invoice data and submit it" \
            --source samples/invoice.json

Or fully containerized — the Playwright base image ships the browsers, zero downloads:

docker compose up          # add --profile ollama for a bundled local model

Ships with a built-in demo form so the killer demo runs with zero setup. To prove the point about resilience, run the same goal against samples/demo-form-v2.html — a redesigned version of the form where every field id and label was renamed. Script-based automation breaks; webpilot just reads the new page:

node run.js --goal "Submit the invoice on the form" --source samples/invoice.txt \
            --url "file://$(pwd)/samples/demo-form-v2.html"

node run.js --help shows all options (--budget, --headed, …).

Tech stack

Node.js (zero runtime dependencies) · Playwright (optional peer — only needed for real runs) · works with any OpenAI-compatible LLM: OpenAI, Ollama's /v1, or llm-gateway · Docker

Project structure

run.js               # CLI entrypoint: env, args, logging, run report
src/
  agent.js           # the observe → plan → act loop, step budget, recovery
  schema/            # strict JSON action schema + hand-rolled validation
  planner/           # prompt templates, LLM planning, JSON repair + retry
  executor/          # Executor interface, Playwright impl (lazy-loaded),
                     #   in-memory FakeExecutor for browser-free tests
  observer/          # page snapshot → compact observation for the LLM
  extract/           # document → structured invoice data (strict schema)
  llm/               # fetch-based chat client + scripted FakeLLM
  util/              # loose-JSON parsing, tiny .env loader
samples/             # demo form (+ redesigned v2) and sample invoice (.json/.txt)
tests/               # node --test suite — runs without any browser or network

Development

npm test             # node --test — 53 tests, no browser, no network, no API key

The Playwright executor sits behind a small interface and is imported lazily, so the whole agent — schema, planner repair loop, recovery, budget exhaustion, extraction — is tested against FakeExecutor + FakeLLM. CI runs the suite on Node 18 and 20.

Roadmap

• Week 1 — Playwright wrapper + page-observation loop
• Week 2 — action planner with strict JSON action schema
• Week 3 — error recovery + the invoice → web form demo (incl. the redesigned-form variant)
• Week 4 — README, CI, Docker image
• Demo video / gif
• PDF source extraction (pdf-parse) — today: .txt / .json
• Screenshot-to-LLM observation for vision models (DOM summary only today)
• Later — parallel sessions, human-approval mode for sensitive actions

License

MIT