Computer-Use Automation System

An LLM discovers how to accomplish a goal against a live back-office web app once. That run is recorded as a typed, versioned artifact — a reusable capability. Production traffic replays the artifact deterministically, with no model in the loop, and gets back a structured success / known-business-outcome / failure result.

Built for the interface.ai take-home (/REPORT.md has the full design write-up).

Requirements

• Node.js 20+
• A Chromium browser for Playwright (npx playwright install chromium)
• An OpenRouter API key, only for the discovery step (replay never calls the model)

Setup

npm install
npx playwright install chromium
cp .env.example .env   # then put your OPENROUTER_API_KEY in .env

Run without live services

Nothing in this project calls an external service except the model API during discovery, and that only happens if you run npm run discover. Replay, the target app, and the type/schema layer all run fully offline against the local mock app below.

The target surface

npm run target-app starts a small mock "MemberCore Servicing Console" on http://localhost:4100 — a stand-in for a legacy credit-union back-office app. It's deliberately hostile on purpose (see REPORT.md §4): server-rendered HTML, table-based layout, clickable <span onclick> elements instead of real <button>s, no data-testid attributes, an <iframe> for "recent activity," a native confirm() dialog on the risky action, and simulated runtime conditions (member not found, access denied, a transient "loading" interstitial). No real data — see the assignment's ground rules.

npm run target-app
# -> Mock target app listening on http://localhost:4100

Leave this running in one terminal for everything below.

Demo path

1. Discovery (LLM-driven, real model calls)

npm run discover -- --goal "look up member 12345 and read their current savings balance" \
  --target http://localhost:4100/search \
  --name lookup_member_balance

This drives a real Chromium browser with OpenRouter deciding each action from an accessibility-tree view of the page (not screenshots/coordinates — see REPORT.md §1 for why). On success it writes:

• artifacts/lookup_member_balance.v1.json — the reusable artifact
• evidence/discovery-<runId>/log.jsonl — structured step-by-step log
• evidence/discovery-<runId>/*.png — a screenshot at every step

A login/session step is deliberately not part of the recorded flow: the CLI establishes the session cookie directly before handing control to the agent, so credentials never enter the model loop or the artifact (see REPORT.md §6, Safety).

2. Replay (deterministic, no model)

npm run replay -- --artifact artifacts/lookup_member_balance.v1.json \
  --params '{"memberId":"12345"}'

Prints a structured JSON result — status: "success" with outputs, "business_outcome" for a legitimate non-error answer, or "failure" with the failing step and enough detail to debug. Every replay writes its own evidence/replay-<runId>/ with a log and screenshots, ending with a screenshot of either the success state or the point of failure.

Try the same artifact against different inputs to see all three result shapes:

# success
npm run replay -- --artifact artifacts/lookup_member_balance.v1.json --params '{"memberId":"12345"}'

# known business outcome — not an error
npm run replay -- --artifact artifacts/lookup_member_balance.v1.json --params '{"memberId":"99999"}'

# known business outcome — permission denied
npm run replay -- --artifact artifacts/lookup_member_balance.v1.json --params '{"memberId":"40404"}'

3. Risky-action flow (native dialog + non-retryable steps)

A second hand-validated artifact exercises the guardrail-driven risky/irreversible path — selecting an account type, typing a deposit, and clicking through a native confirm() dialog that only auto-accepts because it follows a step recorded as non-retryable:

npm run replay -- --artifact artifacts/open_subaccount.v1.json --params '{"depositAmount":"50.00"}'

4. Human escalation

Pass --escalate true to have any replay failure pause and hand the live browser session to a CLI operator console instead of failing immediately:

npm run replay -- --artifact artifacts/lookup_member_balance.v1.json \
  --params '{"memberId":"12345"}' --escalate true --headless false

At the prompt: state lists the current interactive elements by index, click <index> / type <index> <text> operate the same page a human operator would see, screenshot saves a snapshot, and resume hands control back to automation. A retryable step is retried once automation regains control; a non-retryable (risky) step is treated as handled by the human and the flow moves on, to avoid automation double-submitting it. See REPORT.md §5 for the design behind this and what a production co-browsing console would add.

Repo layout

src/
  target-app/server.ts   mock legacy back-office app (the demo target)
  core/
    types.ts             the artifact schema + replay result contract (Zod)
    perception.ts         accessibility-tree perception + locator resolution
    llmAgent.ts            discovery loop (OpenRouter function calling)
    recorder.ts            executed-step transcript -> Artifact
    replay.ts               deterministic replay engine + error taxonomy
    guardrails.ts           allowlist enforcement + risk classification
    escalation.ts           human handoff on the live session
    logger.ts               structured evidence logging + redaction
  cli/
    discover.ts  replay.ts
config/allowlist.json     domains, routes, and action types the agent may touch
artifacts/                 saved capability artifacts
evidence/                   discovery + replay run logs and screenshots

A note on this sandbox vs. a full submission

Two artifacts are checked in, both produced by hand-authoring the artifact and running it through the real replay engine against the real live target app (real Chromium, real DOM):

• artifacts/lookup_member_balance.v1.json — the search → detail → extract flow. Validated against all three replay result types: success, and both business_outcome paths (member_not_found, access_denied).
• artifacts/open_subaccount.v1.json — the risky, multi-field, dialog-confirmed flow. Validated the guardrail-driven risky/non-retryable path end to end: selecting an account type, typing a deposit amount, clicking through a native confirm() dialog (auto-accepted specifically because it followed a non-retryable step — see REPORT.md §6), and extracting a real confirmation number.

Building these by hand also surfaced and fixed three real bugs, not hypothetical ones: a role-based locator failing to resolve a <span onclick> with no real ARIA role, a substring text-match colliding with an unrelated page heading, and a race condition in the escalation console where piped/fast operator input could let resume return control before an earlier click/type command had actually finished. All three are fixed in the current code.

Project output proof

A successful, real LLM-driven discovery run was completed using openrouter/free. The run generated the artifact artifacts/lookup_member_balance.v1.json and saved logs and screenshots to evidence/discovery-78b1de49/.

Discovery step-by-step trace

1. Start state: The browser opens the search console.

2. Step 0: The agent types the member ID 12345 into the search field.

3. Step 1: The agent clicks the Search button.

4. Step 2: The agent lands on the detail page, locates the cell containing $4210.55, extracts the savings balance, and declares the success checkpoint.

5. End state: The discovery run concludes successfully.

Replay execution proof

The generated artifact replays deterministically without calling any model API.

Running replay for member 12345:

npm run replay -- --artifact artifacts/lookup_member_balance.v1.json --params '{"memberId":"12345"}'

The console logs the run status and identifies the business outcome:

Replay verification for edge cases and risky actions

Replay runs were executed against different parameters and workflows to validate edge-case handling.

1. Member not found: Replaying the search flow for member ID 99999 ends in a known business outcome, capturing the not found screen.

2. Access denied: Replaying the search flow for member ID 40404 matches the access restriction page.

3. Risky actions and dialog confirm: Replaying the sub-account creation artifact (artifacts/open_subaccount.v1.json) navigates the form, handles the initial deposit value, auto-accepts the native browser confirmation dialog for the risky submission step, and extracts the generated confirmation number.