perf(demo-agent): stop paying for JSON punctuation and a redundant tool

[debuggingfuture]

A product-demo run measured at $5.36 of model spend — 1,013,992 input tokens against 11,776 output across 110 calls. 95% of the bill is what the agent sends, not what it gets back, and the input is flat across the run rather than growing, which points at a large payload re-sent per action rather than an accumulating history.

Problem & Insight

Two things dominate that payload.

The accessibility snapshot is JSON.stringify of the whole tree, uncapped — roughly 707k of the 1.01M input tokens, 7,069 per call, with one call at ~76,300. The waste is structural, not semantic: JSON repeats the keys "role", "name" and "children" on every one of thousands of nodes, and spends braces and quotes on nodes whose entire content is two short strings. Puppeteer's interestingOnly filter has already dropped the uninteresting nodes, so there is little left to prune — the encoding is the cost.

The screenshot tool spends a full model round-trip per story to nominate a key frame, and play.ts already takes an unconditional final screenshot as a fallback. In 8 of the 10 chapters of the demo that produced these numbers, the prose reaches for it immediately before done, so the nominated frame and the fallback frame are the same picture.

Take

Serialize the tree as indented role "name" prop=value lines. Measured on an app-shaped tree — nav, a 40-row table, a form — 11,438 characters of JSON become 6,399, a 44% cut. Every scalar property survives, so a node's value, disabled, checked and expanded still reach the model, which the play prompt explicitly asks it to reason about; this is a re-encoding, not a filter. A 3,000-node budget bounds pathological pages, and it cuts at a node boundary with the remainder stated, where a slice on the old JSON produced an unparseable string that would sink whichever chapter hit it.

Delete the screenshot tool. ModelAction's screenshot variant and play.ts's handling stay as an inert path — nothing emits one now, they cost no tokens, and the capability is one tool definition away. rationale leaves nav/key/wait, where the argument already is the intent, and stays on click and type, the two actions that actually go wrong.

Key actions

• Consumers must drop "capture a screenshot" from story prose in the same rollout. toolCallToAction's Match.orElse maps an unrecognised tool name to {type:"done", status:"failed"}, so prose that still asks for a deleted tool fails the chapter on the spot.
• The snapshot line format is described to the model in the system note; a consumer with its own prompt override needs the same sentence.

81 tests pass, tsc --noEmit clean. Nine new cases cover the serializer: property preservation, false-boolean dropping, quote-and-newline escaping so a crafted accessible name cannot forge a node line, node-boundary truncation with a descendant-aware remainder, and the size comparison itself.

Not in this PR: routing anthropic/ models to the gateway's native endpoint so the stable prefix can carry cache_control. Prompt caching is unavailable on the OpenAI-compat path this adapter uses today, and Anthropic silently ignores the field rather than erroring, so it needs its own branch and its own review. A probe against the native endpoint confirms the gateway passes cache_control through verbatim and reports cache_creation_input_tokens then cache_read_input_tokens on a repeat call, so the premise holds.

[flaredispatch-fractalboxdev]

AI code review — 💬 Comment

Risk tier: full · 0 critical · 1 warnings · 0 suggestions

Reviewers: security ⚠️ · performance ⚠️ · code-quality ⚠️ · documentation ⚠️ · release-management ⚠️ · compliance ⚠️ · agents-md 1

1. ⚠️ Warning — Snapshot budget does not bound traversal work

📍 packages/demo-agent/src/cdp.ts:67-68

When the 3,000-node budget is exhausted, 'countNodes' recursively walks the entire omitted subtree to compute 'dropped'. A very large or deeply nested accessibility tree can therefore still incur unbounded traversal time or a call-stack overflow, undermining the protection this budget is intended to provide. Use an iterative counter or maintain subtree counts while walking, with a safe cap for remainder reporting.

📋 View full logs & reviewed diff ↗

[debuggingfuture]

Review team ran four lenses over the diff — correctness, tests, security, architecture. Nine findings, all addressed in 113b49c. Four of them are corrections to claims the original commit asserted confidently, two of those in comments describing properties the code did not have.

Two blockers.

backendNodeId and loaderId are set by puppeteer's serialize() on every node — a number and a ~32-char hex string — and both passed the typeof v === "number" | "string" prop filter. Every line carried ~45 characters of junk, ~135KB on a 3,000-node page: a change whose whole thesis is removing per-node repetition was re-adding it. Every test fixture was hand-built, so none of them carried the fields a real tree always has. Now excluded by name, with a test that feeds a node carrying both.

JSON.stringify does not escape U+2028, U+2029 or U+0085, and this format makes the newline structural. A crafted accessible name — attacker-influenceable on any page the agent is pointed at — could forge a node line, or a line byte-identical to the truncation marker, and the model's next output is a click or a type with credentials in scope. This was a regression against the JSON encoding it replaces, where a raw line terminator inside a string created no structure. escapeInline now escapes all three on top of JSON.stringify, applied to the role, the name and every string property value; the marker sits at column 0 behind a lone backslash, which the escaper doubles everywhere else, so no node line can produce it.

Corrections. Puppeteer treats checked/pressed as tristate and emits false when the state is present, so dropping it made an unchecked box indistinguishable from one with no checked state — false is kept for those two only. The "survives as an inert path" claim about the removed screenshot tool was wrong in the other direction: Match.orElse maps an unrecognised tool name to done/failed, so a model emitting it would fail the chapter, not skip it. The legacy name now maps to a 0ms wait, which also removes the "consumers must update story prose in the same rollout" coordination this PR previously required; the genuinely dead ModelAction variant and its play.ts handling are deleted.

Also: dropped the AxNode index signature that was letting a cast do the type's work (puppeteer's SerializedAXNode now assigns structurally, and a rename upstream would fail the build); a per-name 280-char cap, since the budget counts nodes and paragraph-length StaticText names cost more than 3,000 buttons while never tripping the marker; the docblock cut from 25 lines to 10 with the depth-first caveat stated — one wide early subtree can consume the budget and hide the region the model needs; rationale removed from the orphaned schema variants; and the size assertion tightened from "smaller than JSON" to the documented ratio.

89 tests pass (20 in cdp.test.ts, 7 in model.test.ts), tsc --noEmit clean, oxlint clean.