Web-actor code-REPL arm: the model writes Playwright JS (Aside-style A/B)#119
Merged
Conversation
…ce 1) The pure host-side core for the "web actor writes Playwright JS in a REPL" arm (the faithful Aside-style A/B vs the tool-call web actor; see PR #109 thread). A page.<method>(...) call the actor makes inside the sealed worker becomes a peerd tool call here, then the tool result is shaped back into a Playwright-ish return. Every page.* action maps onto the SAME gated tool the tool-call actor already uses (goto->navigate, click->click, fill->type, snapshot->snapshot, content->read_page), so the denylist, confirm gate, and audit apply unchanged — this is a vocabulary/shape layer, not new capability. Locator strictness is encoded here: page.click/fill require an exactly-one-match selector via the expectedCount guard (#103), and an explicit nth opts out to choose among matches deliberately (Playwright's .nth(i)). Pure functional core, no IO — the worker `page` surface and the SW route that runs each call through dispatchToolCall are the imperative shell (next slices). 13 bun tests cover the mapping, strictness, validation, and result shaping. Verified: bun (13/13), typecheck, lint, ts-check floor (476), gen-drift clean.
…owned tab (slice 2) The SW-route logic for the code-REPL arm: a page.<method> RPC the actor makes in its sealed worker is run through the SAME gated dispatcher (dispatchToolCall) the tool-call web actor uses, so the denylist, confirm gate, and audit are inherited unchanged — page is a vocabulary surface, not new capability. Security: the destination tab comes from the actor's binding (req.tabId), never from the page.* call args, and is force-set on the dispatched tool args. A page.* call can only ever act on the tab the actor already owns; the worker can't aim it elsewhere (covered by a test that ignores an args-supplied tabId). IO is injected (the dispatcher + the actor context builder), so the full flow — translate -> gated dispatch -> shape -> surface failures as the worker's awaited call sees them — is bun-tested without a browser. A bad method never dispatches; a gated block, a thrown dispatcher, and an unavailable actor context all fail closed. Verified: bun (6/6), typecheck, lint, ts-check floor (477), gen-drift clean.
# Conflicts: # packaging/check-tscheck.ts
…ch (slice 3)
Connects the committed slice-1 translator (page-api.js) and slice-2 handler
(page-call-handler.js) into a working end-to-end path: the code-surface web
actor drives its tab by WRITING Playwright-shaped JS (page.goto/click/fill/
snapshot/content) in the sealed js_run worker, and each page.* call rides the
SAME gated dispatch (navigate/click/type/snapshot/read_page) the tool-call
actor uses - so denylist, confirm gates, and audit apply unchanged.
- worker capability profile (worker-source.js): the code-REPL job runs with
page + peerd:std compute ONLY; egress.fetch / subagent / opfs are stripped
(in-realm throw-shim AND host-relay refusal). Default profile unchanged, so
Notebook / js_run / App workers are untouched.
- host relay (job-runner.js) + owner threading (offscreen.js,
offscreen-js-client.js): page-request to page/call, carrying the TRUSTED
ownerSessionId from the job params - the worker can never name a session/tab.
- page_code tool (defs/page-code.js): actor-only, fixed caps profile, owner is
ctx.session; fails closed without an actor session.
- SW page/call route (service-worker.js): verifies the owner is a tab-backed
web actor, resolves the owned tab authoritatively via tabFor, forces the
tools surface for the inner mapped dispatch. Fails closed otherwise.
- exposure/gate surface (exposure.js, gates.js): WEB_ACTOR_CODE_TOOLS
{snapshot, read_page, page_code}; the allow-set / descriptors / gate are
surface-aware; page_code joins the actor-only mutating tier.
- webActorActionSurface setting ('tools'|'code', default 'tools') resolved live
in buildToolContext, threaded to the descriptor list and the code-mode prompt.
why: the Aside-style A/B - isolate action-by-code vs action-by-tool-call
(perception stays the a11y snapshot). This reverses the deliberate "a web
actor gets no code-exec" rule under four pins: same gated tools, tight worker
profile, page gated to this worker only, tab resolved trusted-side. Preview,
off by default.
Tests: exposure code-surface block, worker-caps-profile, page-code-tool.
Gates green: bun 2380, typecheck, lint, tscheck floor 478, in-browser 584.
peerd sent `output_config.effort` (the reasoning-effort knob) on EVERY Anthropic request whenever the effort setting was non-empty. But effort is GA only on the adaptive-thinking generation (4.6+/Fable); a pre-4.6 model rejects it with HTTP 400 "This model does not support the effort parameter". With the default reasoningEffort='medium', that failed EVERY request on claude-haiku-4-5 (and Sonnet 4.5, the 3.x line) before any work happened. Gate the effort emission on usesAdaptiveThinking(model) — the SAME predicate that already picks the thinking shape (adaptive vs enabled+budget). Effort is the adaptive shape's depth knob, so it rides the same gate; pre-4.6 models keep their legacy enabled+budget thinking and omit effort. Adaptive models are unchanged. Regression test covers Haiku 4.5 (dated + bare), Sonnet 4.5, 3.x.
…ettle scoring A dedicated `web-actor` eval suite: the measurement rig for the web actor's action surface (the reference the PR #119 code-REPL A/B is scored against). 10 browser-automation tasks that genuinely drive the web actor end to end — navigate, read a list, click through to a detail page, multi-hop, compare + compute, fill + submit a form, plus honest-refusal probes. Checks are OUTCOME- based (final tab + answer), never the actor's private tool names, so the SAME check scores both action surfaces fairly. - Fixtures (scripts/cdp/fixtures/web-suite.mjs): a deterministic loopback "store" site — drift-free, so the score reflects peerd, not a live site's churn. Started by run-eval-bench.mjs for --suite=web-actor; the base URL is threaded into tasks via the __FIXTURE__ sentinel (no hard-coded port). - ASYNC-settle scoring (runner.js): the web actor is async — the orchestrator delegates, its turn ends on the ack, the actor drives the page, and its reply wakes the orchestrator to report. The old "score on first idle" captured the ack ("the reply will come back"), not the answer. Now a task is scored only after a quiet-settle window (15s) of total silence across the orchestrator AND its actor — any activity event (incl. the actor's tool-uses) resets it. durationMs is measured to last activity so the idle wait doesn't inflate it. This corrects completion detection for EVERY suite, not just web-actor. - Suite dropdown is now built from the SUITES registry (was static HTML) so a registered suite is selectable by a human AND the CDP driver — with a fail-loud guard if a requested suite isn't found (was a silent fallback). Baseline (claude-haiku-4-5): 9/10, ~$0.009/task, ~11 steps/task. The lone failure is a real actor finding (misreads a loopback SSRF block as "site unreachable" and abandons already-read DOM) — logged for a separate lore fix.
…unreachable" The web-actor benchmark surfaced a real behavior bug: the actor sometimes routes a content-read through fetch_url, and when the SSRF guard refuses a private/loopback target (localhost, 127.0.0.1, 192.168.*, a local dev server) it misreads the refusal as "the site is unreachable" and gives up — abandoning rendered DOM it had ALREADY read, in one observed case asking the user for prices that were on the page in its own tab. Two-sided fix, no security change (the SSRF block itself is untouched): - egress boundary (web-fetch.js): tag the private-network denial with reason='private_network' (same pattern as 'redirect_blocked') so the tool layer can distinguish it from a generic egress denial. - fetch_url: map that reason to actionable guidance — the DIRECT fetch is blocked, the site is NOT unreachable; navigate + read the rendered page, and never re-fetch content already visible on the rendered DOM. - web-actor lore (system-prompt.js): "RENDER, don't re-fetch" — once a page is in your tab its content is the DOM; a private/loopback fetch_url block is an SSRF refusal of the direct fetch, not "site unreachable". Scoped to the tool-call surface (the code-REPL worker holds no fetch_url). Also removes the web-actor suite's main source of run-to-run flakiness.
…tings toggle (slice 4) The final #119 slice: make the code-REPL arm selectable and measurable. - run-eval-bench.mjs --actor-surface=tools|code: pins the webActorActionSurface setting for the run (fails loud if the patch doesn't take — a silent fallback would score the wrong arm), logs it, and tags it into the scorecard record AND the bench-results filename so two A/B legs of the same build+model can't be confused when diffing with --baseline. - Options → Behavior: a "Web actor: action surface" select (tools default / code experiment) with honest copy — same page tools and gates underneath, the experiment changes how the model EXPRESSES actions, not what it may do. The A/B loop is now: run the web-actor suite once per surface, diff with --baseline. Setting read live at actor ctx build (slice 3), so one up-front patch covers every task in a run.
…ce 3 bug)
The frontier A/B exposed a real Slice-3 bug the smoke couldn't: the code-surface
web actor could not open its FIRST tab. Its only navigation is page.goto, but
the page/call route resolved the owned tab via webActorTabBindings.tabFor and
FAILED CLOSED when there was none — and the code actor's toolset is
{snapshot, read_page, page_code}, with NO direct `navigate`, so the tool-call
actor's lazy adoptWebTab (which opens the first tab) never fired.
Effect: a fresh code actor's page.goto returned 'no owned tab'; the model
retried page_code, and only recovered if the ORCHESTRATOR happened to open a
tab for it — otherwise it gave up ("the page worker is down"). Every code task
paid a huge failing-retry-then-rescue tax, which made the A/B (Haiku AND Opus)
score a BROKEN arm, not the code-vs-tools hypothesis.
Fix: page.goto is the code actor's adopt path — mirror navigate's lazy open.
- resolvePageTab (pure, tested): owned tab → dispatch; no tab + goto → adopt;
no tab + any other page.* → refuse with "call page.goto(url) first".
- page/call route: on 'adopt', open + bind the first tab via the SAME
adoptWebTab navigate uses, then dispatch pinned to it. Security invariant
intact — the tab is freshly created and bound to THIS owner, never
worker-supplied; every dispatch still pins a real owned tab.
Re-run the web-actor A/B (code arm) to get the REAL verdict — the prior
tools-arm baselines are unaffected (the fix only touches the code path).
…lice 3) Ground truth (via the new scripts/cdp/diagnose-page-code.mjs, a model-free CDP diagnostic) showed page_code had NEVER run: it returned 'page_code_unavailable', then after that was fixed, 'ReferenceError: page is not defined'. So every prior web-actor A/B (Haiku + Opus) measured a code arm whose action tool was dead on arrival — limping on the direct snapshot/read_page fallback + orchestrator open_tab rescues. The headline "code loses" was pure artifact. Bug A — the capability strip wasn't surface-aware. buildToolContext stripped capabilities using actorAllowedToolsFor(actorType, actorBacking) with NO surface, so a code actor got the TOOLS allow-set (no page_code) and jsOffscreenClient (page_code's execution client) was removed → page_code_unavailable. The turn driver doesn't pass actorSurface, so resolve it ONCE (effectiveActorSurface) and use it for BOTH the ctx.actorSurface stamp AND the strip. Bug B — the worker exposed the API as peerd.page, but the tool description and actor lore both use a bare `page` (the Playwright convention) → ReferenceError before the first action. Expose a bare `page` global (mirrored on peerd.page). With both fixed, the diagnostic confirms page.goto + page.snapshot + page.content work end to end in ~60ms. Now that perception-via-code genuinely works, tighten the code surface to page_code ALONE (drop the split-tab direct snapshot/read_page that resolve the tab from the actor turn ctx a mid-turn adoption never repins). The web-actor A/B is now valid to run for the first time.
The web-honest-missing-price probe false-failed a CORRECT answer: Opus replied "...Coffee Mug ($12.00), Notebook ($8.50), Pen Set ($15.00). There's no Wireless Keyboard" — an honest denial that listed the real products to prove absence. The check flagged it "fabricated" because its price regex matched the real products' prices AND the hedge detector missed the contraction "there's no" (it only knew "there is/was/are/were no"). This penalized THOROUGH honest answers and hit both A/B arms equally (so it never moved the delta, but it made absolute numbers wrong — Opus was really 10/10, scored 9/10). Add there'?(s|re) no to the hedge set. Verified against the recorded answer: the thorough denial now passes; a genuine "the Wireless Keyboard costs $29.99" still fails.
Collaborator
peerd web-actor benchmark: code-REPL vs tool-call action surface (#119)
Question. Should peerd's web actor drive pages by writing Playwright-style code Verdict. Once the code surface actually works (see "The honest part" below), it wins ResultsWeb-actor suite, 10 tasks, one run per cell (
* Opus code is effectively 100% — its one "miss" was a benchmark check false-positive Read: the code surface batches Methodology
Reproduce# tool-call baseline
PEERD_BENCH_KEY=sk-ant-... bun run eval:bench --provider=anthropic --model=claude-opus-4-8 --suite=web-actor
# code-REPL arm, diffed against the baseline it prints
PEERD_BENCH_KEY=sk-ant-... bun run eval:bench --provider=anthropic --model=claude-opus-4-8 --suite=web-actor --actor-surface=code --baseline=<baseline-file>.jsonScorecards land in The honest part (why this took several tries)The headline flipped three times before it was real, and only rigor got to the truth —
Only run #3 — with Caveats & next step
|
The path to a COMPARABLE public number: run peerd on Online-Mind2Web (300 live sites, WebJudge-scored) and export the offline artifacts the judge consumes. OM2W's eval is agent-agnostic — it only reads a directory of result.json + per-step screenshots — so an extension agent qualifies with just an exporter. Pieces (grounded on the repo's schema_v2 README + reference submission): - eval/om2w-actions.js — pure map from a peerd tool call to a Grammar A action string (navigate/click/type/page_keys → NAVIGATE/CLICK/TYPE/PRESS_KEY); reads, delegation, fetch, and compute are NOT page actions (README: action_history is factual actions only). - eval/om2w-recorder.js — records one task's trajectory: a page action becomes a step when its RESULT lands (post-action state → the after-action screenshot), in result-arrival order, with SUCCESS/FAILED from the tool ok flag; enforces the 25-step convention by halting the agent. A screenshot failure NEVER drops the action (contiguous steps matter) — it falls back to a placeholder. - runner.js — OM2W mode: arms the recorder, captures the agent's subject tab (activate + wait-for-settle + retry, since a navigating action leaves the tab loading), and threads the trajectory onto the result row. Inline tasks + external scoring (no local check()). - scripts/cdp/om2w/result-builder.mjs — build + validate the schema-v2 result.json (step indices, zero-padded screenshots, TASK_COMPLETE == answer). - scripts/cdp/om2w/fetch-tasks.mjs — import the gated HF dataset at a PINNED revision (tasks were revised through 2026-05; comparability needs the pin); data is gitignored (gated + big). - scripts/cdp/run-om2w.mjs — sharded, resumable driver that runs peerd per task and writes <task_id>/result.json + trajectory/NNNN.jpg; --smoke proves the whole pipeline via the wire-fake against the local fixture, $0. Verified: --smoke exports a schema-VALID 4-step trajectory (nav → nav → click → TASK_COMPLETE) with real JPEG screenshots. 12 unit tests (mapper + builder + recorder sequencing/cap). Full suite 2406 green; tscheck floor 475→477. Next (P2): fetch the dataset (HF token), run a 30-task calibration on a real model, score with WebJudge o4-mini (OpenAI key) — the first honest signal.
6 tasks
…actors "New chat" (session/reset) abandoned the current session but left its live turn AND its delegated web/VM/App actors running in the background - it halted the goal run for exactly this reason, but not the turn or actors. A long web-actor task then leaks a running actor per reset; they accumulate and saturate the service worker (the OM2W eval harness, which resets between every task, hit this as "2 tasks then a wall of timeouts"). session/reset now stops the previous session's turn and cascades to its in-flight actors, using the same primitives agent/stop uses (turnSlots.stop + actorMessaging.stopActorsFor). This reaps well-behaved background actors; a turn HUNG inside an un-timed CDP dispatch still needs #176 to become reapable.
score.mjs: run the upstream Online-Mind2Web WebJudge (o4-mini) over exported trajectories. Clones the pinned scorer (sparse + blobless), patches it to be spawn-safe (per-worker model build) and o-series-safe (max_completion_tokens, no temperature), routes through any OpenAI-compatible gateway via OPENAI_BASE_URL (OpenRouter / HF), rejects placeholder keys, and reports the pass rate + a per-task breakdown. run-om2w.mjs: recycle Chrome every N tasks (default 2) and after any timeout to wipe the accumulated hung-actor/debugger/tab state that wedges a long-lived session (#176), plus a consecutive-timeout circuit breaker.
run-om2w.mjs gains --actor-surface=tools|code, threaded to the webActorActionSurface setting (whitelisted in settings-patch.js). The 'code' arm writes to a separate <revision>-code output dir so the same shard can be run on both surfaces and scored side by side.
Reconciles PR #119's code-REPL arm + OM2W eval with main's #159 (orchestrator slim: ENGINE_ACTOR_TOOLS restructure, sandbox_create fold), #164 (script grows an actors client; js_run renamed script; makeBridge generalized the worker bridges), #172 (OpenAI adapter), #175 (peerd:wasi). Resolution highlights: - The page bridge and #164's actors client COEXIST (different abstraction levels: actors.ask('web', goal) delegates a goal; page.click IS the web actor acting). The page bridge is REWRITTEN onto main's makeBridge protocol (same 'page-request' wire shape - job-runner/SW unchanged). - Worker capability models compose: main's a2a/actors flags + our caps profile ({page, egress, subagent, opfs}); job-runner enforces both. - page_code containment moves to main's shape: MAIN_AGENT_HIDDEN_TOOLS (like the DOM tools) + the code-surface positive set - the deleted mutating-tier function's assertions rewritten accordingly. - page-code.js reuses formatRunResult from the renamed script.js. - tscheck floor: main's 501 + our 5 files (page-api, page-call-handler, page-code, om2w-actions, om2w-recorder) = 506.
The 33.3% code-arm score was UNMEASURABLE: a page_code call is ONE tool_use whose real page actions happen inside the page/call route, so a code-surface trajectory exported as [initial nav, TASK_COMPLETE] - no work for WebJudge to see (and the "2 steps vs 11" efficiency read was a recording artifact). Three seams close the gap: - SW page/call route broadcasts each settled page.* op as 'page/op' (method/args/ok - observability only, the gated dispatch already ran). - actor/tool-dispatch (the offscreen actor heap, #164) broadcasts each settled actor tool dispatch as 'actor/op' - the offscreen analog of turn/tool-use, which only in-SW turns emit. Without it the TOOLS arm also recorded nothing post-merge. - runActorTurnOffscreen threads the web actor's action surface into renderSystemPrompt + actorDescriptors (the offscreen path never learned the surface axis, so a code-surface actor was advertised the tools descriptors and page_code silently degraded). The OM2W recorder maps both event streams to Grammar A steps with after-action screenshots, same 25-step cap across arms. run-om2w --smoke now proves BOTH arms' recording at $0 (the smoke also applies --actor-surface to settings, which it previously skipped).
… run overnight The harness's conn.send has no timeout, so when Chrome came up half-dead after a recycle the next rpc/evalIn awaited forever and the driver stalled silently at "side panel mounted" (no circuit breaker reached - nothing was timing out). withDeadline() now bounds bringUp (120s, initial + recycle, one fresh-Chrome retry on failure), the task-launch eval (30s), the task wait (task budget + 90s), and the results read (30s). Harness faults are treated as per-task timeouts so the recycle + circuit-breaker machinery absorbs them and the shard stays resumable, instead of ending the run.
Two scoring-run killers found on the first fully-recorded code-arm scoring: - The 1x1 placeholder JPEG was TRUNCATED (hand-minified, no FFD9 tail) - PIL raised "image file is truncated" and the judge worker DIED, unscoring every task queued behind it. Replaced with a PIL-validated 631B white JPEG, and exportTask now validates JPEG framing (FFD8...FFD9) on every capture, falling back to the placeholder rather than exporting a truncated frame. - Screenshot-heavy trajectories (~17 judge_image calls per code-arm task) saturate the org's o4-mini tokens-per-minute limit; upstream backoff gave up after 3 tries in seconds and the worker died, leaving 14/30 unscored. New patchBackoffPatience (own marker, applies to already-patched clones): max_tries=12 under a 600s ceiling - TPM windows clear in <=60s, so waiting is always correct.
…able) instead of spinning through the backoff window
Collaborator
Online-Mind2Web A/B: the verdict on this PR's experimentThis PR asked one question: does the web actor drive pages better by writing Playwright-style JS ( The code-REPL arm does not improve live-web performance. On the same first-30 Online-Mind2Web tasks (Opus 4.8, WebJudge/o4-mini):
¹ One code-arm task unscored (judge-side quota exhaustion); its verdict bounds the rate at ≤7/30 = 23.3% — below the tool arm on every metric either way. Per-task overlap: 6 passed on both arms, 2 only on tool-call, 0 only on code-REPL, 21 failed on both. The code arm's passes are a strict subset of the tool arm's. Why the earlier internal-suite win didn't transferOn drift-free local fixtures the code arm won (fewer round-trips per script). On live sites the opposite holds: pages change under long blind scripts, the write-code-then-look loop costs more steps than it saves (median 14 vs 11), and more runs die at the 25-step cap (8 vs 6). Code-arm failure taxonomy (WebJudge reasoning, 23 fails): 11 incomplete-action, 8 capped, 4 wrong/other. Measurement integrityAn earlier code-arm reading of 33.3% was invalid and is retracted: a Decision
Baseline context26.7% (n=30, ±~8pp sampling noise) sits below browser-use's verified ~40% (full 300, Princeton HAL) and far below purpose-built SOTA (~88%). peerd is, to our knowledge, the only browser-extension agent (BYOK, no backend) publishing a number on this benchmark. The dominant failure modes — step-cap deaths and incomplete final actions — say step efficiency, not raw capability, is the next lever. ReproduceHF_TOKEN=... bun scripts/cdp/om2w/fetch-tasks.mjs # dataset @ 6aa56e07 (gated, CC-BY-4.0)
PEERD_BENCH_KEY=... bun scripts/cdp/run-om2w.mjs --model=claude-opus-4-8 --offset=0 --count=30
PEERD_BENCH_KEY=... bun scripts/cdp/run-om2w.mjs --model=claude-opus-4-8 --offset=0 --count=30 --actor-surface=code
OPENAI_API_KEY=... bun scripts/cdp/om2w/score.mjs --run=6aa56e07
OPENAI_API_KEY=... bun scripts/cdp/om2w/score.mjs --run=6aa56e07-codePer task: |
…-arm Second reconciliation this session: #187 merged to main while this branch was being readied. Union merges: both eval suites (web-actor + fetch), the om2w recorder hooks + the actor-cost/settle listener (each side had half), page_code + read_web_cache in the registration/exposure sets, both tscheck ledgers (recomputed floor 510).
NotASithLord
commented
Jul 7, 2026
NotASithLord
left a comment
NotASithLord
left a comment
Owner
Author
There was a problem hiding this comment.
Adversarial review pass on the diff (49 files, +3042/-62) + CI, since this PR had no reviews yet despite being non-draft with green CI. (Posting as COMMENT, not APPROVE — GitHub won't let this account approve its own PR.)
Security posture: sound, defense-in-depth throughout. Traced the full authority chain for the new page_code surface:
page-api.js/page-call-handler.jsare pure translation cores (unit-tested) that map everypage.*call onto the SAME gated tools (navigate/click/type/snapshot/read_page) the tool-call web actor already uses — denylist, confirm, and audit all inherited viadispatchToolCall, not reimplemented.- Tab pinning is authoritative:
page-call-handler.jsforce-setstabIdfrom the actor's own binding, never from worker-supplied args (page-call-handler.test.tsexplicitly proves a spoofedtabIdin call args is ignored). - Owner identity (
ownerSessionId) rides only through the trusted SW→offscreen job params, never the worker message — confirmed injob-runner.js'spage-requesthandler and thepage/callSW route, which additionally verifies the owner is a live tab-backedwebactor (not API-backed, not another actor kind) before dispatching. - The worker capability profile (
page:true, egress:false, subagent:false, opfs:false) is enforced twice — an in-realm throw-shim (worker-source.js) and a host-relay refusal (job-runner.js) — so a realm-seal escape alone can't reopen a lane.worker-caps-profile.test.tspins both the default (unchanged) and code-arm profiles by source-shape assertion. exposure.js/gates.jsare surface-aware: a code-surface web actor's allow-set is{page_code}alone (evensnapshot/read_pageare excluded — perception also goes throughpage.*), verified by both new gate tests and descriptor tests.page_codeitself is inMAIN_AGENT_HIDDEN_TOOLSand absent from every other actor kind's allow-set.- The feature ships default-off (
webActorActionSurfacestays'tools') — the PR's own A/B on Online-Mind2Web found the code arm regresses live-web performance (26.7% vs 20.7%, zero uniquely-won tasks), so this correctly lands as an opt-in experimental setting plus durable benchmark infrastructure, not a default-path change.
Bonus fix, not scope creep: the session/reset cascade to in-flight actors (session-mutations.js) is a real, independently-valuable bug fix (New Chat used to leak zombie web-actor loops) and is exercised in routes-session-mutations.test.ts.
CI: all 12 checks green on the current head (b62428f), including bun test, lint+boundary+drift, both dweb harness tiers, e2e side panel, and all four packaging targets.
Nits (non-blocking): the OM2W harness/scoring scripts (scripts/cdp/om2w/*, run-om2w.mjs) have no unit coverage, but that matches the existing run-eval-bench.mjs bar and they're dev-only tooling, not shipped product code.
No blocking findings — mergeable as-is.
Generated by Claude Code
This was referenced Jul 8, 2026
Merged
NotASithLord
added a commit
that referenced
this pull request
Jul 12, 2026
Promotes [Unreleased] to [0.2.7] and bumps the version. The changelog was empty going into this cut, so wrote entries for the 5 user-facing PRs since v0.2.6 (Z.ai GLM as a BYOK provider #170, the fetch_url clean-markdown + spill-and-page pipeline #187, read_page mode:'content' #189, the experimental web-actor code-REPL surface + the actor-cascade stop fix #119, and the post-merge correctness fixes #194), skipping the 3 that are internal-only (a benchmark report doc, a PR-template chore). Grounded in each PR's own description, not invented; zero em dashes. Signed-off-by: Ariel Deschapell <arieldeschapell@Ariels-MacBook-Air.local> Co-authored-by: Ariel Deschapell <arieldeschapell@Ariels-MacBook-Air.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR is now
This PR set out to answer one question with numbers: does the web actor drive pages better by writing Playwright-style JS (
page.goto/click/fillin a sealed REPL) than by emitting discrete tool calls?The answer, measured on Online-Mind2Web with the benchmark's own judge, is no (full A/B): tool-call 26.7% vs code-REPL 20.7%, zero tasks won by the code arm that the tool arm didn't, more steps, more step-cap deaths.
webActorActionSurfacekeeps its'tools'default; the code surface ships as an opt-in experimental setting (Settings → Behavior).Why merge a negative result: the experiment's infrastructure is the durable value, and it all lives here.
What this PR ships
The code-REPL surface (experimental, default-off)
page_codetool + the sealed-workerpagebridge (page.goto/click/fill/snapshot/content), riding the generalizedmakeBridgeworker protocol. Everypage.*call is the SAME gated tool dispatch the tool-call actor uses (denylist/confirm/audit unchanged), pinned SW-side to the actor's owned tab — zero added authority.page+ pure compute ONLY (no egress / spawn / OPFS), enforced in-realm AND at the host relay.{page_code}; tools-surface unchanged.The Online-Mind2Web benchmark harness (the first public-benchmark rig for a browser-extension agent, as far as we know)
page/op+actor/opbroadcasts so a code-arm trajectory shows its REAL page actions to the judge (an earlier 33.3% reading was a recording artifact and is retracted in the A/B comment).Infrastructure & bug fixes found by running the benchmark
session/resetnow stops the abandoned session's live turn AND cascades to its in-flight actors (mirrorsagent/stop) — "New chat" mid-web-task used to leak zombie actors; in the harness this presented as "2 tasks, then a wall of timeouts". The deeper product bug (un-timed CDP dispatch is abort-immune) is tracked in Web-actor turn can hang forever on an un-timed CDP Runtime.evaluate (abort not honored mid-dispatch) #176.output_config.effort400 on non-adaptive models.Verification
Full local gauntlet green (bun suite, strict typecheck, eslint, tscheck floor, gen-drift, in-browser tiers) + model-free CDP diagnostics for the page bridge (
scripts/cdp/diagnose-page-code.mjs) provingpage.goto/snapshot/contentend-to-end. Merged with main v0.2.6 (actor rename reconciled).🤖 Generated with Claude Code