evals: adaptive tool-surface harness (LLMExposure) + v4 code mode - #2473
Closed
shriyatheunicorn wants to merge 2 commits into
Closed
evals: adaptive tool-surface harness (LLMExposure) + v4 code mode#2473shriyatheunicorn wants to merge 2 commits into
shriyatheunicorn wants to merge 2 commits into
Conversation
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
cubic analysis
13 issues found across 7 files
Confidence score: 3/5
- The highest-risk gap is final-state grounding across
packages/evals/core/tools/playwright_code.tsandpackages/evals/framework/claudeCodeToolAdapter.ts: completed work in popup/new tabs and LLMJ runs can be graded from the wrong page or without terminal artifacts, which can mis-score longer-horizon tasks—track the terminal page before capture and invoke the harness callback so screenshot/URL are attached. packages/evals/core/tools/v4_code.tscurrently drops or rejects key interaction primitives (viewport, coordinate click/hover, and coordinate-preserving scroll), so v4 tasks can fail despite SDK support and produce false negatives—delegate to nativePage.setViewportSize/click/hover/scrolland advertiseviewport/coordstarget support.- Navigation and lifecycle reliability still has user-impacting timeout behavior: in
packages/evals/core/tools/v4_code.ts, reload/back/forward ignoretimeoutMs, and inpackages/evals/framework/claudeCodeToolAdapter.ts, init-failure cleanup can hang without the shared timeout, which can wedge eval rows—plumbtimeoutMsinto v4 navigation options and reuseEVAL_EXPOSURE_CLEANUP_TIMEOUT_MSin the catch cleanup path. - There’s moderate regression/debuggability risk from smaller correctness gaps:
packages/evals/core/tools/playwright_code.tslacks focused coverage for setup/cleanup/final-artifact paths,packages/evals/core/tools/cdp_code.tscan unsubscribe the wrong listener when callbacks are reused across methods, andpackages/evals/core/tools/v4_code.tsbypasses eval-scoped logging with process-global console—add targeted mocks, key CDP subscriptions by method+listener, and keep logger-backed console wiring.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/core/tools/registry.ts">
<violation number="1" location="packages/evals/core/tools/registry.ts:26">
P3: `v4_code` registration has no focused regression coverage, so a future list/switch mismatch can make the new surface unavailable without a test failure. Add assertions that `listCoreTools()` includes `v4_code` and `getCoreTool("v4_code")` constructs it.
(Based on your team's feedback about unit tests.)</violation>
</file>
<file name="packages/evals/core/tools/playwright_code.ts">
<violation number="1" location="packages/evals/core/tools/playwright_code.ts:605">
P3: This new browser exposure can regress setup, cleanup, or final-state capture without a focused test. Add mocked coverage for valid/invalid profiles, target/connection failure cleanup, and terminal artifacts.
(Based on your team's feedback about adding unit tests for new behavior.) .</violation>
<violation number="2" location="packages/evals/core/tools/playwright_code.ts:692">
P2: Tasks completed in a new or popup tab are graded against the initial page, so the terminal screenshot and URL can miss the completed state. Track the task's terminal page before capture, rather than always reading the original `page`.</violation>
</file>
<file name="packages/evals/core/tools/cdp_code.ts">
<violation number="1" location="packages/evals/core/tools/cdp_code.ts:1202">
P3: This new exposure lifecycle has no focused test for successful initialization or failure cleanup, so endpoint/attachment regressions will reach eval runs. Add coverage for target cleanup after connect/attach failure and final-state capture.
(Based on your team's feedback about unit tests.) .</violation>
<violation number="2" location="packages/evals/core/tools/cdp_code.ts:1383">
P3: `cdp.off(method, listener)` can unsubscribe a different event when one callback is registered for multiple methods, leaving the requested listener active. Key subscriptions by both method and listener, and remove only the requested registration.</violation>
</file>
<file name="packages/evals/framework/claudeCodeToolAdapter.ts">
<violation number="1" location="packages/evals/framework/claudeCodeToolAdapter.ts:429">
P2: LLMJ runs receive no harness-observed terminal screenshot or URL, so final-page grading remains ungrounded. According to linked Linear issue STG-2671, longer-horizon tasks are rubric-graded; invoke this callback after the agent completes and pass its artifact into the trajectory/verifier before cleanup.</violation>
<violation number="2" location="packages/evals/framework/claudeCodeToolAdapter.ts:456">
P2: Initialization failures can still wedge an eval row when a surface cleanup hangs, because this catch path bypasses the cleanup timeout. Reuse `EVAL_EXPOSURE_CLEANUP_TIMEOUT_MS` here before removing the temp directory.</violation>
</file>
<file name="packages/evals/core/tools/v4_code.ts">
<violation number="1" location="packages/evals/core/tools/v4_code.ts:124">
P2: Reload, back, and forward ignore callers' `timeoutMs`, so bounded navigation can exceed the task timeout. Map `timeoutMs` to v4's `timeout` option for each navigation method.</violation>
<violation number="2" location="packages/evals/core/tools/v4_code.ts:165">
P2: Viewport calls always fail despite the loaded v4 SDK exposing `Page.setViewportSize`. Delegate both CorePageHandle methods and advertise `viewport`; otherwise v4 core tasks cannot exercise viewport behavior.</violation>
<violation number="3" location="packages/evals/core/tools/v4_code.ts:227">
P2: Coordinate clicks fail even though the v4 SDK exposes native `Page.click(x, y)` and `Page.hover(x, y)`. Route numeric and `{ kind: "coords" }` targets to those methods and include `coords` in `supportedTargetKinds`.</violation>
<violation number="4" location="packages/evals/core/tools/v4_code.ts:239">
P2: Scrolling a pane at `(x, y)` instead scrolls the main document because the target coordinates are discarded. Use native `Page.scroll` to preserve the CorePageHandle wheel contract.</violation>
<violation number="5" location="packages/evals/core/tools/v4_code.ts:354">
P3: This new tool surface has no focused tests, leaving startup, page delegation, cleanup, and final-state behavior unguarded. Add mocked `initV4` tests for normal startup plus unsupported-profile and page-operation edge cases.
(Based on your team's feedback about unit tests for new behavior.) .</violation>
<violation number="6" location="packages/evals/core/tools/v4_code.ts:459">
P3: `console.log` from v4 snippets bypasses per-eval `claude_code` logging because this overrides the adapter's logger-backed console with the process-global one. Omit this field so run-tool diagnostics remain attached to the eval.</violation>
</file>
Linked issue analysis
Linked issue: STG-2671: v4 eval task suites (deterministic + non-deterministic) + LLM-judge harness
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Add a generic LLMExposure contract (kinds: code_handles, mcp_server, cli), LLM_RUN_TOOL_SERVER/NAME constants, and TerminalArtifact type. | The tool contract file was extended with LLMExposure, LLMRunToolSpec, TerminalArtifact, and the run-tool constants. |
| ✅ | Implement v4_code CoreTool (V4CodeTool) that starts a v4 SDK-backed session and register the tool in the core registry. | A new v4_code tool file implements V4CodeTool and init/start logic; registry was updated to list and return v4_code. |
| ✅ | Provide per-surface LLMExposure implementations for playwright_code and cdp_code that expose handles, promptInstructions, captureFinalState, and runTool specs. | Both playwright_code and cdp_code were extended with prepareLLMExposure functions that return code_handles exposures including prompt text, handles, runTool spec, captureFinalState, and cleanup. |
| ✅ | Refactor the Claude Code harness to mount code_handles exposures into a single run tool, use exposure.promptInstructions, and surface captureFinalState/cleanup. | claudeCodeToolAdapter was rewritten to accept LLMExposure, create a generic run MCP tool from exposure.runTool, mount exposure.handles as snippet scope, and wire exposure.captureFinalState and exposure.cleanup with bounded timeouts. |
| ✅ | Update tests and resolver logic to accept v4_code and SDK-owned startup profiles. | Tests and resolve/resolveStartupProfile logic were updated to include v4_code and to map its SDK-owned startup profiles accordingly. |
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| captureFinalState: async () => { | ||
| const artifact: { screenshot?: Buffer; url?: string } = {}; | ||
| try { | ||
| artifact.screenshot = await page.screenshot(); |
Contributor
There was a problem hiding this comment.
P2: Tasks completed in a new or popup tab are graded against the initial page, so the terminal screenshot and URL can miss the completed state. Track the task's terminal page before capture, rather than always reading the original page.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/playwright_code.ts, line 692:
<comment>Tasks completed in a new or popup tab are graded against the initial page, so the terminal screenshot and URL can miss the completed state. Track the task's terminal page before capture, rather than always reading the original `page`.</comment>
<file context>
@@ -590,3 +596,138 @@ export class PlaywrightCodeTool implements CoreTool {
+ captureFinalState: async () => {
+ const artifact: { screenshot?: Buffer; url?: string } = {};
+ try {
+ artifact.screenshot = await page.screenshot();
+ } catch {
+ // best-effort only
</file context>
| await connection?.close(); | ||
| } catch { | ||
| // best-effort only | ||
| await exposure.cleanup().catch((): undefined => undefined); |
Contributor
There was a problem hiding this comment.
P2: Initialization failures can still wedge an eval row when a surface cleanup hangs, because this catch path bypasses the cleanup timeout. Reuse EVAL_EXPOSURE_CLEANUP_TIMEOUT_MS here before removing the temp directory.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/claudeCodeToolAdapter.ts, line 456:
<comment>Initialization failures can still wedge an eval row when a surface cleanup hangs, because this catch path bypasses the cleanup timeout. Reuse `EVAL_EXPOSURE_CLEANUP_TIMEOUT_MS` here before removing the temp directory.</comment>
<file context>
@@ -545,37 +422,48 @@ async function prepareCdpCodeAdapter(
- await connection?.close();
- } catch {
- // best-effort only
+ await exposure.cleanup().catch((): undefined => undefined);
+ if (cwd) {
+ await fsp.rm(cwd, { recursive: true, force: true });
</file context>
Suggested change
| await exposure.cleanup().catch((): undefined => undefined); | |
| await withTimeout( | |
| exposure.cleanup(), | |
| readPositiveIntEnv("EVAL_EXPOSURE_CLEANUP_TIMEOUT_MS", 30_000), | |
| ).catch((): undefined => undefined); |
| }, | ||
| promptInstructions: buildCdpCodePromptInstructions(input.plan), | ||
| promptInstructions: exposure.promptInstructions, | ||
| ...(exposure.captureFinalState && { |
Contributor
There was a problem hiding this comment.
P2: LLMJ runs receive no harness-observed terminal screenshot or URL, so final-page grading remains ungrounded. According to linked Linear issue STG-2671, longer-horizon tasks are rubric-graded; invoke this callback after the agent completes and pass its artifact into the trajectory/verifier before cleanup.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/claudeCodeToolAdapter.ts, line 429:
<comment>LLMJ runs receive no harness-observed terminal screenshot or URL, so final-page grading remains ungrounded. According to linked Linear issue STG-2671, longer-horizon tasks are rubric-graded; invoke this callback after the agent completes and pass its artifact into the trajectory/verifier before cleanup.</comment>
<file context>
@@ -545,37 +422,48 @@ async function prepareCdpCodeAdapter(
},
- promptInstructions: buildCdpCodePromptInstructions(input.plan),
+ promptInstructions: exposure.promptInstructions,
+ ...(exposure.captureFinalState && {
+ captureFinalState: async (): Promise<TerminalArtifact> => {
+ try {
</file context>
| * initialized Playwright page/context/browser connected over a | ||
| * runner-provided CDP endpoint. | ||
| */ | ||
| export async function prepareLLMExposure( |
Contributor
There was a problem hiding this comment.
P3: This new browser exposure can regress setup, cleanup, or final-state capture without a focused test. Add mocked coverage for valid/invalid profiles, target/connection failure cleanup, and terminal artifacts.
(Based on your team's feedback about adding unit tests for new behavior.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/playwright_code.ts, line 605:
<comment>This new browser exposure can regress setup, cleanup, or final-state capture without a focused test. Add mocked coverage for valid/invalid profiles, target/connection failure cleanup, and terminal artifacts.
(Based on your team's feedback about adding unit tests for new behavior.) .</comment>
<file context>
@@ -590,3 +596,138 @@ export class PlaywrightCodeTool implements CoreTool {
+ * initialized Playwright page/context/browser connected over a
+ * runner-provided CDP endpoint.
+ */
+export async function prepareLLMExposure(
+ plan: ExternalHarnessTaskPlan,
+ env: "LOCAL" | "BROWSERBASE",
</file context>
| * (page-scoped send, browser-level send, and event helpers) attached to a | ||
| * runner-provided CDP endpoint. | ||
| */ | ||
| export async function prepareLLMExposure( |
Contributor
There was a problem hiding this comment.
P3: This new exposure lifecycle has no focused test for successful initialization or failure cleanup, so endpoint/attachment regressions will reach eval runs. Add coverage for target cleanup after connect/attach failure and final-state capture.
(Based on your team's feedback about unit tests.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/cdp_code.ts, line 1202:
<comment>This new exposure lifecycle has no focused test for successful initialization or failure cleanup, so endpoint/attachment regressions will reach eval runs. Add coverage for target cleanup after connect/attach failure and final-state capture.
(Based on your team's feedback about unit tests.) .</comment>
<file context>
@@ -1165,3 +1171,367 @@ export class CdpCodeTool implements CoreTool {
+ * (page-scoped send, browser-level send, and event helpers) attached to a
+ * runner-provided CDP endpoint.
+ */
+export async function prepareLLMExposure(
+ plan: ExternalHarnessTaskPlan,
+ env: "LOCAL" | "BROWSERBASE",
</file context>
| _method: string, | ||
| listener: (event: CdpEventMessage) => unknown | Promise<unknown>, | ||
| ): void => { | ||
| const unsubscribe = listenerUnsubscribes.get(listener); |
Contributor
There was a problem hiding this comment.
P3: cdp.off(method, listener) can unsubscribe a different event when one callback is registered for multiple methods, leaving the requested listener active. Key subscriptions by both method and listener, and remove only the requested registration.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/cdp_code.ts, line 1383:
<comment>`cdp.off(method, listener)` can unsubscribe a different event when one callback is registered for multiple methods, leaving the requested listener active. Key subscriptions by both method and listener, and remove only the requested registration.</comment>
<file context>
@@ -1165,3 +1171,367 @@ export class CdpCodeTool implements CoreTool {
+ _method: string,
+ listener: (event: CdpEventMessage) => unknown | Promise<unknown>,
+ ): void => {
+ const unsubscribe = listenerUnsubscribes.get(listener);
+ listenerUnsubscribes.delete(listener);
+ unsubscribe?.();
</file context>
| } | ||
| } | ||
|
|
||
| export class V4CodeTool implements CoreTool { |
Contributor
There was a problem hiding this comment.
P3: This new tool surface has no focused tests, leaving startup, page delegation, cleanup, and final-state behavior unguarded. Add mocked initV4 tests for normal startup plus unsupported-profile and page-operation edge cases.
(Based on your team's feedback about unit tests for new behavior.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/v4_code.ts, line 354:
<comment>This new tool surface has no focused tests, leaving startup, page delegation, cleanup, and final-state behavior unguarded. Add mocked `initV4` tests for normal startup plus unsupported-profile and page-operation edge cases.
(Based on your team's feedback about unit tests for new behavior.) .</comment>
<file context>
@@ -0,0 +1,498 @@
+ }
+}
+
+export class V4CodeTool implements CoreTool {
+ readonly id = "v4_code";
+ readonly surface = "code";
</file context>
| "JavaScript function body to execute. stagehand/page/startUrl/task/z are already in scope.", | ||
| denyMessage: `Use Bash for inspection and ${LLM_RUN_TOOL_NAME} for browser automation.`, | ||
| task: { instruction: plan.instruction, startUrl: plan.startUrl }, | ||
| console, |
Contributor
There was a problem hiding this comment.
P3: console.log from v4 snippets bypasses per-eval claude_code logging because this overrides the adapter's logger-backed console with the process-global one. Omit this field so run-tool diagnostics remain attached to the eval.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/v4_code.ts, line 459:
<comment>`console.log` from v4 snippets bypasses per-eval `claude_code` logging because this overrides the adapter's logger-backed console with the process-global one. Omit this field so run-tool diagnostics remain attached to the eval.</comment>
<file context>
@@ -0,0 +1,498 @@
+ "JavaScript function body to execute. stagehand/page/startUrl/task/z are already in scope.",
+ denyMessage: `Use Bash for inspection and ${LLM_RUN_TOOL_NAME} for browser automation.`,
+ task: { instruction: plan.instruction, startUrl: plan.startUrl },
+ console,
+ },
+ ...(v4.sessionUrl && { metadata: { sessionUrl: v4.sessionUrl } }),
</file context>
Contributor
There was a problem hiding this comment.
cubic analysis
1 issue found across 7 files
Confidence score: 3/5
- In
packages/evals/core/tools/cdp_code.ts, using a genericnew Error()in the custom agent path means failures bubbling throughcdp.waitForEvent/cdp.oncemay skip consistent sanitization and typed handling, which can lead to leaked raw error details or incorrect downstream error behavior. Replace it with the project’s typed/sanitized error class (or map at the boundary) so propagated messages are normalized before reaching agent code.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/core/tools/cdp_code.ts">
<violation number="1" location="packages/evals/core/tools/cdp_code.ts:1514">
P2: Custom agent: **Exception and error message sanitization**
Generic `new Error()` is used here instead of a typed error class. This rejection propagates through `cdp.waitForEvent` and `cdp.once` to agent code, and when caught in `executeCodeExposureRunTool` the message is serialized directly into the tool result returned to the agent — crossing a user-facing boundary. Switch to `EvalsError`, which is already imported and used elsewhere in this file, to comply with the typed-error-class requirement.</violation>
</file>
Linked issue analysis
Linked issue: STG-2671: v4 eval task suites (deterministic + non-deterministic) + LLM-judge harness
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Add a generic LLMExposure contract (kinds: code_handles, mcp_server, cli), LLM_RUN_TOOL_SERVER/NAME constants, and TerminalArtifact type. | The tool contract file was extended with LLMExposure, LLMRunToolSpec, TerminalArtifact, and the run-tool constants. |
| ✅ | Implement v4_code CoreTool (V4CodeTool) that starts a v4 SDK-backed session and register the tool in the core registry. | A new v4_code tool file implements V4CodeTool and init/start logic; registry was updated to list and return v4_code. |
| ✅ | Provide per-surface LLMExposure implementations for playwright_code and cdp_code that expose handles, promptInstructions, captureFinalState, and runTool specs. | Both playwright_code and cdp_code were extended with prepareLLMExposure functions that return code_handles exposures including prompt text, handles, runTool spec, captureFinalState, and cleanup. |
| ✅ | Refactor the Claude Code harness to mount code_handles exposures into a single run tool, use exposure.promptInstructions, and surface captureFinalState/cleanup. | claudeCodeToolAdapter was rewritten to accept LLMExposure, create a generic run MCP tool from exposure.runTool, mount exposure.handles as snippet scope, and wire exposure.captureFinalState and exposure.cleanup with bounded timeouts. |
| ✅ | Update tests and resolver logic to accept v4_code and SDK-owned startup profiles. | Tests and resolve/resolveStartupProfile logic were updated to include v4_code and to map its SDK-owned startup profiles accordingly. |
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| }); | ||
| timeout = setTimeout(() => { | ||
| cleanup(); | ||
| reject(new Error(`Timed out waiting for CDP event "${method}"`)); |
Contributor
There was a problem hiding this comment.
P2: Custom agent: Exception and error message sanitization
Generic new Error() is used here instead of a typed error class. This rejection propagates through cdp.waitForEvent and cdp.once to agent code, and when caught in executeCodeExposureRunTool the message is serialized directly into the tool result returned to the agent — crossing a user-facing boundary. Switch to EvalsError, which is already imported and used elsewhere in this file, to comply with the typed-error-class requirement.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/cdp_code.ts, line 1514:
<comment>Generic `new Error()` is used here instead of a typed error class. This rejection propagates through `cdp.waitForEvent` and `cdp.once` to agent code, and when caught in `executeCodeExposureRunTool` the message is serialized directly into the tool result returned to the agent — crossing a user-facing boundary. Switch to `EvalsError`, which is already imported and used elsewhere in this file, to comply with the typed-error-class requirement.</comment>
<file context>
@@ -1165,3 +1171,367 @@ export class CdpCodeTool implements CoreTool {
+ });
+ timeout = setTimeout(() => {
+ cleanup();
+ reject(new Error(`Timed out waiting for CDP event "${method}"`));
+ }, timeoutMs);
+ });
</file context>
Suggested change
| reject(new Error(`Timed out waiting for CDP event "${method}"`)); | |
| reject(new EvalsError(`Timed out waiting for CDP event "${method}"`)); |
Contributor
There was a problem hiding this comment.
cubic analysis
No issues found across 7 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Linked issue analysis
Linked issue: STG-2671: v4 eval task suites (deterministic + non-deterministic) + LLM-judge harness
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | Add a generic LLMExposure contract (kinds: code_handles, mcp_server, cli), LLM_RUN_TOOL_SERVER/NAME constants, and TerminalArtifact type. | The tool contract file was extended with LLMExposure, LLMRunToolSpec, TerminalArtifact, and the run-tool constants. |
| ✅ | Implement v4_code CoreTool (V4CodeTool) that starts a v4 SDK-backed session and register the tool in the core registry. | A new v4_code tool file implements V4CodeTool and init/start logic; registry was updated to list and return v4_code. |
| ✅ | Provide per-surface LLMExposure implementations for playwright_code and cdp_code that expose handles, promptInstructions, captureFinalState, and runTool specs. | Both playwright_code and cdp_code were extended with prepareLLMExposure functions that return code_handles exposures including prompt text, handles, runTool spec, captureFinalState, and cleanup. |
| ✅ | Refactor the Claude Code harness to mount code_handles exposures into a single run tool, use exposure.promptInstructions, and surface captureFinalState/cleanup. | claudeCodeToolAdapter was rewritten to accept LLMExposure, create a generic run MCP tool from exposure.runTool, mount exposure.handles as snippet scope, and wire exposure.captureFinalState and exposure.cleanup with bounded timeouts. |
| ✅ | Update tests and resolver logic to accept v4_code and SDK-owned startup profiles. | Tests and resolve/resolveStartupProfile logic were updated to include v4_code and to map its SDK-owned startup profiles accordingly. |
Introduces a harness-agnostic exposure contract so LLM coding harnesses (claude_code today, codex next) drive any browser tool surface through one seam. Each surface implements a single prepareLLMExposure(); the harness keeps three generic mount points (code handles / MCP server / CLI) and zero per-surface knowledge. - core/contracts/tool.ts: LLMExposure, LLMRunToolSpec, TerminalArtifact, and the shared run-tool naming constants - core/tools/v4_code.ts: v4 code mode - the agent writes real Stagehand v4 SDK code (stagehand.act/extract/observe, locators) against the extension stack; CoreTool for the deterministic tier plus its exposure - playwright_code/cdp_code: per-surface exposures moved in from the claude_code adapter; each also captures harness-observed terminal state (screenshot + URL) for artifact-grounded grading - framework/claudeCodeToolAdapter.ts: one generic exposure consumer replaces all per-surface setup (net -638 lines here); bounded exposure cleanup so a hung surface close cannot wedge a row This makes v4 vs Playwright vs CDP benchable under the same harness, model, and grader - score deltas attribute to the tool surface alone. Verified live on WebVoyager rows for both v4_code and playwright_code arms; model-visible prompt strings byte-identical through the refactor. Part of STG-2671
shriyatheunicorn
force-pushed
the
stg-2671-v4-harness-spike
branch
from
August 4, 2026 08:48
9bf27f8 to
d040f37
Compare
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
This was referenced Aug 5, 2026
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Replaces the per-surface logic inside the adapter with one generic exposure consumer: any surface declaring an exposure gets its handles mounted as the single local-MCP run tool, with the surface's own prompt instructions. Net deletion — the bespoke playwright_code/cdp_code blocks move behind the contract, and adding a surface no longer means editing the adapter. Part 6/6 of the #2473 port. Requires parts 1-5. Verified across the stack: tsc and oxlint per part, evals vitest (404 passing at this tip), live core runs on Browserbase for --tool v4_code, playwright_code, and cdp_code (3/3 each, prompt exit), one long-horizon row (webvoyager x claude_code x v4_code, 3/3), and the paired webvoyager acceptance run (v4_code 2/3 vs playwright_code 3/3, both cells traced to Braintrust).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
A uniform declaration of what a tool surface offers a coding agent: code_handles (in-scope objects the agent writes code against, mounted by the harness as a single local-MCP run tool), mcp_server, or cli — plus the LLM_RUN_TOOL_SERVER/LLM_RUN_TOOL_NAME bindings the mount uses. Types and constants only; no surface or adapter changes. Part 1/4 of the #2473 port onto the current SDK generation.
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
The surface now owns its exposure: prompt instructions (carried over verbatim from the per-surface block inside the claude_code tool adapter), a code_handles declaration, and captureFinalState() returning the terminal screenshot and URL. Behavior is unchanged until the adapter consumes exposures generically in 6/6. Part 4/6 of the #2473 port. Requires the LLMExposure contract (1/6).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Same shape as 4/6 for the CDP surface: prompt instructions carried over verbatim, code_handles declaration, captureFinalState() for the terminal screenshot and URL. Behavior is unchanged until the adapter consumes exposures generically in 6/6. Part 5/6 of the #2473 port. Requires the LLMExposure contract (1/6).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Replaces the per-surface logic inside the adapter with one generic exposure consumer: any surface declaring an exposure gets its handles mounted as the single local-MCP run tool, with the surface's own prompt instructions. Net deletion — the bespoke playwright_code/cdp_code blocks move behind the contract, and adding a surface no longer means editing the adapter. Part 6/6 of the #2473 port. Requires parts 1-5. Verified across the stack: tsc and oxlint per part, evals vitest (404 passing at this tip), live core runs on Browserbase for --tool v4_code, playwright_code, and cdp_code (3/3 each, prompt exit), one long-horizon row (webvoyager x claude_code x v4_code, 3/3), and the paired webvoyager acceptance run (v4_code 2/3 vs playwright_code 3/3, both cells traced to Braintrust).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Replaces the per-surface logic inside the adapter with one generic exposure consumer: any surface declaring an exposure gets its handles mounted as the single local-MCP run tool, with the surface's own prompt instructions. Net deletion — the bespoke playwright_code/cdp_code blocks move behind the contract, and adding a surface no longer means editing the adapter. Part 6/6 of the #2473 port. Requires parts 1-5. Verified across the stack: tsc and oxlint per part, evals vitest (404 passing at this tip), live core runs on Browserbase for --tool v4_code, playwright_code, and cdp_code (3/3 each, prompt exit), one long-horizon row (webvoyager x claude_code x v4_code, 3/3), and the paired webvoyager acceptance run (v4_code 2/3 vs playwright_code 3/3, both cells traced to Braintrust).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Replaces the per-surface logic inside the adapter with one generic exposure consumer: any surface declaring an exposure gets its handles mounted as the single local-MCP run tool, with the surface's own prompt instructions. Net deletion — the bespoke playwright_code/cdp_code blocks move behind the contract, and adding a surface no longer means editing the adapter. Part 6/6 of the #2473 port. Requires parts 1-5. Verified across the stack: tsc and oxlint per part, evals vitest (404 passing at this tip), live core runs on Browserbase for --tool v4_code, playwright_code, and cdp_code (3/3 each, prompt exit), one long-horizon row (webvoyager x claude_code x v4_code, 3/3), and the paired webvoyager acceptance run (v4_code 2/3 vs playwright_code 3/3, both cells traced to Braintrust).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 5, 2026
Replaces the per-surface logic inside the adapter with one generic exposure consumer: any surface declaring an exposure gets its handles mounted as the single local-MCP run tool, with the surface's own prompt instructions. Net deletion — the bespoke playwright_code/cdp_code blocks move behind the contract, and adding a surface no longer means editing the adapter. Part 6/6 of the #2473 port. Requires parts 1-5. Verified across the stack: tsc and oxlint per part, evals vitest (404 passing at this tip), live core runs on Browserbase for --tool v4_code, playwright_code, and cdp_code (3/3 each, prompt exit), one long-horizon row (webvoyager x claude_code x v4_code, 3/3), and the paired webvoyager acceptance run (v4_code 2/3 vs playwright_code 3/3, both cells traced to Braintrust).
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 6, 2026
A uniform declaration of what a tool surface offers a coding agent: code_handles (in-scope objects the agent writes code against, mounted by the harness as a single local-MCP run tool), mcp_server, or cli — plus the LLM_RUN_TOOL_SERVER/LLM_RUN_TOOL_NAME bindings the mount uses. Types and constants only; no surface or adapter changes. Part 1/4 of the #2473 port onto the current SDK generation. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Updates the tool-surface contract to include agent delivery (`AgentMount`) and final-state evidence capture, keeping native `CoreTool.surface` independent from how it’s delivered to the agent. Types + tests only; supports STG-2671. - **New Features** - Types: `AgentMount` (`via: handles | mcp | cli` with `promptInstructions`), `AgentRunToolSpec`; `ToolStartResult` gains optional `agentMount` and `captureEvidence(): Promise<ProbeEvidence>`. - Enums: `ToolSurface` adds `stagehand_code`; `CoreTool.family` adds `stagehand`. - Constants: `AGENT_RUN_TOOL_SERVER` ("stagehand_browser"), `AGENT_RUN_TOOL_NAME` (`mcp__stagehand_browser__run`), `AGENT_RUN_TOOL_RESERVED_HANDLES` (`startUrl`, `task`, `console`). - Contract semantics: delivery is independent of native surface; CLI env merges over harness env; reserved harness bindings; capture-before-cleanup ordering. - Tests: `tool-contract.test.ts` validates delivery independence, handle mounts, and evidence capture. <sup>Written for commit d3ba958. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2590?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Miguel <36487034+miguelg719@users.noreply.github.com>
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a){kind=link}
shriyatheunicorn
added a commit
that referenced
this pull request
Aug 8, 2026
A uniform declaration of what a tool surface offers a coding agent: code_handles (in-scope objects the agent writes code against, mounted by the harness as a single local-MCP run tool), mcp_server, or cli — plus the LLM_RUN_TOOL_SERVER/LLM_RUN_TOOL_NAME bindings the mount uses. Types and constants only; no surface or adapter changes. Part 1/4 of the #2473 port onto the current SDK generation. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Updates the tool-surface contract to include agent delivery (`AgentMount`) and final-state evidence capture, keeping native `CoreTool.surface` independent from how it’s delivered to the agent. Types + tests only; supports STG-2671. - **New Features** - Types: `AgentMount` (`via: handles | mcp | cli` with `promptInstructions`), `AgentRunToolSpec`; `ToolStartResult` gains optional `agentMount` and `captureEvidence(): Promise<ProbeEvidence>`. - Enums: `ToolSurface` adds `stagehand_code`; `CoreTool.family` adds `stagehand`. - Constants: `AGENT_RUN_TOOL_SERVER` ("stagehand_browser"), `AGENT_RUN_TOOL_NAME` (`mcp__stagehand_browser__run`), `AGENT_RUN_TOOL_RESERVED_HANDLES` (`startUrl`, `task`, `console`). - Contract semantics: delivery is independent of native surface; CLI env merges over harness env; reserved harness bindings; capture-before-cleanup ordering. - Tests: `tool-contract.test.ts` validates delivery independence, handle mounts, and evidence capture. <sup>Written for commit d3ba958. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2590?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Miguel <36487034+miguelg719@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nondeterministic-suite stack 1/4, restacked onto the v4-spike migration (#2472). Supersedes #2388 (same commit, replayed and conflict-resolved on the new base).
Summary by cubic
Adds a harness-agnostic exposure seam and
v4_codeso the coding harness can drivev4_code,playwright_code, andcdp_codethrough one run tool. Supports STG-2671 by enabling consistent non-deterministic runs and artifact-grounded grading across tool surfaces.New Features
LLMExposurecontract (code_handles/mcp_server/cli),LLM_RUN_TOOL_SERVER,LLM_RUN_TOOL_NAME, andTerminalArtifactfor screenshot+URL capture.v4_codeCoreTool and exposure: runs against the Stagehand v4 SDK (stagehand/page/zin scope) with prompt and run-tool bindings; registered in the core tool registry; uses SDK-owned startup profiles (tool_launch_local/tool_create_browserbase).playwright_codeandcdp_code, each now providing model-facing prompt instructions, run-tool specs, andcaptureFinalState();cdp_codeincludes CDP event helpers (on/once/waitForEvent) with timeout handling.Refactors
claudeCodeToolAdapterwith one generic code-exposure consumer that mounts a single run tool and delegates deny messages/prompt text to the surface; retains prior prompt copy.browse_clibuild-check message.v4_codeand SDK-owned startup profiles.Written for commit b0f8bf6. Summary will update on new commits.