evals: add the v4_code code-mode exposure

[shriyatheunicorn]

The agent-facing half of the v4 code-mode surface: prepareLLMExposure
initializes a v4 Stagehand client via initStagehand (evals-v4-root) and
declares a code_handles LLMExposure — stagehand/page/z in the snippet
scope, surface-owned prompt instructions and run-tool copy, final-state
artifact capture (screenshot + URL), and cleanup that closes the browser
as well as the client (stagehand.close() alone leaves the browser
running). Nothing consumes the exposure yet; the adapter does in 6/6,
and core-tier support arrives separately in 3/6.

Part 2/6 of the #2473 port. Requires the LLMExposure contract (1/6).

---

Summary by cubic

Adds the stagehand_code code-mode tool and registers it in the core tool registry, wiring a v4 @browserbasehq/stagehand client with scoped handles, run-tool execution, artifact capture, and full teardown. Enables the v4 code-mode surface for STG-2671 with a fixed SDK model and local/Browserbase startup profiles.

• New Features
  • New StagehandCodeTool (id: stagehand_code) with session, navigation, evaluation, screenshot, viewport, wait, click, hover, scroll, type, press, tabs, and representation.
  • Targets: selector, coords, focused.
  • Agent mount via handles: stagehand, page, and z; scoped prompt instructions; run-tool description, code param description, and deny message. Snippets also get startUrl, task, and console in scope.
  • Captures screenshot, URL, and ARIA tree; cleanup closes both the Stagehand client and the browser.
  • Profiles: tool_launch_local and tool_create_browserbase; connection mode derived; model fixed to openai/gpt-4.1-mini.
  • Registered in listCoreTools/getCoreTool; tests assert stagehand_code is retrievable and that prompt guidance includes awaited locator actions.

^{Written for commit e579140. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: e579140

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

2 issues found across 1 file

Confidence score: 3/5

• In packages/evals/core/tools/v4_code.ts, artifact capture appears to resolve the active page inconsistently (popup/new-tab runs can be graded against the original page), which risks incorrect grading outcomes and flaky eval signals—resolve the active page once and use that same handle for both artifact collection and screenshot capture.
• In packages/evals/core/tools/v4_code.ts, prompt examples omit await on async locator actions, so agents copying the pattern may read or return state before click/fill/type completes, leading to race-condition-like misbehavior—update the examples to show awaited locator calls consistently.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/core/tools/v4_code.ts">

<violation number="1" location="packages/evals/core/tools/v4_code.ts:85">
P2: According to linked Linear issue STG-2671, grading must use harness-observed terminal artifacts; runs ending in a popup or selected new tab capture the original page instead. Resolve the active page once for both artifact fields, falling back to `v4.page`.</violation>

<violation number="2" location="packages/evals/core/tools/v4_code.ts:117">
P2: Locator actions shown in the prompt are missing `await`, causing agents that follow the example to return or inspect state before click/fill/type completes. Show awaited calls for each async action.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant HAR as External Harness
    participant PREP as prepareLLMExposure()
    participant INIT as initStagehand()
    participant SDK as Stagehand v4 SDK
    participant BROWSER as Local/Browserbase Browser
    participant AGENT as Agent (LLM)

    Note over HAR,AGENT: NEW: v4 Code-Mode Exposure Flow

    HAR->>PREP: prepareLLMExposure(plan, env, logger, profile?)
    alt profile not provided
        PREP->>PREP: Resolve profile based on env (LOCAL or BROWSERBASE)
    else invalid profile
        PREP->>PREP: Throw EvalsError
    end
    PREP->>INIT: initStagehand(modelName, environment)
    INIT-->>PREP: Return {stagehand, page}
    PREP->>PREP: Extract stagehand from v4

    Note over PREP: Build LLMExposure with code_handles kind

    PREP-->>HAR: Return LLMExposure
    HAR->>AGENT: Pass exposure to agent

    loop Agent uses run tool
        AGENT->>AGENT: Execute code against SDK (page.goto, stagehand.act, etc.)
        SDK->>BROWSER: Browser actions
        BROWSER-->>SDK: Results
        SDK-->>AGENT: Return values
    end

    opt Final state capture
        HAR->>HAR: captureFinalState()
        HAR->>HAR: Take screenshot (best-effort)
        HAR->>HAR: Get current URL (best-effort)
    end

    opt Cleanup
        HAR->>HAR: cleanup()
        HAR->>HAR: stagehand.close() (RPC client)
        HAR->>HAR: browser.close() (browser process)
    end

    Note over PREP: Surface model pinned to openai/gpt-4.1-mini

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Locator actions shown in the prompt are missing await, causing agents that follow the example to return or inspect state before click/fill/type completes. Show awaited calls for each async action.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/v4_code.ts, line 117:

<comment>Locator actions shown in the prompt are missing `await`, causing agents that follow the example to return or inspect state before click/fill/type completes. Show awaited calls for each async action.</comment>

<file context>
@@ -0,0 +1,124 @@
+    "Browser tool surface: v4_code (Stagehand v4 SDK).",
+    `Use the ${LLM_RUN_TOOL_NAME} tool for browser automation. It exposes an initialized Stagehand v4 client (stagehand), its active page, startUrl, and task object.`,
+    "AI methods live on the client: await stagehand.act('instruction'), await stagehand.observe('instruction'), await stagehand.extract('instruction', zodSchema) — a zod `z` is in scope for extract schemas (use single-word keys).",
+    "Deterministic methods live on the page: await page.goto(url), page.locator(selector).click()/fill()/type(), await page.url(), await page.title(), await page.screenshot().",
+    "Page accessors are async RPCs — always await them.",
+    "The first browser action should usually be: await page.goto(startUrl, { waitUntil: 'domcontentloaded' }).",
</file context>

[cubic-dev-ai]

P2: According to linked Linear issue STG-2671, grading must use harness-observed terminal artifacts; runs ending in a popup or selected new tab capture the original page instead. Resolve the active page once for both artifact fields, falling back to v4.page.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/core/tools/v4_code.ts, line 85:

<comment>According to linked Linear issue STG-2671, grading must use harness-observed terminal artifacts; runs ending in a popup or selected new tab capture the original page instead. Resolve the active page once for both artifact fields, falling back to `v4.page`.</comment>

<file context>
@@ -0,0 +1,124 @@
+      task: { instruction: plan.instruction, startUrl: plan.startUrl },
+      console,
+    },
+    captureFinalState: async () => {
+      const artifact: { screenshot?: Buffer; url?: string } = {};
+      try {
</file context>

Suggested change

	captureFinalState: async () => {
	const artifact: { screenshot?: Buffer; url?: string } = {};
	try {
	artifact.screenshot = await v4.page.screenshot();
	} catch {
	// best-effort only
	}
	try {
	artifact.url = await v4.page.url();
	} catch {
	// best-effort only
	}
	return artifact;
	},
	captureFinalState: async () => {
	let page = v4.page;
	try {
	page = (await stagehand.context.activePage()) ?? page;
	} catch {
	// best-effort only
	}
	const artifact: { screenshot?: Buffer; url?: string } = {};
	try {
	artifact.screenshot = await page.screenshot();
	} catch {
	// best-effort only
	}
	try {
	artifact.url = await page.url();
	} catch {
	// best-effort only
	}
	return artifact;
	},

[shriyatheunicorn]

no, bc we would like to omit the v4 prefix