evals: run act/extract/observe bench tasks on a v4 Stagehand client

[shriyatheunicorn]

The deterministic suite is ported to the v4 SDK in place (#2494), but the
bench harness only ever built a v3 instance, so every ported task failed
its defineBenchV4Task guard before its own code ran.

• initStagehand: v4 client factory beside initV3, mirroring its
  environment/key resolution. selfHeal is on (the server defaults it off,
  which would let the heal_* benchmarks pass while measuring nothing) and
  debug SDK log lines are dropped rather than bridged (bridging produces
  ~18MB Braintrust payloads the API rejects). If init fails after launch,
  the browser is closed rather than leaked.
• benchHarness: act/extract/observe dispatch to the v4 client by task
  category — the directory a task lives in — so tasks carry no marker and
  no flag exists. Agent tasks and --api fail loudly on the v4 path.
  Cleanup closes the browser as well as the client: stagehand.close()
  tears down the RPC client but leaves the browser running, which leaked
  one Chrome process (LOCAL) or live Browserbase session per task and
  kept the CLI from exiting.
• benchRunner: forwards { stagehand, page } from the harness context into
  the task context.
• benchRunner.test: the legacy-task fixture moves from category act to
  combination — act now correctly routes to the v4 client, and legacy v3
  tasks only exist outside the deterministic categories.

Merge after #2494: category dispatch assumes the a/e/o tasks are the v4
ports; with the v3 tasks still in place it hands them a v4 context.

Verified on Browserbase (3-model matrix, -c 10): act 35/40, extract
24/25, observe 11/12 union — at or above the v3 baseline per category.

---

Summary by cubic

Runs act/extract/observe benchmark tasks on the v4 @browserbasehq/stagehand client. Restores Browserbase session/debug URLs, makes the verifier model selectable, and hardens init/cleanup to prevent leaked browsers and lost errors.

• New Features

  • Added initStagehand to launch local or Browserbase with self‑heal on, resolve provider API keys, pass BROWSERBASE_PROJECT_ID/BB_PROJECT_ID, forward SDK logs (drop debug), and create the Browserbase session first, attach via browserbase.connect, and provide sessionUrl/debugUrl with explicit session release on cleanup and failures.
  • Dispatches act/extract/observe by category to v4; tasks receive { stagehand, page }. Blocks agent tasks/modes and --api on the v4 path. Context is now discriminated (sdk: "v4" | "v3"); legacy tasks on a v4 context throw.
  • Verifier model selectable via EVAL_VERIFIER_MODEL; resolves provider key and applies it to the verifier backend, supporting keyless providers like bedrock and ollama.
  • Introduced @browserbasehq/stagehand-integrations with a persistent code‑mode executor, MCP stdio server, env config, and bundled SKILL.md/REFERENCE.md.

• Bug Fixes

  • Close the v4 client and launched browser on cleanup and on init/page‑acquisition failures; use the browser context to obtain activePage.
  • Preserve benchmark error messages (error.message); parse decorated Claude Code results by extracting the first JSON object after EVAL_RESULT.

^{Written for commit 555b783. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 555b783

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 4 files

Architecture diagram

sequenceDiagram
    participant CLI as CLI Runner
    participant Harness as benchHarness
    participant Runner as benchRunner
    participant Task as Eval Task (act/extract/observe)
    participant Init as initStagehand
    participant SDK as Stagehand v4 SDK
    participant Browser as Browser (local/Browserbase)
    participant Braintrust as Braintrust API

    Note over CLI,Braintrust: Deterministic Benchmark Flow (act/extract/observe categories)

    CLI->>Harness: stagehandHarness(task)

    Harness->>Harness: Check task.primaryCategory

    alt Category is act/extract/observe
        Harness->>Harness: Block agent tasks & --api
        Harness->>Init: initStagehand(modelName, environment)
        
        Init->>Init: Resolve provider API key from env
        alt environment == "BROWSERBASE"
            Init->>Browser: browserbase.launch({apiKey})
        else LOCAL
            Init->>Browser: localBrowser.launch({headless: false})
        end
        Browser-->>Init: browser handle

        Init->>SDK: Stagehand.create({browser, selfHeal: true, model})
        Note over Init,SDK: selfHeal enabled (server default is off)

        SDK->>SDK: onLog filter: drop debug lines
        Note over SDK: Prevents ~18MB Braintrust payload rejection

        alt Creation fails
            SDK-->>Init: Error
            Init->>Browser: browser.close()
        else Success
            SDK-->>Init: stagehand instance
            Init->>SDK: stagehand.context.activePage()
            SDK-->>Init: page handle
            Init-->>Harness: {stagehand, page}
        end

        Harness-->>CLI: ctx with {stagehand, v4Page}

        CLI->>Runner: executeBenchTask(ctx)
        Runner->>Runner: Forward stagehand from harnessCtx
        Runner->>Runner: Forward v4Page as page for task
        Runner->>Task: Execute task(stagehand, page)
        Task->>SDK: act/extract/observe calls
        SDK-->>Task: Results

        Task-->>Runner: Complete
        Runner-->>CLI: Done

        CLI->>Harness: cleanup()
        Harness->>SDK: stagehand.close()
        Note over Harness,SDK: Tears down RPC client only

        Harness->>Browser: stagehand.browser.close()
        Note over Harness,Browser: NEW: Prevents leaked Chrome/Browserbase session

    else Other category (e.g., combination, agent)
        Note over Harness: Falls through to existing v3 flow
    end

    Note over CLI,Braintrust: Failure Path: init failure after launch
    alt Init throws after browser launch
        Init->>Browser: browser.close() (in catch block)
        Init-->>Harness: Throw error
        Harness-->>CLI: Error
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Confidence score: 4/5

• In packages/evals/initStagehand.ts, the new page-acquisition failure cleanup and Browserbase project-id pass-through paths are untested, so a regression could silently route sessions to the wrong Browserbase project or leave failure handling inconsistent — add targeted unit tests for both the project-id branch and page-acquisition error cleanup behavior.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/initStagehand.ts">

<violation number="1" location="packages/evals/initStagehand.ts:124">
P3: The new page-acquisition failure cleanup and the Browserbase project-id pass-through are behavior worth locking down with unit tests: a regression in the project-id branch silently lands sessions in the wrong Browserbase project, and one in the page-failure branch re-introduces exactly the leaked-browser/process-per-task bug this change fixes. There's currently no unit test covering initStagehand's launch, cleanup, or activePage failure paths. Consider adding a couple of focused tests (e.g. that projectId is forwarded to browserbase.launch when set, and that a failing activePage() still closes both stagehand and browser) to prevent regressions.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new page-acquisition failure cleanup and the Browserbase project-id pass-through are behavior worth locking down with unit tests: a regression in the project-id branch silently lands sessions in the wrong Browserbase project, and one in the page-failure branch re-introduces exactly the leaked-browser/process-per-task bug this change fixes. There's currently no unit test covering initStagehand's launch, cleanup, or activePage failure paths. Consider adding a couple of focused tests (e.g. that projectId is forwarded to browserbase.launch when set, and that a failing activePage() still closes both stagehand and browser) to prevent regressions.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/initStagehand.ts, line 124:

<comment>The new page-acquisition failure cleanup and the Browserbase project-id pass-through are behavior worth locking down with unit tests: a regression in the project-id branch silently lands sessions in the wrong Browserbase project, and one in the page-failure branch re-introduces exactly the leaked-browser/process-per-task bug this change fixes. There's currently no unit test covering initStagehand's launch, cleanup, or activePage failure paths. Consider adding a couple of focused tests (e.g. that projectId is forwarded to browserbase.launch when set, and that a failing activePage() still closes both stagehand and browser) to prevent regressions.</comment>

<file context>
@@ -109,10 +118,19 @@ export async function initStagehand({
+  // Page acquisition failures need the same cleanup as create failures: the
+  // client is up and the browser is running, and stagehand.close() alone
+  // tears down the RPC client without closing the browser.
+  let page: Page | null;
+  try {
+    page = await stagehand.context.activePage();
</file context>

[shrey150]

Not adding new initializer unit coverage in this PR. The cleanup and Browserbase configuration paths were exercised through the real eval lifecycle, including successful browser cleanup; broader test expansion is deferred while the eval stack is still changing.

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Confidence score: 5/5

• In packages/evals/framework/verifierAdapter.ts, the new EVAL_VERIFIER_MODEL override logic is only validated for the happy path, so an untested error branch and unasserted default could allow misconfigured env values to slip through and cause confusing verifier behavior at runtime — add targeted tests for the missing error path and default-model fallback.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/verifierAdapter.ts">

<violation number="1" location="packages/evals/framework/verifierAdapter.ts:38">
P3: The new EVAL_VERIFIER_MODEL override path has one untested edge case plus an unasserted default. The only new test covers the happy path (model + key present). The error branch introduced here — `EVAL_VERIFIER_MODEL` set but `loadApiKeyFromEnv` returning no key, causing the throw — is never exercised, and none of the existing tests assert the constructor options on the env-unset default branch (they pass through `createVerifierEvaluator` and would not catch a regression where the default path wrongly starts requiring an API key). Since this refactor routes all verifier construction through the new function, a quick test asserting the default `{ backend: "verifier" }` options and one asserting the missing-key error would lock in both behaviors and cover the error path end-to-end (e.g. via `verifierError` on the result).</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new EVAL_VERIFIER_MODEL override path has one untested edge case plus an unasserted default. The only new test covers the happy path (model + key present). The error branch introduced here — EVAL_VERIFIER_MODEL set but loadApiKeyFromEnv returning no key, causing the throw — is never exercised, and none of the existing tests assert the constructor options on the env-unset default branch (they pass through createVerifierEvaluator and would not catch a regression where the default path wrongly starts requiring an API key). Since this refactor routes all verifier construction through the new function, a quick test asserting the default { backend: "verifier" } options and one asserting the missing-key error would lock in both behaviors and cover the error path end-to-end (e.g. via verifierError on the result).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/verifierAdapter.ts, line 38:

<comment>The new EVAL_VERIFIER_MODEL override path has one untested edge case plus an unasserted default. The only new test covers the happy path (model + key present). The error branch introduced here — `EVAL_VERIFIER_MODEL` set but `loadApiKeyFromEnv` returning no key, causing the throw — is never exercised, and none of the existing tests assert the constructor options on the env-unset default branch (they pass through `createVerifierEvaluator` and would not catch a regression where the default path wrongly starts requiring an API key). Since this refactor routes all verifier construction through the new function, a quick test asserting the default `{ backend: "verifier" }` options and one asserting the missing-key error would lock in both behaviors and cover the error path end-to-end (e.g. via `verifierError` on the result).</comment>

<file context>
@@ -18,6 +20,34 @@ import { RubricCache } from "./rubricCache.js";
+
+  const provider = modelName.includes("/") ? modelName.slice(0, modelName.indexOf("/")) : undefined;
+  const apiKey = loadApiKeyFromEnv(provider, () => {});
+  if (!apiKey) {
+    throw new Error(
+      `${VERIFIER_MODEL_ENV} is set to "${modelName}", but no API key was found for provider "${provider ?? "unknown"}".`,
</file context>

[shrey150]

Not expanding verifier unit coverage in this PR. The existing focused verifier/trajectory suite passes, and the configurable verifier path was also exercised in the real benchmark harness.

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Confidence score: 4/5

• In packages/evals/framework/verifierAdapter.ts, the new keyless-provider path (bedrock/ollama skipping missing-key errors and omitting modelClientOptions) is untested, so regressions could silently break provider initialization or error handling in keyless setups—add targeted unit tests for both key-present and key-absent flows in gradeExternalTrajectory coverage.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/verifierAdapter.ts">

<violation number="1" location="packages/evals/framework/verifierAdapter.ts:48">
P3: The new keyless-provider behavior (skip the missing-API-key error for bedrock/ollama and omit `modelClientOptions` when no key is present) has no unit test coverage. The existing `gradeExternalTrajectory` test only exercises the keyed groq path and even asserts `modelClientOptions: { apiKey }` is always set, so it would not catch a regression in the keyless branch. Consider adding focused tests for: (1) a bedrock/ollama verifier model with no API key resolving without throwing and without `modelClientOptions`, and (2) a non-keyless provider with no API key still throwing the configured error.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new keyless-provider behavior (skip the missing-API-key error for bedrock/ollama and omit modelClientOptions when no key is present) has no unit test coverage. The existing gradeExternalTrajectory test only exercises the keyed groq path and even asserts modelClientOptions: { apiKey } is always set, so it would not catch a regression in the keyless branch. Consider adding focused tests for: (1) a bedrock/ollama verifier model with no API key resolving without throwing and without modelClientOptions, and (2) a non-keyless provider with no API key still throwing the configured error.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/verifierAdapter.ts, line 48:

<comment>The new keyless-provider behavior (skip the missing-API-key error for bedrock/ollama and omit `modelClientOptions` when no key is present) has no unit test coverage. The existing `gradeExternalTrajectory` test only exercises the keyed groq path and even asserts `modelClientOptions: { apiKey }` is always set, so it would not catch a regression in the keyless branch. Consider adding focused tests for: (1) a bedrock/ollama verifier model with no API key resolving without throwing and without `modelClientOptions`, and (2) a non-keyless provider with no API key still throwing the configured error.</comment>

<file context>
@@ -44,7 +45,7 @@ export function createVerifierEvaluator(v3: V3): V3Evaluator {
     backend: "verifier",
     modelName: modelName as AvailableModel,
-    modelClientOptions: { apiKey },
+    ...(apiKey ? { modelClientOptions: { apiKey } } : {}),
   });
 }
</file context>

[shrey150]

Not adding new verifier unit cases in this PR while the eval stack is still changing. The keyless branch omits modelClientOptions by construction, the existing focused verifier/trajectory suite passes, and keyed verifier selection was exercised through the real benchmark harness.

[miguelg719]

@shriyatheunicorn shouldn't this pr target #2494 ?

[shriyatheunicorn]

yes indeed; there was some code changing and rebasing w/o my knowing last night - im retargeting it rn

[shriyatheunicorn]

retargeted