evals: restore session URLs on the v4 path and split the harness context

[miguelg719]

Summary

Review follow-up for #2570, implemented. Three fixes and one test cleanup; only functional change is the session-URL restoration.

• Session/debug URLs on the v4 path (the one functional gap): initStagehand now creates the Browserbase session first via the shared launchRunnerProvidedBrowserbaseChrome() creator and attaches with browserbase.connect({ apiKey, sessionId }), instead of browserbase.launch (which hides the session id behind the opaque StagehandBrowser handle, Define TypeScript Stagehand browser contracts #2517). sessionUrl/debugUrl flow into every TaskResult and the Braintrust replay click-through again, and the session is explicitly released (REQUEST_RELEASE) on cleanup and on every init failure path — browser.close() on a connected handle only disconnects.
• BenchHarnessContext → discriminated union (sdk: "v4" | "v3"): the runner narrows on the discriminant instead of probing five optional fields; the v4Page ?? page fallback is gone, and a legacy task reaching a v4 context is an explicit EvalsError instead of v3: undefined.
• Dead systemPrompt parameter dropped from initStagehand — nothing passes it, and the v3 equivalent is agent-only.
• Test fixtures off the deleted combination category → agent, the only non-deterministic category left after evals: make the bench framework v4-only for act/extract/observe #2587.

Not touched, for reviewer attention on #2570 itself: the EVAL_VERIFIER_MODEL / keyless-provider and Claude Code result-parsing commits are orthogonal to running a/e/o on v4 (they only affect the agent/external-harness grading paths and change nothing unless the env var is set) — candidates for splitting into their own PR.

Verification

• typecheck, fmt, and 410/410 unit tests green
• LOCAL end-to-end through the built CLI: run dropdown -e local -t 1 -m google/gemini-2.5-flash → 1/1 passed (first full v4 pass through the harness; requires packages/extension built for the local launch's extension preload)
• BROWSERBASE connect path not exercised here (no key in this environment) — needs one run dropdown -e browserbase -t 1 to confirm sessionUrl lands in results

---

Summary by cubic

Restores session and debug URLs for v4 eval runs by creating Browserbase sessions first and connecting, and simplifies the harness context with an explicit v3/v4 split for clearer task handling.

• Bug Fixes

  • Restored sessionUrl and debugUrl for v4 tasks. initStagehand now creates the session via launchRunnerProvidedBrowserbaseChrome and attaches with browserbase.connect({ apiKey, sessionId }).
  • Releases the session on cleanup and on init failure paths.

• Refactors

  • BenchHarnessContext is now a discriminated union (sdk: "v4" | "v3"). Removes v4Page ?? page fallback and throws on legacy tasks reaching the v4 context.
  • Dropped the unused systemPrompt from initStagehand in @browserbasehq/stagehand v4 flow.
  • Moved legacy test fixtures from combination to agent.

^{Written for commit e31cb72. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: e31cb72

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 4 files

Architecture diagram

sequenceDiagram
    participant CLI as CLI Runner
    participant Bench as benchHarness.ts
    participant Init as initStagehand.ts
    participant BB as Browserbase SDK
    participant Task as Task Definition
    participant Result as TaskResult

    Note over CLI,Result: NEW: v4 Session URL Flow

    CLI->>Bench: stagehandHarness.start(input)
    Bench->>Init: initStagehand({environment, modelName, logger})
    
    alt environment === "BROWSERBASE"
        Init->>BB: launchRunnerProvidedBrowserbaseChrome()
        BB-->>Init: {sessionId, sessionUrl, debugUrl, cleanup}
        Init->>Init: Store sessionUrl & debugUrl
        Init->>BB: browserbase.connect({apiKey, sessionId})
        alt connect fails
            BB-->>Init: error
            Init->>BB: endSession() (release session)
            Init-->>Bench: throw
        end
        BB-->>Init: browser handle
    else environment === "LOCAL"
        Init->>Init: localBrowser.launch({headless: false})
        Init-->>Init: sessionUrl="" debugUrl=""
    end

    Init->>Init: stagehand.init({browser, model})
    alt init fails
        Init->>BB: browser.close()
        Init->>BB: endSession()
        Init-->>Bench: throw
    end
    Init-->>Bench: {stagehand, page, sessionUrl, debugUrl, endSession}

    Note over Bench,Task: CHANGED: Context is now discriminated union

    Bench->>Bench: Build ctx with sdk: "v4" discriminant
    Bench->>Bench: Store sessionUrl & debugUrl in ctx
    Bench-->>CLI: {ctx, cleanup}

    CLI->>Task: executeBenchTask(ctx)
    alt sdk === "v4"
        Task->>Task: Uses stagehand + page (v4 SDK)
    else sdk === "v3"
        alt legacy task on v4 context
            Task-->>CLI: EvalsError ("Legacy task cannot run on v4")
        else
            Task->>Task: Uses v3 + agent + page
        end
    end
    Task-->>CLI: TaskResult

    Note over CLI,Result: CHANGED: sessionUrl flows to TaskResult

    CLI->>Result: withBenchSessionUrls(result, ctx)
    Result->>Result: Inject sessionUrl & debugUrl from ctx
    Result-->>CLI: Enriched TaskResult

    Note over CLI,BB: CHANGED: Cleanup releases session explicitly

    CLI->>Bench: cleanup()
    Bench->>Init: stagehand.close()
    Bench->>BB: browser.close() (disconnects only)
    Bench->>BB: endSession() (RELEASE session - best effort)
    Note over Bench,BB: browser.close() on connected handle <br/>does not release Browserbase session

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>