evals: add the LLMExposure tool-surface contract

[shriyatheunicorn]

A uniform declaration of what a tool surface offers a coding agent:
code_handles (in-scope objects the agent writes code against, mounted by
the harness as a single local-MCP run tool), mcp_server, or cli — plus
the LLM_RUN_TOOL_SERVER/LLM_RUN_TOOL_NAME bindings the mount uses.
Types and constants only; no surface or adapter changes.

Part 1/4 of the #2473 port onto the current SDK generation.

---

Summary by cubic

Updates the tool-surface contract to include agent delivery (AgentMount) and final-state evidence capture, keeping native CoreTool.surface independent from how it’s delivered to the agent. Types + tests only; supports STG-2671.

• New Features
  • Types: AgentMount (via: handles | mcp | cli with promptInstructions), AgentRunToolSpec; ToolStartResult gains optional agentMount and captureEvidence(): Promise<ProbeEvidence>.
  • Enums: ToolSurface adds stagehand_code; CoreTool.family adds stagehand.
  • Constants: AGENT_RUN_TOOL_SERVER ("stagehand_browser"), AGENT_RUN_TOOL_NAME (mcp__stagehand_browser__run), AGENT_RUN_TOOL_RESERVED_HANDLES (startUrl, task, console).
  • Contract semantics: delivery is independent of native surface; CLI env merges over harness env; reserved harness bindings; capture-before-cleanup ordering.
  • Tests: tool-contract.test.ts validates delivery independence, handle mounts, and evidence capture.

^{Written for commit d3ba958. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: d3ba958

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 1 file

Architecture diagram

sequenceDiagram
    participant Harness as Eval Harness (claude_code, codex, …)
    participant Surface as Tool Surface (LLMExposure)
    participant LLM as LLM / Agent
    participant MCP as MCP Runtime (stagehand_browser)
    participant CLI as CLI Subprocess

    Note over Harness,CLI: Tool-surface mount contract (types only, no runtime)

    Harness->>Surface: read .kind, choose mount strategy
    
    alt kind == "code_handles"
        Harness->>Surface: read .handles (named scope values)
        Harness->>Surface: read .runTool (description, codeParam, task, console)
        Harness->>Harness: wrap into single MCP "run" tool
        Harness->>MCP: mount tool under server "stagehand_browser"
        MCP-->>LLM: expose tool as LLM_RUN_TOOL_NAME
        LLM->>MCP: invoke "run" with snippet
        MCP->>MCP: execute snippet with bound handles + task + console
        MCP-->>LLM: return result (stringified, no raw images)
    else kind == "mcp_server"
        Harness->>Surface: read .mcpServers (config object)
        Harness->>MCP: mount server config directly
        MCP-->>LLM: expose surface tools
        LLM->>MCP: direct tool calls
    else kind == "cli"
        Harness->>Surface: read .command (bin + env)
        Harness->>CLI: spawn subprocess with env
        CLI-->>LLM: expose via stdio / args
    end

    Note over Harness,Surface: Post-agent-run artifact capture

    Harness->>Surface: read .captureFinalState (function or undefined)
    alt captureFinalState is defined
        Surface->>Surface: capture screenshot + URL from page
        alt screenshot fails
            Surface->>Surface: swallow error, omit screenshot from result
        end
        alt URL fetch fails
            Surface->>Surface: swallow error, omit URL from result
        end
        Surface-->>Harness: TerminalArtifact (partial allowed)
    else captureFinalState is undefined
        Harness->>Harness: no terminal state capture
    end

    Note over Harness,CLI: Cleanup

    Harness->>Surface: read .cleanup (function)
    Surface->>Surface: release resources (browser, processes, …)

    Note over Harness,LLM: Constants referenced by mounting code

    Harness->>Harness: read LLM_RUN_TOOL_SERVER = "stagehand_browser"
    Harness->>Harness: read LLM_RUN_TOOL_NAME = "mcp__stagehand_browser__run"
    Harness->>LLM: bind tool name in run-tool prompt instructions
    LLM->>LLM: use tool name in function-call requests

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

No issues found across 1 file

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Harness as Eval Harness (claude_code, codex, …)
    participant Surface as Tool Surface (LLMExposure)
    participant LLM as LLM / Agent
    participant MCP as MCP Runtime (stagehand_browser)
    participant CLI as CLI Subprocess

    Note over Harness,CLI: Tool-surface mount contract (types only, no runtime)

    Harness->>Surface: read .kind, choose mount strategy
    
    alt kind == "code_handles"
        Harness->>Surface: read .handles (named scope values)
        Harness->>Surface: read .runTool (description, codeParam, task, console)
        Harness->>Harness: wrap into single MCP "run" tool
        Harness->>MCP: mount tool under server "stagehand_browser"
        MCP-->>LLM: expose tool as LLM_RUN_TOOL_NAME
        LLM->>MCP: invoke "run" with snippet
        MCP->>MCP: execute snippet with bound handles + task + console
        MCP-->>LLM: return result (stringified, no raw images)
    else kind == "mcp_server"
        Harness->>Surface: read .mcpServers (config object)
        Harness->>MCP: mount server config directly
        MCP-->>LLM: expose surface tools
        LLM->>MCP: direct tool calls
    else kind == "cli"
        Harness->>Surface: read .command (bin + env)
        Harness->>CLI: spawn subprocess with env
        CLI-->>LLM: expose via stdio / args
    end

    Note over Harness,Surface: Post-agent-run artifact capture

    Harness->>Surface: read .captureFinalState (function or undefined)
    alt captureFinalState is defined
        Surface->>Surface: capture screenshot + URL from page
        alt screenshot fails
            Surface->>Surface: swallow error, omit screenshot from result
        end
        alt URL fetch fails
            Surface->>Surface: swallow error, omit URL from result
        end
        Surface-->>Harness: TerminalArtifact (partial allowed)
    else captureFinalState is undefined
        Harness->>Harness: no terminal state capture
    end

    Note over Harness,CLI: Cleanup

    Harness->>Surface: read .cleanup (function)
    Surface->>Surface: release resources (browser, processes, …)

    Note over Harness,LLM: Constants referenced by mounting code

    Harness->>Harness: read LLM_RUN_TOOL_SERVER = "stagehand_browser"
    Harness->>Harness: read LLM_RUN_TOOL_NAME = "mcp__stagehand_browser__run"
    Harness->>LLM: bind tool name in run-tool prompt instructions
    LLM->>LLM: use tool name in function-call requests

Loading

_{Re-trigger cubic}