feat(verifier): record agent trajectories

[miguelg719]

Why

The new verifier needs richer evidence than a final screenshot, especially for DOM and Hybrid agent modes where the important facts often live in tool returns, ARIA snapshots, and per-step observations. This PR adds trajectory recording without changing the verifier judgment engine.

What Changed

• Added typed agent events for screenshot, step-finished, step-observed, and final-answer events.
• Added listener-gated post-step probes for screenshots and ARIA trees.
• Attached the settled post-turn probe to every tool call in a DOM/Hybrid turn.
• Added CUA step evidence pairing and final answer capture.
• Added TrajectoryRecorder persistence and a smoke script for trajectory shape and disk layout.

Tests

• pnpm --filter @browserbasehq/stagehand run typecheck
• pnpm --filter @browserbasehq/stagehand-evals run typecheck
• node --import tsx packages/evals/scripts/verify-trajectory-recorder.ts
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 2c95836

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/browse-cli	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cubic-dev-ai]

No issues found across 8 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Agent as Agent Handlers
    participant Bus as Event Bus
    participant Recorder as TrajectoryRecorder
    participant FS as File System
    participant Page as Browser Page

    Note over Agent,FS: NEW: Step-level evidence capture (DOM/Hybrid mode)

    Agent->>Agent: onStepFinish callback fires
    Agent->>Agent: stepCounter++ (per tool call)
    Agent->>Bus: emit agent_step_finished_event
    Note over Bus: stepIndex, actionName, actionArgs, reasoning, toolOutput, finishedAt
    Agent->>Page: page.screenshot() (post-step probe)
    Page-->>Agent: screenshot Buffer
    Agent->>Agent: captureAriaTreeProbe(v3)
    Note over Agent: Best-effort, token-budgeted a11y tree capture
    Agent-->>Agent: ariaTree string | undefined
    loop For each tool call in turn
        Agent->>Bus: emit agent_screenshot_taken_event
        Note over Bus: stepIndex, screenshot, url, evidenceRole: "probe"
        Agent->>Bus: emit agent_step_observed_event
        Note over Bus: stepIndex, url, ariaTree (optional), scroll (optional)
    end
    opt done tool call present
        Agent->>Agent: Build lastFinalAnswer
        Agent->>Bus: emit agent_final_answer_event
    end

    Note over Agent,FS: NEW: Step-level evidence capture (CUA mode)

    Agent->>Agent: screenshotProvider called
    Agent->>Page: page.screenshot()
    Page-->>Agent: screenshot Buffer
    Agent->>Bus: emit agent_screenshot_taken_event
    Note over Bus: stepIndex++, screenshot, url, evidenceRole: "agent"
    Agent->>Agent: executeAction(action)
    Agent->>Agent: emitCuaActionStep()
    Agent->>Bus: emit agent_step_finished_event
    Note over Bus: stepIndex paired with preceding screenshot
    Agent->>Page: page.screenshot() (post-action probe)
    Page-->>Agent: probe screenshot
    Agent->>Bus: emit agent_screenshot_taken_event
    Note over Bus: same stepIndex, screenshot, url, evidenceRole: "probe"
    Agent->>Agent: captureAriaTreeProbe(v3)
    Agent->>Bus: emit agent_step_observed_event
    Note over Bus: stepIndex, url, ariaTree (optional)

    Note over Agent,FS: NEW: Trajectory assembly and persistence

    Recorder->>Bus: subscribe to agent_step_finished_event
    Recorder->>Bus: subscribe to agent_screenshot_taken_event
    Recorder->>Bus: subscribe to agent_step_observed_event
    Recorder->>Bus: subscribe to agent_final_answer_event
    Bus-->>Recorder: events arrive (may be out-of-order)
    Recorder->>Recorder: ensurePartial(stepIndex)
    Recorder->>Recorder: Merge evidence into partial steps

    alt persistEnabled (env-gated by VERIFIER_PERSIST_TRAJECTORIES)
        Recorder->>Recorder: assembleSteps()
        Recorder->>FS: mkdir -p .trajectories/{runId}/{taskId}/
        Recorder->>FS: write trajectory.json
        Recorder->>FS: write core.log
        Recorder->>FS: write task_data.json
        Recorder->>FS: write times.json
        Recorder->>FS: write screenshots/probe/{1..N}.png
        alt verdict provided
            Recorder->>FS: write scores/mmrubric_v1.json
            Recorder->>FS: update task_data.json with verdict
        end
    else persistence disabled
        Recorder->>Recorder: Return in-memory Trajectory only
    end
    Recorder-->>Agent: Trajectory object

Loading

_{Re-trigger cubic}

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 10 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[monadoid]

Do we need these three different patch files or just 1?

[miguelg719]

I can consolidate at the end if we want to have all be 1 patch

[cubic-dev-ai]

2 issues found across 15 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment @cubic-dev-ai review.

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

0 issues found across 4 files (changes from recent commits).

_{Re-trigger cubic}