Add screenshot option to Extract

[miguelg719]

why

Some pages do not properly encapsulate the necessary information required to extract the content requested by the user, and can benefit from a hybrid approach with vision.

what changed

• New extract({ options: { screenshot: true } }) flag. Captures the current viewport as PNG and sends it as an image_url part alongside the a11y tree text in the extraction LLM call.
• Default is false — existing callers see no behavior change.
• Gated to AI SDK clients; throws StagehandInvalidArgumentError otherwise (validated in both the handler and inference.extract, with the handler check running before the screenshot is taken).
• Screenshot capture is bracketed by ensureTimeRemaining() so it respects the extract timeout.
• Wired end-to-end: public ExtractOptions type, ExtractOptionsSchema, OpenAPI ExtractOptions, StagehandAPIClient.extract wire payload, V3.extract → ExtractHandler → inference.extract → buildExtract{System,User}Prompt.

test plan

• extract-screenshot.test.ts (new) — multimodal message shape on happy path; rejection for non-AI SDK clients.
• timeout-handlers.test.ts — handler captures viewport when enabled, skips capture by default, rejects screenshot + non-AI SDK client without touching page.screenshot or the snapshot pipeline.
• api-client-serialization.test.ts (renamed from api-client-observe-variables.test.ts) — wire payload preserves options.screenshot.

---

Summary by cubic

Adds a screenshot option to extract() that sends the current viewport screenshot with the accessibility tree to improve results on visually-driven pages.

• New Features
  • extract({ options: { screenshot: boolean } }) captures a PNG of the current viewport and includes it with the accessibility tree; defaults to false; only supported with aisdk clients (throws StagehandInvalidArgumentError otherwise).
  • Prompts are consolidated and multimodal when enabled: system and user prompts mention the screenshot and using it with the accessibility tree; attaches an image_url data URI; logs when a screenshot is used.
  • Public API, types, and OpenAPI updated (ExtractOptions.screenshot); the API client forwards the flag; tests cover prompt shape, viewport capture, default-off behavior, timeout guards, client-type validation, and serialization.

^{Written for commit d0c7500. Summary will update on new commits. Review in cubic}

[changeset-bot]

🦋 Changeset detected

Latest commit: d0c7500

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@browserbasehq/stagehand	Minor
@browserbasehq/browse-cli	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch
@browserbasehq/stagehand-server-v4	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

✱ Stainless preview builds for stagehand

This PR will update the stagehand SDKs with the following commit message.

feat: Add `screenshot` option to Extract

⚠️ stagehand-python studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ✅ → lint ✅ → test ✅

pip install https://pkg.stainless.com/s/stagehand-python/393ebbee487278360fc844e1fb5034651f37cce4/stagehand-3.20.0-py3-none-any.whl

✅ stagehand-openapi studio · code

Your SDK build had at least one "note" diagnostic.
generate ✅

⚠️ stagehand-csharp studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ✅ → lint ✅ → test ✅

✅ stagehand-ruby studio · code

Your SDK build had at least one "note" diagnostic.
generate ✅ → build ⏭️ → lint ✅ → test ✅

✅ stagehand-kotlin studio · conflict

Your SDK build had at least one note diagnostic.

✅ stagehand-java studio · conflict

Your SDK build had at least one note diagnostic.

⚠️ stagehand-php studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → lint ✅ → test ✅

⚠️ stagehand-typescript studio · conflict

Your SDK build had at least one warning diagnostic.

✅ stagehand-go studio · code

Your SDK build had at least one "note" diagnostic.
generate ✅ → build ⏭️ → lint ✅ → test ✅

go get github.com/stainless-sdks/stagehand-go@ba29e1e97d4744a504adf2990fa4cc1eeabf9bba

---

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-05-20 21:02:43 UTC

[cubic-dev-ai]

No issues found across 13 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Client as "Client (User Code)"
    participant V3 as "V3.extract()"
    participant ExtractHandler as "ExtractHandler"
    participant Page as "Page (Playwright)"
    participant Inference as "extract() (inference.ts)"
    participant LLM as "LLMClient (AI SDK)"
    participant Prompts as "Prompt Builders"

    Note over Client,Prompts: NEW: Screenshot extraction flow

    Client->>V3: extract(instruction, { screenshot: true })
    V3->>ExtractHandler: extract({ screenshot: true, ... })

    ExtractHandler->>ExtractHandler: resolveLlmClient(model)
    
    alt llmClient.type is NOT 'aisdk'
        ExtractHandler-->>Client: throw StagehandInvalidArgumentError
    end

    ExtractHandler->>Page: captureHybridSnapshot(...)
    Page-->>ExtractHandler: snapshot (combinedTree)

    ExtractHandler->>Page: page.screenshot({ fullPage: false, type: 'png' })
    Page-->>ExtractHandler: screenshotBuffer (Buffer)

    ExtractHandler->>Inference: extract({ screenshot: screenshotBuffer, ... })

    alt llmClient.type is NOT 'aisdk'
        Inference-->>ExtractHandler: throw StagehandInvalidArgumentError
    end

    Inference->>Prompts: buildExtractSystemPrompt(includeScreenshot=true)
    Prompts-->>Inference: system prompt referencing accessibility tree and screenshot

    Note over Inference: Builds multimodal user message

    Inference->>Prompts: buildExtractUserPrompt(instruction, domElements, screenshotDataUrl)
    Prompts->>Prompts: Convert screenshot buffer to data URL

    alt screenshotDataUrl present
        Prompts-->>Inference: Multimodal message (text + image_url)
    else no screenshot
        Prompts-->>Inference: Text-only message
    end

    Inference->>LLM: createChatCompletion(messages with image_url)
    LLM-->>Inference: LLM response + usage

    Inference-->>ExtractHandler: extracted data

    ExtractHandler-->>V3: results
    V3-->>Client: extracted content

Loading

_{Re-trigger cubic}

[pirate]

looks good, just make sure we dont emit full base64 to any DB/logs