Skip to content

Browser: 'Add Area Screenshot to Chat' and 'Add Full Page Screenshot to Chat (Experimental)'#318791

Merged
jruales merged 7 commits into
mainfrom
jruales/2026.05-area-full-screenshot
May 29, 2026
Merged

Browser: 'Add Area Screenshot to Chat' and 'Add Full Page Screenshot to Chat (Experimental)'#318791
jruales merged 7 commits into
mainfrom
jruales/2026.05-area-full-screenshot

Conversation

@jruales
Copy link
Copy Markdown
Contributor

@jruales jruales commented May 28, 2026

New feature: 'Add Area Screenshot to Chat'

area.mp4
  • Great for cases when 'Add Element to Chat' won't capture the exact region you would like to share with the agent

New feature: 'Add Full Page Screenshot to Chat (Experimental)'

full-page.mp4
  • Gated under an experimental setting, default off
  • Great for giving the agent the "big picture" of a page without needing it to take multiple screenshots. Seems to work well as long as pages aren't super long.
  • Known issues:
    • Every once in a while it can produce a glitchy result where instead of rendering the whole page, it just renders part of the page repeatedly. I have seen less likelihood of glitchiness when zoom level is 100%, but even then, it has happened.
      • image
    • It relies on an experimental CDP method
    • Using it with very long pages brings issues:
      • Slow to capture
      • Can result in a corrupted image being attached to chat
      • The very large resulting image causes perf issues in the chat panel, especially because that panel uses the original image for thumbnails instead of pre-computing a resized thumbnail sized image. For example, if you add multiple such large image attachments, the UI can become sluggish to remove attachments or even to type in the chat.
    • Very long web pages can also cause crashes. In the future, we might want to have a maximum height that we allow or otherwise have some other way to isolate and prevent crashes when a page is too long.
    • It can sometimes trim off a few pixels from the right side of the full page image because of how it disables the scrollbar when taking the screenshot but doesn't adjust the screenshot size accordingly.
  • Workarounds:
    • By default, CDP's full page screenshots don't work well with zoom level other than 100%, because for example, if you set zoom level 50% and take a full page screenshot, the screenshot will be twice as wide and tall as it should be. And if you take a full page screenshot at 200% zoom, it will end up cropping it to 1/2 the size it should be, cutting content. (You can try this by asking the agent to take a full page screenshot using the Run Playwright Code tool and see the result). I have worked around this by setting a clip parameter that adjusts for the zoom level.
      • This is what 50% zoom looked like before I added the workaround: image

Copilot AI review requested due to automatic review settings May 28, 2026 16:34
@jruales jruales changed the title Jruales/2026.05 area full screenshot Browser: 'Add Area Screenshot to Chat' and 'Add Full Page Screenshot to Chat (Experimental)' May 28, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds new Integrated Browser "Add to Chat" screenshot flows for selected areas and experimental full-page capture, extending the browser view service/model, preload picker UI, and workbench chat actions.

Changes:

  • Adds browser view API/state/events for area selection and full-page screenshot options.
  • Implements main-process area selection routing and CDP-based full-page screenshot capture.
  • Adds workbench menu actions, context keys, telemetry, and configuration gating for new screenshot tools.
Show a summary per file
File Description
src/vs/workbench/contrib/browserView/electron-browser/features/browserEditorChatFeatures.ts Adds chat attachment helpers, area/full-page screenshot actions, menu grouping, telemetry, and experimental setting.
src/vs/workbench/contrib/browserView/common/browserView.ts Mirrors new area-selection state, events, and service calls on the workbench model.
src/vs/platform/browserView/electron-main/browserViewMainService.ts Exposes new area-selection service methods/events through the main service.
src/vs/platform/browserView/electron-main/browserViewInspector.ts Adds top-frame area picker lifecycle and IPC handling.
src/vs/platform/browserView/electron-main/browserView.ts Adds area-selection state and full-page screenshot capture support.
src/vs/platform/browserView/electron-browser/preload-browserView.ts Adds preload-side drag-to-select area picker overlay.
src/vs/platform/browserView/common/browserView.ts Adds command IDs, shared rectangle type, full-page screenshot option, and area-selection service API.

Copilot's findings

  • Files reviewed: 7/7 changed files
  • Comments generated: 2

Comment thread src/vs/platform/browserView/electron-main/browserViewInspector.ts
@jruales jruales requested a review from kycutler May 28, 2026 18:14
@jruales jruales marked this pull request as ready for review May 28, 2026 18:14
@jruales jruales enabled auto-merge (squash) May 29, 2026 05:39
@jruales jruales disabled auto-merge May 29, 2026 05:39
@jruales jruales enabled auto-merge (squash) May 29, 2026 05:39
@jruales jruales requested review from Copilot and removed request for kycutler May 29, 2026 05:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 7/7 changed files
  • Comments generated: 1

Comment on lines +656 to +670
const result = await this.debugger.sendCommand('Page.captureScreenshot', {
format: 'jpeg',
quality,
captureBeyondViewport: true,
// In theory, `clip` defaults to the full area when not explicitly passed, but in practice it doesn't work when
// the zoom level isn't 100, because it doesn't multiply the width and height by zoomFactor like we do here.
// Setting the clip explicitly, we can multiply by zoomFactor and thus work around this Chromium bug.
// Note that even with this workaround, we often see that the page isn't fully captured and might repeat
// visual content from the top at the bottom, instead of showing the bottom of the page.
// - Sidenote: Setting the scale here to be zoomFactor or 1/zoomFactor has strange effects and doesn't solve the issue.
// - Another sidenote: Currently the scrollbar width isn't accounted for. If a scrollbar exists, we should add the
// vertical scrollbar's width and horizontal scrollbar's height to the clip dimensions, since the image is currently
// clipped by that amount (this also happens when no clip parameter is provided; ideally it should be fixed upstream
// in Chromium).
clip: { x: 0, y: 0, width: size.width * zoomFactor, height: size.height * zoomFactor, scale: 1 }
@jruales jruales merged commit f211180 into main May 29, 2026
40 of 41 checks passed
@jruales jruales deleted the jruales/2026.05-area-full-screenshot branch May 29, 2026 07:01
@vs-code-engineering vs-code-engineering Bot added this to the 1.123.0 milestone May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants