[功能建议 / Feature Request] 侧边栏浏览器批注与 Add to chat / Sidebar Browser Annotate & Add to chat

[leung-alex]

中文

希望 DeepSeek Harness 官方桌面端在现有右侧栏 Browser 中加入类似 Codex 的 Annotate / Add to chat：用户直接选中页面对象、添加评论，把明确的页面上下文加入当前会话，让 Agent 准确理解“我说的是这里”。文件预览中的选区也希望能使用同一套入口。

使用场景

• 在网页上点选某个按钮，留言“把这个按钮的间距调大”，无需截图后手动描述位置。
• 选中表格中的一个金额或图表区域，留言“解释这里的异常”，附上页面位置和必要上下文。
• 在右侧文件预览中选中几行代码或一段 Markdown，点击 Add to chat，再让 Agent 解释、修改或继续处理。

目前官方 ui-sidebar-browser 文档说明浏览器内容不会进入模型请求。浏览器操作后端可以让 Agent 自行读取页面，但我希望补上的是用户主动指明目标并提供反馈的交互。

希望的交互流程

1. 当前 Browser tab 的工具栏提供 Annotate 入口。
2. 用户可点选 DOM 元素、选中文字，或框选区域；页面高亮目标，并允许填写评论。
3. 点击 Add to chat 后，在当前会话的 Composer 中形成可预览、编辑、删除的上下文条目，支持一次加入多个目标。
4. 用户确认发送时，评论和上下文随消息一起提交；Agent 能获取真正的目标信息，而不只是一个界面上的蓝框。

网页上下文建议包括 URL、页面标题、目标 selector/定位信息、相关文本、区域位置，以及可选的元素或区域截图。开发页面可在条件允许时补充组件名称、源文件及行列号，但普通生产网页也应能使用基础批注，不以 React 或源码映射为前提。

文件预览的 Add to chat 则包括文件路径、选中文字和可定位的行列范围，并尽量保留选择时的文件版本信息；使用现有预览组件即可，不需要强制换用某一种编辑器。

官方能力与插件扩展

优先希望这成为官方 Browser / Document Preview 的基础交互，保留当前 tab、会话和页面状态。如果短期更适合由插件实现，也希望官方提供稳定的扩展点：目标选取、页面上下文采集、截图、批注生命周期，以及向当前 Composer 添加上下文的接口。这样社区插件可以增强现有浏览器，而不必另外搭一套预览页或独立桌面外壳。

首版可以先做 DOM/区域选择 + 评论 + 可选截图 + Add to chat；文件选区和 Agent 回复/解决批注状态可逐步补充。

相关讨论和参考

• 已有 #8750：聊天回答选区作为 Composer annotations。本建议重点扩展到官方侧边栏里的网页元素、区域和文件预览，并统一上下文体验。
• 非官方社区参考：alaliqing/dsh-annotate、Nono-neko/dsh-browser。它们已探索批注和 Agent 回传，但使用各自的预览或浏览器载体；希望官方现有 Browser 也能提供这样的基础能力。
• 交互参考：Codex 浏览器批注。这里请求的是浏览器本身的选择、批注和会话回传能力。

---

English

I would like the official DeepSeek Harness desktop app to support Annotate / Add to chat in the existing sidebar Browser, similar to Codex. Users could select a page element, add a comment, and attach precise page context to the current conversation so the Agent understands exactly what “this part” refers to. File-preview selections could use the same workflow.

Use cases

• Select a button on a webpage and comment “increase this button’s spacing,” without taking a screenshot and manually describing its location.
• Select a value in a table or a chart region and ask “explain this anomaly,” with the relevant page location and context attached.
• Select several lines of code or a Markdown passage in the sidebar file preview, click Add to chat, and ask the Agent to explain, modify, or continue working on it.

The current official ui-sidebar-browser documentation says browser content is not included in model requests. Browser automation backends can let an Agent read pages, but the missing interaction I am requesting is user-directed selection and feedback about a specific target.

Suggested workflow

1. Add an Annotate entry point to the current Browser tab’s toolbar.
2. Let users select a DOM element, highlight text, or draw a region. Highlight the target and provide a comment field.
3. Add to chat creates a context item in the current conversation’s Composer. Users can preview, edit, or remove it and attach multiple targets.
4. When the user sends the message, include the comments and context in the Agent request, so the Agent receives actionable target information rather than only a visual highlight.

Suggested webpage context includes the URL, page title, a selector or other locator, relevant text, region coordinates, and an optional element or region screenshot. Development pages could also provide component names, source files, and line/column locations when available. Basic annotations should also work on ordinary production webpages, without requiring React or source mapping.

For file previews, Add to chat should include the file path, selected text, a locatable line/column range, and, where possible, the file version at selection time. This could build on existing preview components without requiring a specific editor library.

Built-in capability and plugin extension points

Ideally, this would be a basic interaction in the official Browser / Document Preview while preserving the current tab, conversation, and page state. If a plugin is the better short-term path, stable extension points would help: target selection, page-context collection, screenshots, annotation lifecycle, and an API for adding context to the current Composer. Community plugins could then enhance the existing browser without needing a separate preview page or desktop shell.

A first version could focus on DOM/region selection + comments + optional screenshots + Add to chat. File selections and Agent replies or annotation resolution states could follow incrementally.

Related discussions and references

• #8750: Add selected response text as composer annotations is related. This request extends the interaction to webpage elements, regions, and file previews in the official sidebar and asks for a shared context workflow.
• Unofficial community references: alaliqing/dsh-annotate and Nono-neko/dsh-browser. They explore annotations and passing feedback to Agents using their own preview/browser surfaces. I would like the existing official Browser to offer this basic capability as well.
• Interaction reference: Codex browser annotations. The requested feature is the browser’s selection, annotation, and conversation-context workflow.

[TheStreamCode]

Cross-reference: I added implementation notes to #7075, and the same missing seat covers region selection and annotation.

The constraints are identical: the Browser body declares no child slot, no public service exposes the page (executeJavaScript / capturePage / tab identity), and image attachments cannot be minted through IConversation. So neither a picker nor a region overlay can ship as a plugin on 0.2.1-rc.2. The minimal ask is listed in #7075: a Browser-body child slot, a public browser-tab service, public image-draft minting, and a documented Web position. Related: #9909.

[dshapp]

同样的需求，在官方侧栏浏览器里还没有之前，Chrome 里已经可以这样用了（利益相关：这是我们做的 Chrome 扩展加 dsh-chrome-control 插件，非官方）。

在扩展侧栏里点「圈选元素」，鼠标划过时实时高亮，点一下就在点击位置弹出批注框，写一句「把这个按钮的间距调大」，元素信息会以卡片的形式进入 DSH 的输入框。也可以打开开关，把当前页面整页作为上下文带进对话。

和你提的版本相比，区别有两点：它在你自己的 Chrome 里，不在桌面端的侧栏浏览器里；文件预览里选中内容加入对话这一块，我们没有做。截图在 #10996。如果官方在侧栏浏览器里做了这个功能，我们也想看看交互是怎么设计的。

[leung-alex]

更新一下：我已经把这套批注能力抽离成独立的开源库 annotation-kit，并发布了 0.1.0，采用 MIT 许可证。

目前覆盖网页元素/区域批注、代码与 Markdown 选区、PDF/Office、XLSX/CSV，以及助手回复选区，包含评论编辑、独立批次发送和发送后卡片。仓库里提供了真实运行截图、独立 React 示例和完整的 DSH 接入示例。

• GitHub / Source & screenshots
• npm：annotation-kit@0.1.0
• 使用文档（中文） / Host integration (English)
• DSH rc.5 接入示例、补丁及运行步骤 / English

DSH 示例固定到官方 0.2.1-rc.5，提供补丁检查、应用、构建、启动与撤销命令，以及无需远程模型账号的本地演示模型。当前接入仍需要应用补丁并重新构建 DSH；这次发布的是公共库和源码集成示例，还不能直接从插件市场安装到未经修改的 DSH。也期待官方后续提供稳定的浏览器、截图及 Composer 扩展接口，让这类能力能以普通插件交付。欢迎试用和反馈！

Quick update: annotation-kit is now open source (MIT) and published on npm as 0.1.0. It includes React 18 annotation components, snapshots/state, an Electron bridge, runnable examples and real screenshots. The DSH example targets official 0.2.1-rc.5 and currently requires the supplied source patch and a rebuild. Stable upstream extension points would still help make this available as a regular marketplace plugin.