UI Context Picker

[gstangl]

Body

What are you proposing?

A visual DOM-to-context feature in Zed. Either a built-in browser pane next to the editor, or a tight integration with an external browser window. When I click on a rendered element in the running app, Zed automatically attaches the corresponding source (component file, JSX/HTML node, CSS/Tailwind classes) as context to the agent. From there I can describe the change in natural language and the edit lands in the right file.

Why does this matter?

Today, the loop between seeing something in the browser and editing it in code is slow:

1. See an element in the browser I want to change
2. Switch to editor
3. Hunt for the right file and the right line
4. @-mention it in the agent panel
5. Describe which element I mean in words
6. Hope the agent picks the right one

With visual selection, this collapses to: click, type the change, done.

This matters because:

• Frontend and design-to-code work is one of the biggest use cases for AI-assisted editing
• Describing UI elements in words is lossy and error-prone, especially in dense layouts
• The current @-mention flow assumes I already know the file structure, which breaks for new codebases or unfamiliar projects
• It would make Zed the first editor that properly closes the visual-to-code loop

Are there any examples or context?

The best example is actually in Claude Code and the way I can visually pick elements in there

The gap: nobody has combined a super fast and lean real editor + capable agent + visual element picker.

Possible approach

A few directions, ordered by ambition:

1. External browser, lightweight: a Zed extension or browser companion (Chrome extension) that, on alt-click or similar, sends the element's source location and surrounding context to Zed's agent panel as an attached context item. React/Vue/Svelte already expose source location in dev mode via Fast Refresh / HMR metadata.

2. Built-in preview pane: a webview pane inside Zed showing localhost. Click on an element with a modifier key, and the corresponding source node + file gets added as agent context. Could piggyback on the existing @-mention context system.

3. Framework-aware mapping: for React/Vue/Svelte, use the existing source maps and component source location data that dev tools already rely on. For plain HTML/CSS, fall back to selector-based attachment.

The minimum viable version is probably (1): a browser extension that talks to Zed via a local socket and pushes context into the agent panel. That validates the workflow without requiring a full webview implementation.

Love what you're building. Zed is the only editor where the agent feels like a native citizen, not a bolted-on plugin. This feature would make it the obvious choice for anyone doing UI work.

Gerald