💡 Screen Context Capture: Avatar-Guided Visual Understanding of Active Windows

[github-actions[bot]]

Summary

Let users share their active window or screen region with the avatar via voice command ("look at this") or button click. The avatar analyzes the visual context using Claude's vision capabilities and offers contextual help, converting what it sees into structured overlay responses rather than raw text. This is a direct competitive response to ChatGPT Voice's Appshots feature launched July 23, 2026.

Market Signal

ChatGPT Voice Desktop launched July 23, 2026 with "Appshots" — the ability to analyze the user's active window via voice. This is now a competitive baseline for voice-first desktop AI apps. OpenAI's GPT-Live powers full-duplex audio with screen analysis on macOS and Windows, available to Plus, Pro, Business, Edu, and Enterprise plan users. TalkTerm must match this capability while differentiating through avatar-guided UX rather than text-dump responses.

Microsoft Copilot also supports screen context through Windows Agent Studio and Power Agent Desktop. Google Gemini Spark is rolling out desktop integration with similar capabilities in summer 2026. Screen context awareness is converging as table stakes for desktop AI agents.

User Signal

Existing idea #246 (Screenshot Context: Point-and-Tell) identified the need but predates the ChatGPT competitive pressure. Non-technical users frequently need help understanding what's on their screen — spreadsheets, documents, error messages — and copying text into a chat is a friction barrier. The voice-initiated screen capture model ("look at this") removes that friction entirely.

Technical Opportunity

Electron's desktopCapturer API provides screen/window capture natively. Claude's vision API accepts image inputs. TalkTerm's overlay system can present analysis as structured cards rather than wall-of-text. The preload bridge already defines the IPC seam for new capabilities.

Implementation path:

1. Add screen:capture IPC channel to capture active window or user-selected region
2. Send screenshot to Claude vision via Agent SDK with contextual prompt
3. Avatar narrates findings conversationally
4. Overlay displays structured analysis (e.g., data tables, error explanations, document summaries)

The AgentBackend abstraction layer handles the vision API call; the renderer only needs the capture trigger and result display.

Assessment

Dimension	Score	Rationale
Feasibility	high	Electron desktopCapturer + Claude vision are both production-ready APIs
Impact	high	Competitive parity with ChatGPT Appshots; differentiating avatar UX on top
Urgency	high	ChatGPT shipped this yesterday (July 23); market window for differentiation is narrow

Adversarial Review

Strongest objection: Privacy concerns with screen capture — users may accidentally share sensitive content visible on screen (passwords, personal messages, financial data).

Rebuttal: Screen context is always user-initiated (voice command or button press), never passive or always-on. No screenshots are persisted beyond the active analysis request (aligns with NFR7 audio policy for voice data). The avatar provides a verbal consent prompt: "I'll look at your screen now — make sure nothing sensitive is visible." This matches the consent model ChatGPT and Microsoft Copilot already use. The confirmation gate (FR20) can be applied to screen capture as a destructive-action-equivalent.

Suggested Next Step

Design the screen:capture IPC channel and renderer-side UI for the capture trigger. Prototype with Electron desktopCapturer + Claude vision API. Define the avatar narration pattern for visual analysis results — the avatar should explain what it sees before what it recommends.

---

Sources: ChatGPT Voice Desktop (9to5Mac), ChatGPT Voice announcement (Fortune), Microsoft Copilot Studio May 2026