💡 Screenshot Context: Point-and-Tell Visual Interaction Mode

[github-actions[bot]]

Summary

Add a system-wide hotkey that captures a screen region or window and sends it to the avatar for visual understanding. Users can point at anything on their screen and ask "what's wrong with this?" or "help me understand this." The avatar uses Claude's vision capabilities to analyze the screenshot and respond conversationally, enabling a natural "show and tell" interaction pattern that's more intuitive than describing problems verbally.

Market Signal

Claude Computer Use (launched March 2026) demonstrates that desktop AI agents benefit from visual context. Codex added an in-app browser for visual iteration on frontend designs. The 2026 trend toward multimodal AI interfaces shows that text-only interaction is insufficient for complex desktop workflows. Claude's vision API accepts images natively, and Opus 4.8's 1M context window easily accommodates multiple screenshots alongside conversation history.

User Signal

TalkTerm's PRD includes file upload (FR16) but no screen capture interaction. Non-technical users often struggle to describe visual problems verbally — "the spreadsheet looks wrong" is harder to articulate than to show. Existing idea #67 (Vision-Native Document Intake) covers scanned PDF processing, and #65 (Avatar-Fronted Computer Use) covers avatar-controlled desktop automation. Neither covers the user-initiated "show the avatar what I'm looking at" interaction pattern, which is passive visual understanding rather than active desktop control.

Technical Opportunity

Electron provides desktopCapturer API for screen/window capture. Claude API natively accepts images in message content. The interaction flow maps onto existing patterns: capture → confirm (FR20 confirm-plan pattern) → send to agent → avatar responds with visual analysis → ActionCards for next steps. The IPC bridge (Epic 4) already handles binary data transfer between renderer and main process. Global keyboard shortcuts are supported via Electron's globalShortcut API.

Assessment

Dimension	Score	Rationale
Feasibility	high	Electron desktopCapturer + Claude vision API are production-ready; interaction flow maps onto existing confirm-plan pattern
Impact	med	Significantly expands interaction model beyond text/voice for visual contexts; high utility for specific use cases (spreadsheets, charts, UI review)
Urgency	med	Not time-sensitive to external events; should follow core voice/text interaction implementation

Adversarial Review

Strongest objection: This feels like a nice-to-have rather than a core capability. The primary interaction model (voice + text + ActionCards) already works for TalkTerm's use cases.

Rebuttal: For non-technical users, showing is often easier than telling. "Look at this chart and tell me what stands out" is a natural request that currently requires either copying the chart into a file upload or describing it verbally. The hotkey capture pattern (used by screenshot tools, Loom, CleanShot) is familiar to desktop users. Implementation cost is low (Electron desktopCapturer + Claude vision API), and it significantly expands TalkTerm's utility beyond document-centric workflows to any visual desktop context. This is the kind of feature that makes users say "I can't go back to not having this."

Suggested Next Step

Implement a proof-of-concept: register a global hotkey (e.g., Cmd+Shift+T), capture the focused window via desktopCapturer, display a confirmation preview in the avatar overlay, then send to Claude vision API with the current conversation context. Test with common non-technical scenarios: spreadsheet analysis, document review, UI feedback.