💡 Avatar-Fronted Claude Computer Use for Cross-App Workflows

[github-actions[bot]]

Summary

Wrap Anthropic's newly released Computer Use capability in TalkTerm's avatar-first UX so non-technical users can delegate cross-application desktop tasks (filling spreadsheets, navigating browsers, moving files between apps) to the avatar while retaining conversational permission prompts. The avatar becomes the trusted, embodied front-end for a capability that otherwise feels invasive in its raw form — directly extending the confirm-plan pattern (FR20) to actions outside TalkTerm's own window.

Market Signal

Anthropic launched Claude Computer Use into research preview on 2026-03-23, roughly two weeks before this ideation run. It runs a continuous screenshot → analyze → act → verify loop on macOS (Windows support announced), always requests permission before accessing new apps, and ships blocked-by-default categories (banking, crypto, adult). This is a first-party Anthropic capability that sits directly on top of the Claude Agent SDK that TalkTerm already bundles. Multiple 2026 trend pieces describe it as "the agentic shift of 2026." Competitors in the desktop-agent space (Cursor, Windsurf, Warp) are terminal/editor-bound and cannot act across arbitrary apps. No avatar-based product currently fronts Computer Use — the raw Claude Desktop experience is a text chat asking for screen access, which is exactly the kind of friction TalkTerm's persona layer is designed to remove.

User Signal

TalkTerm's PRD targets Sarah the non-technical PM (Journey 1) and explicitly positions the product as bringing agent execution power to users who cannot use a terminal. Multi-app workflows — "pull these three numbers from our dashboard, paste them into the board deck, email it to Alex" — are exactly the kind of task Sarah cannot script but can describe. Journey 3 already documents a conversational failure-recovery pattern; Computer Use extends that same "conversational accountability" model to browser automation and cross-app moves. The PRD's "Explicitly NOT in MVP" list does not exclude computer use, and FR20 (confirmation before non-reversible actions) and FR22 (hide intermediate thinking) map one-to-one onto Computer Use's permission model and screenshot-loop opacity.

Technical Opportunity

TalkTerm's planned architecture isolates backend capability behind an AgentBackend port in src/shared/types/ports/, with claude-sdk-backend.ts as the anti-corruption layer — this is precisely the extension point where Computer Use tools plug in alongside Read/Edit/Write/Bash. The confirm-plan pattern (FR20/FR21) and permission callbacks already specified for destructive file actions generalize directly to screen/click/type actions with no new UX primitives needed. The audit trail (FR27-29) gains a new event type per screen action, reusing the existing AuditRepository port. Because the Claude Agent SDK runs in-process in the Electron main process, no extra subprocess or VM is required — Electron already has the OS permissions (admin check per FR39) to drive screen capture and synthetic input events on macOS/Windows.

Assessment

Dimension	Score	Rationale
Feasibility	medium	SDK support is first-party; main cost is new permission UX and per-platform input event handling (macOS accessibility API, Windows UIAutomation). macOS-first matches MVP platform scope.
Impact	high	Unlocks the entire category of multi-app workflows that today's PRD can only handle inside TalkTerm's own window. Directly strengthens the core value prop ("agent power without a terminal") for the non-technical persona.
Urgency	high	Anthropic shipped this 2 weeks ago. Razer AVA (H2 2026) and NVIDIA ACE are racing to own embodied-agent mindshare. Shipping an avatar-fronted Computer Use in Phase 2 stakes a differentiated position before the category crowds.

Adversarial Review

Strongest objection: Computer Use is a safety minefield. One misclick on a banking site, one wrong paste into a production system, and the "trust built through avatar conversation" (Journey 3's whole point) collapses. The blast radius of this feature is incompatible with a solo-developer scope.

Rebuttal: The proposal is to wrap Computer Use, not extend it. Anthropic's built-in blocked categories (banking, crypto, adult) remain active; TalkTerm adds its own per-action confirmation layer via the existing FR20 confirm-plan pattern. Initial scope is a read-only "narrate mode": the avatar describes what it would click while the SDK takes screenshots but never issues synthetic input events. That validates the UX with zero blast radius. Write mode ships behind a separate opt-in with a per-app allowlist the user builds conversationally ("Mary, you may drive Figma in this session"). Solo-dev scope is respected because Anthropic owns the hard parts (vision, planning, action loop) — TalkTerm owns only the permission UX and the anti-corruption layer.

Suggested Next Step

Spike (1-2 weeks): build a read-only "narrate mode" prototype that wires the Claude Agent SDK Computer Use tool into a FakeComputerUseBackend behind the AgentBackend port, and has the Mary avatar describe what she sees on screen and what she would do next — without issuing synthetic input. Validate (1) screenshot-loop latency against NFR1b's 3-second budget, (2) confirm-plan overlay legibility for multi-step action plans, (3) audit trail schema coverage for screen/click/type events. If all three pass, create a product brief for Phase 2 inclusion.

---

Proposed by the BMAD Analyst (Mary) on 2026-04-06.
Workflow run

[github-actions[bot]]

Weekly Update — 2026-06-19

What Changed

Claude Computer Use is now production-ready on desktop. As of April 24, 2026, Computer Use is available in Cowork and Claude Code in the Claude Desktop application for both macOS and Windows (research preview for Pro and Max plans). The capability has matured from early beta to production: Claude captures screenshots, perceives UI elements (buttons, menus, text fields), opens applications, navigates menus, fills forms, and executes complex multi-step workflows across different applications.

Competitive context: OpenAI Codex also shipped Record & Replay (macOS, June 2026), which turns demonstrated workflows into reusable skills — a related but distinct approach (teaching by demonstration vs. autonomous screen control).

TalkTerm relevance: With Computer Use now in production, TalkTerm can integrate it as a capability the avatar invokes for cross-app workflows. The avatar explains what it sees and what it's about to do, providing the trust layer that raw Computer Use lacks: "I can see your Excel spreadsheet is open. Let me grab those numbers and put them into the report."

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	high	↑ Computer Use is now production-ready on macOS/Windows — no longer experimental
Impact	high	high	→ Cross-app workflow capability remains a transformative differentiator
Urgency	med	high	↑ Production availability means integration can begin; Codex Record & Replay creates competitive pressure

Recommendation

Advance — The avatar-fronted approach (explaining what it sees, asking permission before acting) is TalkTerm's unique angle vs. raw Computer Use. Prototype: have the avatar take a screenshot, describe what it sees, and ask the user what to do — validating the trust-building narration pattern.