💡 Teach-by-Demonstration: Avatar-Guided Record & Replay Skill Builder

[github-actions[bot]]

Summary

Let non-technical users teach the avatar new workflows by recording themselves performing tasks, with the avatar narrating what it learns and creating reusable skills that can be replayed autonomously. The avatar proactively suggests recording when it detects repetitive patterns — turning observation into automation through conversational collaboration.

Market Signal

Teach-by-demonstration is emerging as a real product category in 2026. OpenAI Codex shipped Record & Replay (macOS, June 2026) turning demonstrated workflows into reusable skills. SkillForge launched on ProductHunt converting screen recordings into agent-native SKILL.md format. Academic work (AgentRR, arXiv 2505.17716) validates the record-summarize-replay pipeline. Two production products and peer-reviewed research confirm the pattern.

User Signal

TalkTerm's primary persona (Sarah, non-technical PM) cannot script automations but performs repetitive workflows manually — weekly reports, stakeholder updates, data collection. The avatar's conversational framing ("I noticed you do this every week — want me to learn how?") matches TalkTerm's companion-not-tool design principle. Cross-session memory (FR46-47) provides the observation foundation.

Technical Opportunity

Claude Computer Use is now production-ready on macOS/Windows (April 2026) enabling screen capture and UI interaction. The AgentBackend abstraction layer can be extended with a SkillRecorder port. Recorded skills stored as structured SKILL.md files in the workspace align with the SDK's settingSources: ["project"] which loads skills from the working directory. The avatar's Rive state machine already supports animation states suitable for a "learning" mode.

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires Computer Use integration (now production) and skill storage format; file-level recording is simpler and can be an initial step
Impact	high	Transforms non-technical users into automation creators — directly serves TalkTerm's core value proposition
Urgency	med	Two competitors have shipped; the window to differentiate with avatar-guided teaching is open but narrowing

Adversarial Review

Strongest objection: Non-technical users don't think in "workflows to automate." They won't proactively initiate recordings, making this a feature that sounds good in demos but gathers dust.

Rebuttal: The avatar initiates teaching, not the user. By analyzing session history (cross-session memory, FR46-47) and observing repetitive patterns, the avatar proactively offers to learn. TalkTerm's unique angle is avatar narration during recording ("Got it — step 3 is saving to OneDrive"), creating a collaborative learning experience rather than a passive recording tool. This inversion — the AI asks to learn from the human — is differentiated from both Codex (user-initiated) and SkillForge (developer-focused).

Suggested Next Step

Spike: prototype a minimal skill recorder that captures file operations in the workspace via SDK tool-use events, stores them as a replay script, and re-executes on voice command. Defer screen-level recording to post-Computer Use integration.