💡 Phoneme-Driven 2D Avatar Lip Sync: Real-Time Viseme Animation via Rive State Machine

[github-actions[bot]]

Summary

Enable real-time mouth animation synchronized with avatar speech by mapping TTS phoneme output to Rive viseme state machine inputs. The avatar's mouth shapes follow actual speech sounds (not just open/closed based on audio amplitude), creating emotionally engaging lip-sync that makes the conversational experience feel genuinely alive — reinforcing TalkTerm's core differentiator as a relationship-layer product.

Market Signal

D-ID (best AI avatar generator 2026), HeyGen, and NVIDIA ACE all feature production lip-sync as a baseline capability. A production-ready Rive implementation guide was published in 2026, documenting the full pipeline: TTS → phoneme extraction → viseme mapping → Rive state machine inputs (isTalking, visemeIndex). Interactive avatar assistants with lip sync report measurably higher user engagement vs static or amplitude-only animation. Rive reports teams achieving 4x faster production with file sizes 90% smaller than After Effects/Lottie workflows.

User Signal

Existing idea #58 (3D Avatar Evolution with Lip-Sync and Emotional Expression) focuses on evolving to 3D — a much larger effort requiring WebGPU or a native module. This proposal targets the immediate, achievable opportunity: adding phoneme-based lip sync to the existing 2D Rive avatar using the already-planned @rive-app/react-webgl2 renderer. The emotional engagement gap between a talking-head avatar and one with accurate lip sync is the difference between "neat tool" and "trusted companion."

Technical Opportunity

Rive's state machine natively supports data-driven inputs: Number inputs for viseme indices and Boolean inputs for speaking state. The architecture already uses @rive-app/react-webgl2 (Epic 3). ElevenLabs (existing idea #213) provides phoneme-level timing data in its API response. The standard phoneme-to-viseme mapping is well-defined: A/AA→A, E/EH→E, O/OW→O, M/B/P→Closed (6 viseme shapes + neutral). Rive's new data binding feature can connect animation properties directly to external data sources.

Two-phase implementation path:

• Phase 1 (Web Speech API MVP): Audio amplitude analysis via Web Audio API's AnalyserNode drives basic open/close mouth animation. Achievable immediately with zero additional dependencies.
• Phase 2 (Cloud Voice): Phoneme-accurate visemes when ElevenLabs or another phoneme-capable provider is integrated. Same Rive state machine inputs — avatar design work is done once.

Assessment

Dimension	Score	Rationale
Feasibility	high	Phase 1 requires only Web Audio API (built into Chromium) and Rive state machine inputs (already in the stack). Phase 2 builds on the cloud voice provider work (#213/#56).
Impact	med	Enhances emotional engagement and "companion" feel. Reinforces the relationship-layer differentiator. Not a functional capability but a significant UX quality improvement.
Urgency	low	Not blocking any user workflow. Phase 1 can ship whenever Epic 3 (Avatar Presence) is complete. Phase 2 depends on cloud voice adoption timeline.

Adversarial Review

Strongest objection: Full phoneme-based lip sync requires a TTS provider that outputs phoneme timing data. Web Speech API (the MVP TTS) does NOT provide phonemes — only audio output. This feature is effectively blocked on cloud voice provider adoption (#213/#56).

Rebuttal: The two-phase approach resolves this cleanly. Phase 1 uses audio amplitude analysis (no phonemes needed) to drive basic mouth animation — this alone is a significant upgrade over a static "speaking" state. The Rive state machine is designed once with inputs for both amplitude-based and phoneme-based modes. When cloud voice ships, Phase 2 activates phoneme accuracy with no avatar redesign. Phase 1's "low latency is more important than perfect realism" principle (from the dev.to production guide) validates the amplitude approximation approach.

Suggested Next Step

Design the Rive avatar state machine with viseme inputs (6 mouth shapes + neutral) and implement Phase 1 amplitude-based animation driver using Web Audio API's AnalyserNode on the TTS audio output. Define the SpeechAnimationDriver interface in src/renderer/speech/ to abstract the amplitude vs. phoneme input source.