💡 Adaptive Voice Expressivity: User-Controlled Avatar Speech Style Matching Context

[github-actions[bot]]

Summary

Let users control how their avatar sounds — from formal and measured for professional contexts to warm and conversational for brainstorming. Apple's rebuilt Siri AI (WWDC June 2026) now offers adjustable voice expressivity, establishing this as an emerging table-stakes capability. TalkTerm's TTS abstraction layer should expose expressivity parameters from the start, even though MVP uses Web Speech API, so that Phase 2 cloud TTS integration (ElevenLabs v3, Cartesia Sonic 4 Turbo) unlocks full expressivity without architecture changes.

Market Signal

• Apple Siri AI (WWDC June 2026): Added adjustable voice expressivity as a headline feature of the rebuilt assistant
• ElevenLabs v3: Offers granular voice parameter controls (stability, similarity, style, expressivity) across 70+ languages
• Cartesia Sonic 4 Turbo (May 2026): Achieves ~40ms time-to-first-audio with tunable expression parameters — the fastest commercial TTS available
• ChatGPT Voice Desktop (July 2026): Uses fixed expressivity with conversational fillers ("mhmm", "got it") — an area where user-controlled expressivity could differentiate TalkTerm

User Signal

• TalkTerm targets both professional (PM doing a product review) and casual (brainstorming session) contexts — a single voice tone cannot serve both effectively
• The avatar persona system (FR5: unique voice per persona) already plans for voice variation across personas — expressivity adds user control WITHIN each persona
• Existing ideas 💡 Cloud Voice Provider Pipeline with Spatial Audio Support #56 (Cloud Voice Provider Pipeline) and 💡 ElevenLabs Conversational AI for Expressive, Low-Latency Avatar Speech #213 (ElevenLabs Integration) address voice quality but not user-controlled expressivity

Technical Opportunity

• The TextToSpeech port interface (defined in architecture) is designed for swappable implementations — adding expressivity parameters is a zero-cost architecture decision
• ExpressivityProfile type with normalized floats (pace, warmth, formality) in src/shared/types/
• MVP: Map to Web Speech API's limited controls (rate, pitch) + Rive avatar animation intensity
• Phase 2: Cloud TTS providers (ElevenLabs, Cartesia) expose full expressivity parameter controls
• Rive state machine transitions can mirror expressivity — more animated gestures for casual mode, measured movements for professional mode

Assessment

Dimension	Score	Rationale
Feasibility	high	Zero-cost abstraction design now; limited but functional mapping to Web Speech API for MVP; full unlock in Phase 2
Impact	med	User delight and personalization; competitive parity with Apple Siri AI; but not a core workflow capability
Urgency	low	Apple shipping fall 2026; TalkTerm should design for it now but implementation can follow Phase 2 TTS work

Adversarial Review

Strongest objection: Web Speech API (MVP TTS) does not support expressivity controls. This feature is blocked until Phase 2 cloud TTS integration, making it premature to propose now.

Rebuttal: Designing the TTS abstraction with expressivity parameters is a zero-cost architectural decision that prevents lock-in. MVP can map expressivity to the limited controls Web Speech API does offer (rate, pitch) and to Rive avatar animation intensity. Phase 2 unlocks full expressivity via ElevenLabs or Cartesia. Proposing this now ensures it is designed in, not retrofitted — aligning with TalkTerm's SOLID principles (Open/Closed: new behaviors via new implementations).

Suggested Next Step

Extend the TextToSpeech port interface with an ExpressivityProfile type (pace, warmth, formality as normalized floats). Map Web Speech API rate/pitch to expressivity for MVP. Define corresponding Rive state machine inputs for avatar animation intensity.