💡 Sub-40ms Conversational Voice via Cartesia Sonic: Perceptually Instant Avatar Speech

[github-actions[bot]]

Summary

Integrate Cartesia Sonic 4 Turbo as a premium TTS provider to achieve ~40ms first-byte latency — below the human perception threshold. This transforms avatar conversations from "talking to a computer" into "talking to a person," leapfrogging competitors who offer 200-500ms+ voice latency. No avatar-based AI assistant currently uses sub-100ms TTS.

Market Signal

Cartesia Sonic 4 Turbo achieves ~40ms first-byte TTS latency (Aug 2026), compared to ElevenLabs Flash v2.5 at ~75ms and Web Speech API at 500ms+. The voice AI market is converging on latency as the key differentiator — Apple rebuilt Siri with Gemini (WWDC June 2026) emphasizing responsiveness, and Alexa+ achieved 96% first-attempt success rate through speed. Hume Octave 2 leads emotional fidelity, but Cartesia leads raw latency. No competitor in the avatar-agent space uses sub-100ms TTS yet.

User Signal

The PRD's NFR1b requires first spoken response within 3 seconds — Cartesia could deliver it in under 1 second. The STT/TTS abstraction layer (architecture decision) was designed exactly for provider swaps. Existing ideas #56 (Cloud Voice Pipeline) and #213 (ElevenLabs) explore cloud TTS but neither targets the sub-100ms latency breakthrough. This is complementary: ElevenLabs for voice quality/expressiveness, Cartesia for raw speed, Web Speech API for free tier.

Technical Opportunity

TalkTerm's architecture defines swappable TTS via the TextToSpeech interface in src/shared/types/ports/. Cartesia Sonic is a drop-in provider implementation. Combined with Rive's 120fps avatar animation (proven by Duolingo for lip-synced avatars across 40+ languages), this creates the most natural AI conversation experience available. The barge-in support (FR37) becomes seamless rather than jarring at sub-100ms latencies — the avatar can stop mid-word and respond to interruption naturally.

Assessment

Dimension	Score	Rationale
Feasibility	high	Drop-in TTS provider in existing abstraction layer; Cartesia has a standard REST/WebSocket API
Impact	high	Core experience differentiator — crosses the threshold from "computer" to "person" feel
Urgency	high	Competitors are converging on voice latency; first-mover advantage in avatar-agent space

Adversarial Review

Strongest objection: Web Speech API is free; Cartesia adds per-character API costs that BYOK users must pay. At scale, voice costs could exceed LLM costs.

Rebuttal: The TextToSpeech abstraction layer means Cartesia is opt-in alongside the free Web Speech API default. Power users who value conversational quality will pay for it — same BYOK model as the LLM itself. Cost transparency via existing cost meter ideas (#69, #411) ensures informed choices. The premium experience is the differentiator; the free tier ensures accessibility.

Suggested Next Step

Research Cartesia Sonic 4 Turbo API, document pricing tiers, and prototype a CartesiaTTS implementation of the TextToSpeech interface. Benchmark first-byte latency against Web Speech API and ElevenLabs in the Electron renderer environment.