💡 Local Neural Avatar Voice: Offline Kokoro/Supertonic TTS via the Existing sherpa-onnx Engine

[github-actions[bot]]

Summary

TalkTerm already ships sherpa-onnx-node for local speech-to-text (src/main/speech/sherpa-onnx-stt.ts). The same engine runs open-weight neural TTS models: Kokoro-82M, Supertonic, Piper/VITS. This idea adds a main-process SherpaOnnxTts adapter behind the existing TextToSpeech port, fed sentence by sentence, so the avatar's voice is natural, private, free to run and identical on macOS, Windows and Linux. Today voice quality depends on whatever voices the OS has installed.

Market Signal

• Open-weight TTS has reached frontier quality: several 2026 reports put Kokoro-82M at Install BMad Method v6.2.0 with Claude Code integration #1 on TTS Arena in early 2026, with Supertonic 3 as a faster CPU alternative (offlinetts leaderboard roundup, Kokoro vs Supertonic CPU benchmark). The rankings are disputed, but both are competitive.
• Local speech is mainstream on the desktop: Ollama added speech models to its Mac app (Oct 24, 2026), Liquid AI released LFM3-Audio-2B for laptops (Oct 27), and Granola shipped local Mac dictation (Oct 29) (Voice AI News W44).
• Cloud voice vendors keep churning: Hume's TTS and EVI APIs close on Nov 13, 2026, and OpenAI retires tts-1, tts-1-hd and two gpt-4o-mini-tts snapshots on Jan 6, 2027 (Voice AI News W41). Each cloud dependency is a future migration.
• The engine already supports it: sherpa-onnx publishes a Node addon OfflineTtsKokoroModelConfig (API docs, Kokoro models).

User Signal

• TTS today is src/renderer/speech/web-speech-tts.ts. Voice quality varies by OS and is often robotic on Linux and older Windows builds, which undercuts the "companion, not tool" principle.
• PRD NFR7 restricts audio retention and generally favors privacy. Local synthesis means no assistant text leaves the machine for voice.
• Cloud voice ideas (💡 Sub-40ms Conversational Voice via Cartesia Sonic: Perceptually Instant Avatar Speech #432 Cartesia, 💡 ElevenLabs Conversational AI for Expressive, Low-Latency Avatar Speech #213 ElevenLabs, 💡 Cloud Voice Provider Pipeline with Spatial Audio Support #56, 💡 Instant Voice Persona Cloning: 10-Second Custom Avatar Voices via Cartesia Sonic 3.6 #525 voice cloning) each add a vendor, a second API key and per-character cost. That is extra friction for BYOK non-technical users and a poor default.

Technical Opportunity

• Reuse an existing pattern: sherpa-onnx-stt.ts (main-process adapter with model download and size checks) plus ipc-speech-stt.ts (renderer proxy). A mirrored sherpa-onnx-tts.ts and ipc-speech-tts.ts implements TextToSpeech without changing the port. The architecture rule "only this file imports sherpa-onnx-node" can be kept with one shared engine wrapper.
• Streaming by sentence: the agent's streamed text is split at sentence boundaries, and each sentence is synthesized and queued. Barge-in (barge-in-controller.ts) clears the queue.
• Avatar sync: raw PCM output gives amplitude envelopes, and potentially phoneme timing, to drive Rive mouth shapes. That makes 💡 Phoneme-Driven 2D Avatar Lip Sync: Real-Time Viseme Animation via Rive State Machine #365 (viseme lip-sync) feasible without a cloud vendor.
• Graceful fallback: Web Speech stays the default until the model download finishes, and on hardware that fails a quick synthesis benchmark.

Assessment

Dimension	Score	Rationale
Feasibility	med	Engine and download infrastructure exist. CPU latency on low-end Windows hardware is the open question.
Impact	med	Noticeably better avatar presence and privacy at zero running cost. Unblocks lip-sync.
Urgency	med	No user complaints yet, but vendor churn and competitors' voice quality raise the bar.

Adversarial Review

Strongest objection: CPU latency and size. One community benchmark shows Kokoro ONNX taking about 5.5s for 196 characters on CPU, and the models plus voices add 80–300 MB. That could break NFR1b (first spoken segment within 3s).

Rebuttal: Synthesizing sentence by sentence means only the first short sentence (about 40 characters, roughly 1s at the benchmark's rate) is on the critical path, while later sentences render during playback. Supertonic is about 3x faster on CPU where Kokoro is too slow. The download is optional and deferred, exactly like the STT model today, so first-run setup is not affected. The adapter is also the strategic hedge that makes any cloud voice vendor optional.

Suggested Next Step

Run a 2-day spike: through sherpa-onnx-node, measure first-sentence latency and real-time factor for Kokoro (English v0_19 and multilingual v1.0) and Supertonic on (a) a 4-core Windows laptop from 2021 and (b) an M-series Mac. Then decide the default model, the minimum hardware for local TTS, and whether to bundle or download.