[voice] Voice pipeline for the LifeOps MVP: cloud vs on-device STT/TTS, and proving it end-to-end

[lalalune]

This is the coordination post for the voice workstream of the LifeOps Personal Assistant MVP (GitHub project 15). Voice is a first-class input/output for our audience — hands-busy reminders, users who won't or can't type (children, elderly, atypical speech). So bidirectional voice — STT in, TTS out — must work and be proven to work on every MVP surface.

Full research doc (read this first): packages/docs/ongoing-development/research/07-voice-pipeline.md

The decision, in one line

The architecture is already right. On-device Kokoro TTS + fused ASR / OS recognizers wherever a native runtime exists; Railway-backed cloud voice (Kokoro TTS + Whisper STT behind the Cloud API) for web and unprovisioned devices. The MVP work is measurement, wiring, and e2e proof — not new voice infrastructure. Prefer deleting/simplifying over adding.

What is settled (do not re-litigate)

• On-device Whisper stays retired. The prompt asked whether to ship Kokoro + whisper-small onto devices. Kokoro-on-device already ships everywhere (desktop fused, Android in-process, iOS CoreML). On-device ASR is fused eliza-1-asr (desktop, measured winner) and OS recognizers (mobile, free, no download — already the actual TalkMode behavior). Re-adding whisper.cpp would be a second ASR runtime for zero MVP benefit (transcriber.ts:758).
• A browser tab has no on-device runtime. Web needs the servers. The only open web question is quality/latency, which the benchmark settles.
• The free path stays first. The Kokoro/Whisper branches bypass billing entirely — MVP default voice costs nothing per request. ElevenLabs stays opt-in. No change may demote the free branch.
• Reuse the existing harnesses. voice:workbench, voice:matrix, voice:latency-report, the Playwright voice-realaudio.spec.ts, and the gated Railway contract test already exist. No new bespoke voice harness.

Per-platform default matrix (MVP)

Platform	STT	TTS
Web (any runtime mode)	Cloud Whisper via /api/v1/voice/stt (wire it — today it's browser SpeechRecognition)	Cloud Kokoro if TTFB benchmark ≤ ~1.5 s for a short sentence, else keep edge
Desktop, local/provisioned	fused eliza-1-asr (WER 0.008 / RTF 0.262)	local Kokoro
Desktop, cloud/unprovisioned	cloud Whisper (same wiring as web)	cloud Kokoro / edge
iOS	SFSpeechRecognizer via TalkMode (WER 0 / RTF 0.168)	on-device Kokoro (CoreML)
Android	SpeechRecognizer via TalkMode	on-device fused Kokoro

What is actually broken (why these issues exist)

Cloud voice has zero latency numbers, zero CI, an English-only tiny STT model (faster-whisper-tiny.en, hardcoded), an unwired client capture default (web "eliza-cloud" ASR silently falls back to the browser recognizer and never posts audio to Railway Whisper), unreproducible deploys (no in-repo service definition for either Railway service), and no inline-postable round-trip evidence. On-device voice is measured on desktop/Apple, unmeasured steady-state on Android (a follow-up, not an MVP gate).

Issues (claim these)

Ordered, most important first:

1. [voice] Benchmark Railway cloud voice vs on-device: TTFB, RTT, WER, sizes — publish the decision table (P0) — the only deliverable that settles the web TTS default and validates the cloud path at all. Reuse the live-test request shapes; measure TTFB / total / RTF / WER (tiny vs small) / sizes; publish the table into the research doc.
2. [voice] Bidirectional voice e2e with real audio round-trip: web (Railway path) + desktop (local path), incl. failure paths (P0) — one lane proving mic-audio → STT → live agent → TTS → audible reply on web (Railway) and desktop (local), plus mic-denied / silence / network-drop-mid-stream.
3. [voice] Wire web eliza-cloud ASR to actually POST audio to cloud STT (today it silently falls back to browser SpeechRecognition) (P1) — capture WAV → cloud STT proxy; browser SpeechRecognition demoted to interim-only. Fail loud, no silent downgrade.
4. [voice] Cloud STT model is hardcoded English-only faster-whisper-tiny.en — make it deploy-configurable and fix the misleading model log (P1) — WHISPER_STT_MODEL env, benchmark-chosen default (candidate: faster-whisper-small), kill the dead gpt-5-mini-transcribe log on this path.
5. [voice] Railway Kokoro/Whisper services: commit in-repo service definitions + a scheduled live contract lane (P1) — Dockerfile/railway.json + deploy doc following the packages/feed precedent; scheduled workflow runs the gated contract test so a dead service pages red instead of surfacing as a user report.
6. [voice] Cloud Kokoro time-to-audio-start: extend the first-line cache to the Kokoro branch (conditional on benchmark) (P2) — only if issue 1 shows short-sentence TTFB above threshold; reuse the existing provider-keyed cache, no streaming work. Close-as-unneeded is a valid outcome.
7. [voice] Hygiene: remove the stale whisper.cpp submodule entry and fix provider-defaults vs capture-factory doc drift (P2) — drop the dead .gitmodules whisper.cpp entry; align voice-provider-defaults.ts prose with what actually runs on mobile (TalkMode OS recognizer).

How to claim work

1. Comment on the issue to claim it (say which surface/OS you'll capture on).
2. Move the card on GitHub project 15 to In Progress.
3. Branch feat|fix/<slug>, open a PR against develop. Sync (git fetch origin && git rebase origin/develop) before opening/updating; bun run verify before "done".
4. Keep scope minimal — this workstream turns what we already have into an MVP by fixing/testing/verifying. Do not add voices, streaming synthesis, wake-word, or a second ASR runtime.

Note the dependency order: issue 1 (benchmark) feeds the default in issue 3, the model choice in issue 4, and the go/no-go for issue 6. Issue 2's live web variant is easiest once issue 3 lands, but can drive the raw route in the interim.

Evidence rules — voice needs real audio, posted inline

The .github/issue-evidence committed-file approach is retired. Post evidence inline in the GitHub issue/PR. MP4 renders inline; wrap audio (bare WAV/M4A does not render) in an MP4 so the reviewer can hear it. Use JPG for screenshots (over PNG). A reviewer must confirm it works without reading the code.

For every voice PR, attach — or mark N/A - <reason>, never blank:

• MP4 with audio of the real round-trip: user speaks → transcript appears → agent speaks back. For benchmark/contract issues, wrap one synthesized round-trip WAV in MP4 as audible proof the measured audio is real speech, not a truncated buffer.
• JPG before + after screenshots. For a bug, include the WRONG behavior (e.g. browser-engine transcript / no-STT browser before wiring cloud ASR; garbled .en output for a non-English clip). Failure-path issues need a JPG of each distinguishable error state (mic denied, silence auto-stop, mid-stream drop) — three-state rule, never healthy-empty from a catch.
• Real logs, both sides. Backend [Voice STT API] / [Voice TTS API] lines (e.g. Kokoro stream started in Xms) for the same request; browser console + network log showing the actual /api/asr/cloud or /api/tts/cloud POST.
• Real numbers, no mocks. Every benchmark row comes from a real Railway service or a cited committed artifact (STT_SELECTION.md, voice:latency-report) — no fabricated latencies, no green skips on unreachable services.
• Live-LLM trajectory for the agent turn inside any round-trip loop (the real model, not the proxy) — captured and read.
• Device capture where the change is native (iOS/Android): screenshot + recording on the real simulator/device, not mocked-bridge Chromium.

Docs/build-config-only changes (issue 7) may mark MP4/screenshots N/A with a stated reason and instead post grep output proving zero residual references plus a clean bun install / git submodule status log.

Filed issues

• [voice] Benchmark Railway cloud voice vs on-device: TTFB, RTT, WER, sizes — publish the decision table #14370 — P0 [voice] Benchmark Railway cloud voice vs on-device: TTFB, RTT, WER, sizes — publish the decision table
• [voice] Bidirectional voice e2e with real audio round-trip: web (Railway path) + desktop (local path), incl. failure paths #14371 — P0 [voice] Bidirectional voice e2e with real audio round-trip: web (Railway path) + desktop (local path), incl. failure paths
• [voice] Wire web eliza-cloud ASR to actually POST audio to cloud STT (today it silently falls back to browser SpeechRecognition) #14372 — P1 [voice] Wire web eliza-cloud ASR to actually POST audio to cloud STT (today it silently falls back to browser SpeechRecognition)
• [voice] Cloud STT model is hardcoded English-only faster-whisper-tiny.en — make it deploy-configurable and fix the misleading model log #14373 — P1 [voice] Cloud STT model is hardcoded English-only faster-whisper-tiny.en — make it deploy-configurable and fix the misleading model log
• [voice] Railway Kokoro/Whisper services: commit in-repo service definitions + a scheduled live contract lane #14374 — P1 [voice] Railway Kokoro/Whisper services: commit in-repo service definitions + a scheduled live contract lane
• [voice] Cloud Kokoro time-to-audio-start: extend the first-line cache to the Kokoro branch (conditional on benchmark) #14375 — P2 [voice] Cloud Kokoro time-to-audio-start: extend the first-line cache to the Kokoro branch (conditional on benchmark)
• [voice] Hygiene: remove the stale whisper.cpp submodule entry and fix provider-defaults vs capture-factory doc drift #14376 — P2 [voice] Hygiene: remove the stale whisper.cpp submodule entry and fix provider-defaults vs capture-factory doc drift

---

Design doc · MVP board