# Native streaming speech-to-text

**Status: SUPERSEDED (2026-09-02).** The Swift/CoreML voice helper this
document designed, and the Python/MLX Parakeet worker it was replacing, are
both gone. SparkQuill now runs the shared AgentWorks engine —
`agent_go/pkg/voicestt` (sherpa-onnx + NVIDIA Nemotron streaming, in-process,
CGO) — through the same `/api/voice/stream` WebSocket and the same frontend
hook (`frontend/shared/voice/useVoiceDictation.ts`) as AgentWorks' composer
and Video Studio, on macOS (Intel and Apple Silicon) and Linux alike. One
engine, one model directory (`~/.agentworks/voice-models/`), one protocol.
WhatsApp voice notes go through the same engine; container decoding
(ogg/opus) is `voicestt.DecodeFile`, using macOS's afconvert or ffmpeg.
Packaging: `scripts/build-darwin-voice-binary.sh` (desktop apps) and
`deploy/aws-ec2/build/build-linux-agent.sh` (EC2) build with cgo and stage
the native libraries beside the binary. What follows is kept as the record
of why the earlier engines were tried and what was measured.

**Date:** 2026-08-02
**Repositories:** `mcp-agent-builder-go` (SparkQuill: `agent_go/cmd/family-server`,
`frontend/learning-app`, `desktop-sparkquill`)
**Related:** `docs/refactor/README.md`

## The problem

Live mic dictation re-transcribes the **entire recording from the start** on
every preview refresh. Cost therefore grows with recording length: a call that
takes ~1.2s a second into talking takes materially longer thirty seconds in,
and the preview drifts further behind the speaker the longer they talk. The
final transcription on stop pays the same full-clip cost again.

This is architectural, not a tuning problem. `LIVE_PREVIEW_INTERVAL_MS` in
`useMicDictation.ts` is already self-rescheduling (it waits for each call to
finish rather than firing on a fixed clock), so there is no polling interval
left to shorten — the floor is the transcription call itself.

## What exists today

- **Capture:** browser `MediaRecorder` (webm/mp4 container), chunked at 1s via
  `rec.start(1000)`.
- **Transport:** each preview posts the whole accumulated blob to
  `POST /api/voice/transcribe` (multipart) — `voice_transcribe_api.go`.
- **Inference:** `voice_worker.go` supervises a persistent Python process
  (`voice_worker.py`, `//go:embed`-ed and written to `mlxVoiceDir()`), one JSON
  request per line over stdin/stdout, models kept warm for
  `voiceWorkerIdleTimeout` (15 min).
- **Model:** `mlx-community/parakeet-tdt-0.6b-v2` via `mlx_audio.stt`, called
  through `generate_transcription` — a one-shot batch call.
- **Install:** `voice_mlx_env.go` builds a ~3.1GB Python venv on first use.

### Why the current stack cannot stream (verified, not assumed)

Inspected the installed package directly in
`/Users/mipl/.sunlit-learning/mlx-voice/.venv`:

- `ParakeetTDT.stream_generate(...)` **takes a complete recording** (a path or a
  full `mx.array`) and chunks it internally, yielding progressive results. It is
  streaming *output over a finished file*, not streaming *input*. Useless for a
  live microphone.
- `ParakeetTDT.decode_chunk(audio_data)` is **stateless** — its whole body is
  `log_mel_spectrogram` then `self.decode(mel)[0]`. No decoder state is carried
  between calls, so consecutive chunks cannot be decoded as one continuous
  utterance.

There is no stateful incremental API in `mlx_audio` to migrate onto.

## Target architecture

Replace the Python/MLX voice worker with a native Swift helper built on
[FluidAudio](https://github.com/FluidInference/FluidAudio) (**Apache-2.0**),
whose `StreamingEouAsrManager` + `EncoderCacheManager` keep encoder/decoder
state across appends — the mechanism that makes genuinely incremental
transcription possible.

Licensing note: FluidAudio is Apache-2.0 and is the only dependency taken. The
FluidVoice *application* that popularised it is GPLv3; none of its code is used
or referenced here.

```
AudioWorklet (raw PCM 16k mono)
      │  POST /api/voice/stream/chunk
      ▼
family-server  ──stdin (JSON + base64 PCM)──▶  voice-helper (Swift)
      ▲                                              │
      └──────── partial in response ◀── JSON lines ──┘
```

(The original sketch here used a WebSocket. It was replaced during
implementation: the audio only crosses loopback to a server on the same
machine at ~6 chunks/second, so a socket bought nothing and would have added
framing, a dependency, and its own backpressure failure modes.)

1. **`desktop-sparkquill/voice-helper/`** — SwiftPM executable depending on
   FluidAudio. Reads JSON lines carrying base64 PCM on stdin, writes JSON
   lines on stdout. The line protocol deliberately mirrors `voice_worker.py`'s shape so
   `voice_worker.go`'s supervision, warm-timeout, and teardown logic carry over
   rather than being rewritten.
2. **Frontend** — replace the `MediaRecorder` preview path with an
   `AudioWorklet` producing raw 16kHz mono PCM. This also removes the container
   problem that makes incremental decode impossible today: chunks after the
   first carry no container header, which is exactly why the current code has to
   resend the whole blob every time.
3. **Server** — three POST endpoints (`start`/`chunk`/`finish`) in
   `voice_stream_api.go`, each forwarding to the helper and returning its
   reply. Every one stays curl-debuggable, like the existing voice endpoints.
4. **CI** — `swift build -c release --arch arm64` in
   `.github/workflows/sparkquill-desktop.yml`, binary staged into
   `extraResources` beside `family-server`. Swift 6.3.3 is present on the
   `macos-15` runner and locally.

### Secondary prize: deleting the Python voice env

If WhatsApp voice-note transcription moves to the same helper, the entire
Python/MLX voice stack can be removed:

- the ~3.1GB venv build in `voice_mlx_env.go` and its install/remove UI,
- `voice_worker.py` and its stdin/stdout JSON protocol,
- the `mx.clear_cache()` unbounded-cache workaround (added 2026-08-02 after
  MLX's cache — unbounded by default, on unified memory shared with the whole
  machine — was identified as a real leak in a process that stays warm 15
  minutes and is hit every ~1.2s during dictation),
- two independent Parakeet installs collapsing into one.

This is the strongest argument for the change and should be treated as part of
it, not a follow-up: leaving both stacks in place is strictly worse than either
alone.

## Measured: the helper works, and latency is not the open question

The Swift helper builds and runs. Verified end-to-end against real speech
(8.8s of `say`-generated audio, fed in 160ms chunks — the shape the live mic
path will use):

| | Native helper | Current Python/MLX |
|---|---|---|
| Per preview refresh | **18–25ms** | ~1.2s and rising with length |
| Trend over a recording | **falls** (25ms → 18ms) | grows |
| Final transcription on stop | **~10ms** | ~1.2–2.4s (full re-transcribe) |
| First call after load | 3.6s (one-time JIT) | — |
| First-run model download + load | 107s | ~3.1GB venv build |

The core claim holds and then some: per-chunk cost is flat-to-falling in
recording length, and `finish()` is effectively free because it flushes live
state instead of re-decoding. Text appears while the speaker is still talking.

### Word accuracy is fine — two earlier "defects" were a bad test

The first run appeared to drop a leading word ("The quick brown fox…" → "quick")
and to garble "photosynthesis" into "photosynthesythesis". **Both were artifacts
of the test audio, not the engine.** That clip began speaking at sample 0; real
microphone input always has a moment of quiet first. Re-run with 0.5s of lead-in
silence, the same model returns:

> the quick brown fox jumps over the lazy dog photosynthesis is how plants make
> their own food using sunlight water and carbon dioxide

Correct throughout, including "jumps" and "photosynthesis". Recorded here
because the original claim is wrong and would otherwise have argued against a
sound approach. Any future test must include lead-in silence.

### The one real gap: no punctuation

Confirmed across **all three** streaming variants — `.ms160`, `.ms320`,
`.ms1280` — on the same padded audio: byte-identical output, no punctuation, no
capitalization. This is a property of FluidAudio's streaming EOU Parakeet
models, not a chunk-size tradeoff, so there is nothing to tune here.
`Documentation/ASR/PostProcessing.md` does not address it either: that is
Inverse Text Normalization (numbers, dates, currency), not punctuation.

### Resolved: two-stage, and it is strictly better than today

Punctuation is **solved**, not traded away. The helper now runs two models:
`StreamingEouAsrManager` for the live preview, and `UnifiedAsrManager`
(parakeet-tdt-0.6b-v2, the same family as the MLX checkpoint in use today) for
the committed text. Measured on the same padded clip:

| | Output |
|---|---|
| Live preview (streaming) | `the quick brown fox jumps over the lazy dog photosynthesis is how plants…` |
| **Committed (batch, 102ms)** | `The quick brown fox jumps over the lazy dog. Photosynthesis is how plants make their own food using sunlight, water and carbon dioxide` |

Capitalization, sentence-final period, and an interior comma — all present. So
the committed message is **no worse than today's text and ~12–24x faster**
(102ms against 1.2–2.4s), while the preview is effectively instant. There is no
remaining quality argument against this path; the earlier "not viable for the
composer" conclusion is withdrawn.

Unpunctuated live text is fine because the preview is explicitly allowed to
revise itself, and this shape matches a product decision already taken
independently: the final transcription should always be a full accurate pass
rather than a reused preview.

Note the API in `Documentation/ASR/GettingStarted.md` is stale against v0.15.5
(`AsrManager.initialize`/`transcribe(_:source:)` do not exist). `UnifiedAsrManager`
— `loadModels()` then `transcribe([Float]) -> String` — is the current surface.

A cold-start cliff remains: the first `audio` call after load took 3.6s (JIT),
so the helper needs the same pre-warm the Python path already does via
`/api/voice/warm`. Subsequent loads with weights cached took 0.6s.

## Risks and open questions

- **First-run model download.** FluidAudio fetches its own CoreML Parakeet
  weights from HuggingFace, separate from the MLX checkpoints already on disk.
  The download UX (progress, failure, retry) needs to match what
  `voice_models.go` does today, and the migration must not leave users holding
  both model sets.
- **WebSocket backpressure.** Audio is produced in real time; if the helper
  stalls, frames must be dropped rather than queued without bound.
- **Platform.** macOS/Apple Silicon only — already true of the current voice
  tier (`voice_hardware.go` gates on architecture), so not a regression.
- **CI cost.** Adds a Swift build to a workflow that already builds Go and the
  frontend.
- **Accuracy parity is unproven.** FluidAudio's streaming Parakeet is a
  different model variant from `parakeet-tdt-0.6b-v2`. Parity must be measured
  on real family audio before the Python path is deleted, not assumed — the
  removal above is contingent on that check.

## Sequencing

The Python path stays fully working until the Swift path is verified.

1. ~~Swift helper + protocol, exercised standalone.~~ Done; measured above.
2. ~~`voice_worker.go` able to drive it.~~ Done — `voice_native.go` reuses
   `voiceWorker` with a different launcher, covered by an opt-in integration
   test that drives the real process (`SPARKQUILL_VOICE_STREAM_TEST=1`).
3. ~~Frontend capture on the new path.~~ Done — `nativePcm.ts` (AudioWorklet →
   raw PCM) and a branch in `useMicDictation`. Three POSTs rather than a
   WebSocket; see `voice_stream_api.go` for why.
4. **Next: use it with a real microphone.** Nothing below this line has been
   exercised in the running app.
5. Then: accuracy/latency comparison against the Python path on family audio.
6. Only then: WhatsApp migration and Python removal.

### Outcome: the streaming EOU model was abandoned for the batch model

Two real-microphone tests killed the two-model design, and the second one was
decisive. Both failures are inherent to `StreamingEouAsrManager`, not tuning:

1. **Frozen at pauses** (below) — it is a turn-taking model, so a mid-sentence
   pause ended transcription.
2. **Nothing at all for short speech** — it needs ~2s of audio before emitting
   its first token, so dictating a single word produced no preview whatsoever.

The fix was to stop using it. The batch model (`UnifiedAsrManager`) runs ~120x
realtime, so simply re-transcribing everything said so far is cheap enough to
drive the preview: ~60ms at five seconds of speech. Measured with real-time
chunk pacing:

| | Preview behaviour |
|---|---|
| Single word ("Photosynthesis") | text at 1.12s, correct by 1.60s |
| Sentence with a 3s mid-sentence pause | builds continuously through the pause |

The preview is now **punctuated and identical to the final text**, so stopping
no longer rewrites what the user was reading — and one model ships instead of
two.

The cost profile is the honest tradeoff: it grows with recording length, where
a true streaming decoder's would not. `previewInterval` scales the gap between
passes with length to keep each pass a small fraction of the interval.

Note both bad findings came from tests that were unrealistic in the same way —
first no lead-in silence, then no real-time pacing (feeding a clip as fast as
the pipe allows runs exactly one preview pass, making a working preview look
dead). **Voice tests must pace chunks in real time and include a pause.**

### The pause failure, in detail

The first live test looked like "streaming is wrong". Per-chunk logging
(`SPARKQUILL_VOICE_DEBUG`) made it unambiguous: the transport was perfect —
`samples=2560` every chunk, ~40ms each, 125 chunks, no errors, real audio
(rms to 0.085) — but the partial froze on one word at chunk #23 and never
changed through chunk #125, including plenty of loud speech.

The freeze began right after ~2s of silence. `StreamingEouAsrManager` is built
for voice-assistant **turn-taking**: it detects End-of-Utterance after
sustained quiet and expects a reset for the next turn. Dictation is not
turn-taking — someone pausing mid-sentence is still dictating — so past the
first pause the engine simply stopped emitting tokens. The synthetic test
missed this entirely because that clip had no pauses; **any future voice test
must include a mid-sentence pause.**

The helper now harvests the transcript whenever `eouDetected` fires, resets the
engine, and reports finalized segments plus the in-flight partial as one
running preview. Verified against speech with a deliberate 3s mid-sentence
pause: both halves survive, where previously the second was lost.

### Known-unverified, in likely-to-bite order

- ~~**Chunk upload keeping up with speech.**~~ Measured through the running
  app (62 chunks of real speech over the HTTP endpoints): **28.6ms average,
  231.8ms worst case** against 160ms of audio per chunk — ~5.6x headroom, and
  the one outlier is the first-call JIT. The queue drains comfortably. Final
  committed text came back punctuated in 193ms.
- **`AudioContext({sampleRate: 16000})`.** Chromium honours it; if a device
  refuses, the worklet would emit at another rate and the helper would receive
  mis-timed audio. No resampling guard exists yet.
- **First-run download inside the app.** `/stream/start` blocks while weights
  download (~96s measured). The UI shows "Listening" throughout, with no
  progress — poor, though only once per machine.