feat(gnani): update TTS voices/models and STT sample rates + streaming toggle#6468
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…s.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
- tts.py: _strip_wav_header now strips a fixed-size header and preserves the trailing PCM when no `data` sub-chunk is found, instead of dropping the whole chunk (the fallback branch was dead, returning b"" either way). - tts.py: move finalize/flush/end_segment out of the SynthesizeStream `finally` block to the success path only. Flushing the coalescer's sub-frame remainder on error set pushed_duration > 0.0 and suppressed otherwise-retryable transient WebSocket failures. - utils.py: correct the websockets version threshold for the header kwarg. The top-level `websockets.connect` only switched to the asyncio impl (additional_headers) in 14.0; 12.x and 13.x use the legacy client (extra_headers). Verified against 12.0/13.1/14.0/15.0.1. - tests: update gnani STT/TTS tests for STREAM_SUPPORTED_SAMPLE_RATES (44100/48000 now accepted), the new voice/model defaults (Pranav / timbre-v2.0), and server-side voice validation (arbitrary voices accepted).
… WAV stripping Switch the SSE, WebSocket-chunked, and WebSocket streaming paths to request the configured container (WAV by default) and push bytes straight to the AudioEmitter, whose built-in AudioStreamDecoder decodes and re-frames them. Removes _strip_wav_header, the PCM aligner/coalescer, and the forced-raw streaming payload override.
chenghao-mou
left a comment
chenghao-mou
left a comment
There was a problem hiding this comment.
Two comments. Otherwise LGTM. Tested locally with both STT and TTS.
| (used by the STT/TTS clients) only switched to the asyncio implementation in | ||
| websockets 14.0 — in 12.x and 13.x it is still the legacy client. Select the | ||
| kwarg by that boundary so WebSocket STT/TTS works across ``websockets>=12``. | ||
| """ |
There was a problem hiding this comment.
Q: if we enforce >=14.0 in this plugin, we then don't need this file, right?
There was a problem hiding this comment.
great callout, will bump and remove
Bump websockets to >=14, which always uses the asyncio client (headers via additional_headers). Pass it inline at the three connect() sites and delete utils.ws_header_kwargs, the shim that only existed for the extra_headers kwarg on websockets 12.x/13.x. Also drop mulaw from GnaniTTSContainers (and its docstring/README mentions): it has no libav container mapping, so it would not decode through the AudioEmitter's AudioStreamDecoder anyway.
| if audio_b64: | ||
| output_emitter.push(base64.b64decode(audio_b64)) | ||
| break | ||
|
|
||
| audio_b64 = payload.get("audio", "") | ||
| if audio_b64: | ||
| output_emitter.push(base64.b64decode(audio_b64)) |
There was a problem hiding this comment.
🔴 Streamed speech may come out garbled or cut short
Each streamed audio chunk is now handed straight to the audio player (output_emitter.push(...) at livekit-plugins/livekit-plugins-gnani/livekit/plugins/gnani/tts.py:389) without first removing its embedded file header, so the player treats every chunk's header bytes as sound.
Impact: Users can hear distorted noise at chunk boundaries or only the first fragment of speech whenever SSE or WebSocket TTS streaming is used.
Regression from removing per-chunk WAV header stripping
The removed helper _strip_wav_header and the deleted docstrings explicitly documented the streaming contract: "Each SSE chunk decodes to a complete WAV file" and "Each received chunk's WAV header is stripped; only raw PCM is emitted." The default container is wav (GnaniTTSOptions.container = "wav"), and the request payload keeps sending container in audio_config via _build_payload, so the server continues to return per-chunk WAV-wrapped audio.
With the change, the bytes are pushed unmodified to tts.AudioEmitter, which (for a non-audio/pcm mime type) routes them through a single codecs.AudioStreamDecoder created once with format="wav" (livekit-agents/livekit/agents/tts/tts.py:1333-1342). A single WAV/RIFF demuxer expects exactly one file: after decoding the first chunk's declared data it will either stop (truncating all later chunks) or misinterpret each subsequent chunk's RIFF header bytes as PCM samples (audible corruption).
The same removal affects all three streaming paths: SSE (tts.py:389, tts.py:394), WebSocketChunkedStream (tts.py:460, tts.py:470, tts.py:477), and SynthesizeStream (tts.py:566, tts.py:576, tts.py:583). Only the REST path (single full WAV) remains safe.
If the streaming endpoints have genuinely changed to emit one continuous WAV header followed by raw PCM (rather than a complete WAV per chunk), the change would be correct — but that contradicts the prior documented behavior and is not mentioned in the PR intent, so it needs verification.
Prompt for agents
The PR removed the _strip_wav_header helper and now pushes each streaming audio chunk directly into tts.AudioEmitter with the container mime type (audio/wav by default). Previously the code documented that each SSE/WebSocket chunk is a complete WAV file and stripped the per-chunk RIFF/WAV header, emitting only raw PCM with mime audio/pcm. The AudioEmitter, for a non-audio/pcm mime type, feeds all pushed bytes through a single codecs.AudioStreamDecoder created once with the WAV format; a single WAV demuxer cannot decode multiple concatenated WAV files, so later chunks are either truncated or corrupted. Verify the actual wire format of the Gnani /api/v1/tts/sse and wss /api/v1/tts endpoints for the timbre models. If each chunk is still a complete WAV file, restore per-chunk header stripping (and keep emitting mime audio/pcm), or alternatively decode each chunk independently. This affects SSEChunkedStream._run, WebSocketChunkedStream._run, and SynthesizeStream._run (all the output_emitter.push sites), while the REST path is unaffected because it returns a single WAV.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Verified — not an issue. For audio/wav, AudioStreamDecoder uses _WavInlineDecoder (not PyAV), a state machine that resets on each new RIFF header, so concatenated complete-WAV chunks decode correctly (decoder.py:180-187). I drove the real AudioEmitter as the plugin does — 8 complete WAV files, one per push — and reconstruction is byte-exact in both modes. So per-chunk stripping / audio/pcm isn't needed.
2 participants
No description provided.