feat(smallestai): use Waves continuous WebSocket streaming protocol for TTS#6363
Conversation
…or TTS The SynthesizeStream previously buffered all tokens locally and sent one isolated request per segment, so synthesis could not begin until the full segment text was assembled — adding TTFB latency and ignoring the Waves continuous-streaming protocol that every comparable LiveKit TTS plugin uses. Stream tokens to /waves/v1/tts/live as they arrive with `continue: true`, close each segment with an explicit `flush: true`, and expose the server-side `max_buffer_flush_ms` buffer bound as a constructor/update_options option. Send and receive run as concurrent tasks over the pooled connection; the connection is only returned to the pool after a clean `complete` (the pool discards it on error/interruption), keeping reuse safe. One SynthesizeStream instance now maps to one segment, matching the current base-class contract (extra segments per instance are deprecated upstream). Verified against the live Waves endpoint: it accepts the continue/flush protocol and returns audio.
_send_task sent a flush frame unconditionally and returned None, so the `sent_any` guard in _run was always falsy and _recv_task was cancelled before receiving audio. Return whether any non-whitespace text was sent and only flush when it was: an empty segment now finishes promptly instead of waiting on a `complete` the server never emits.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
ProblemThe Waves WS API supports incremental streaming — continue / flush / max_buffer_flush_ms — but livekit-plugins-smallestai (as of 1.6.4) doesn't use any of it. Its SynthesizeStream buffers every token locally until a flush, then sends one isolated request per segment with none of the streaming fields. Change
@tinalenguyen Requesting a review whenever you get a chance |
| Uses the Waves continuous-streaming protocol: text tokens are forwarded to the | ||
| server as they arrive (``continue: true``) rather than buffered locally, and an | ||
| explicit ``flush: true`` message closes each segment. This lets synthesis begin | ||
| before the full segment text is known, lowering time-to-first-byte. |
There was a problem hiding this comment.
The docstring says this lowers time-to-first-byte, but the scope note under PR description says output timing is unchanged since the server synthesizes on the explicit flush — and the only flush is at end-of-segment, so TTFB should be the same as before.
Would sentence-level flushing be in scope? other providers e.g elevenlabs,cartesia run a tokenize.blingfire.SentenceTokenizer inside SynthesizeStream and flush per sentence instead of per segment. Since Waves already synthesizes on flush, that would let audio start after sentence one rather than after the full LLM response — the TTFB win the docstring describes, with no server-side change needed.
3 participants
Summary
The SmallestAI
SynthesizeStreamcurrently buffers all incoming tokens locally and sends one isolated request per segment, without using the Waves continuous-streaming protocol (continue/flush/max_buffer_flush_ms) that comparable LiveKit TTS plugins (ElevenLabs, Cartesia, Rime, Neuphonic, Inworld, …) use.This PR switches the plugin to the continuous protocol so it speaks the Waves streaming API the way the official SDK and other providers' plugins do:
/waves/v1/tts/liveas they arrive with"continue": true, instead of buffering the whole segment locally."flush": trueframe.max_buffer_flush_msbuffer bound as aTTS(...)constructor arg andupdate_options(...)option (default0).complete; the pool discards it on error/interruption, so reuse stays safe.SynthesizeStreaminstance = one segment, matching the current base-class contract (multi-segment instances are deprecated upstream).Response-frame handling (
chunk/word_timestamp/complete/error) is unchanged, and the HTTPChunkedStreampath is untouched.Scope / note
This change is about protocol parity — using the continuous-streaming fields the Waves WS API exposes, matching the official SDK and other plugins. It is not a latency change: with current server behavior the endpoint synthesizes on the explicit
flush, so output timing is unchanged from the previous buffer-until-flush implementation. The client is now ready to benefit from any future server-side incremental emission with no further plugin changes.Testing
Verified against the live Waves
/tts/liveendpoint:continue/flush/max_buffer_flush_msare accepted and return audio (lightning_v3.1/magnus).lightning_v3.1_pro/meher,word_timestamps=True), the endpoint returnsword_timestampframes — all words surfaced asTimedStringtranscript entries with sensible timings alongside audio. Theword_timestampsflag on the streamed frames is accepted (no "Invalid input data").synthesize()(ChunkedStream) is unchanged and still returns audio.word_timestampsflag — kept local for now, not included in this PR.ruff format,ruff check,mypyall clean.