feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support, and language_options for Whisper STT by A-K-Erol · Pull Request #6700 · livekit/agents

A-K-Erol · 2026-08-04T22:25:14Z

Summary

Baseten hosts Qwen3-ASR Streaming and Qwen3-TTS alongside the Whisper and Orpheus models this plugin already supports. They speak different wire protocols, so the existing STT/TTS classes can't reach them — pointing either at a Qwen3 endpoint connects and then produces nothing.

This adds Qwen3STT and Qwen3TTS as separate classes. The existing Orpheus/Whisper paths are untouched, so this is non-breaking.

	`STT` / `TTS`	`Qwen3STT` / `Qwen3TTS`
STT audio	raw binary PCM	base64 `input_audio_buffer.append`
STT results	`message_type` / `transcript`	`type: "transcription"` / `segments[].text`
TTS transport	`{prompt, voice, …}`, or WS + `__END__` sentinel	`session.config` → `input.text` → `input.done`
TTS voices	preset names (`tara`)	registered voice clones

session = AgentSession(
    stt=baseten.Qwen3STT(model_id="your-qwen3-asr-model-id"),
    tts=baseten.Qwen3TTS(model_id="your-qwen3-tts-model-id", voice="your-voice"),
)

Both accept model_endpoint, model_id, or chain_id with the same precedence as STT (extracted into _endpoint.py).

Notes on the design

A few protocol details drove decisions that aren't obvious from the diff:

input.done is a flush, not a close. The session config stays in effect, so Qwen3TTS keeps one warm socket across turns. Re-dialing per utterance would add a connect plus a config round trip to every agent response.
Parked sockets need an application-level keepalive. The server has a 30s idle timeout that protocol pings don't reset, so an idle socket reads as OPEN long after the server has given up. An empty input.done answers session.done with zero sentences and proves the session is alive.
Interrupted sockets are discarded, not parked. Closing the socket is what stops in-flight generation; a graceful session.close would keep the GPU busy producing audio nobody hears.
One emitter segment per SynthesizeStream. push_text() after a flush is dropped by the framework and _main_task raises on a segment-count mismatch, so a mid-stream flush means "synthesize what's buffered", never "start a new segment".
Qwen3-ASR reports a language name ("English"), so Qwen3STT maps the common ones to ISO codes rather than passing a name where a code is expected.

Voices

Qwen3-TTS Base ships no built-in speakers — there's no tara equivalent. voice names a registered clone, and register_voice/list_voices are exported to manage them. Worth knowing: the server stores uploaded voices on the container's local disk, so a runtime-registered voice lives on one replica and is lost on restart. The README documents baking the reference into the deployment instead, or passing ref_audio/ref_text to clone inline per session.

Also: `language_options` for the existing Whisper `STT`

Bundled here because it is the same plugin and came out of the same customer
conversation. Baseten's streaming transcription API has accepted a
language_options list since Whisper runtime v0.5.0, which scopes detection to
the languages an agent actually supports. The plugin only ever sent a single
audio_language, forcing a choice between a fixed tag that mistranscribes the
other language and auto, which detects across all 99 and is unreliable on the
one- to two-second utterances typical of telephony.

stt = baseten.STT(model_id="...", language="auto", language_options=["en", "de"])

Only added to the handshake when non-empty — StreamingWhisperInput uses
extra="forbid", so sending it unconditionally would break anyone on an older
runtime. Also wired through update_options on both STT and SpeechStream.

Verified against a live Whisper Large V3 Turbo streaming deployment, with a
negative control: language_options: ["en", "de"] is accepted and transcribes
normally, while a deliberately misspelled field name closes the socket with
1011 — so acceptance confirms the field name rather than showing it was
silently ignored.

Testing

Developed against livekit-agents 1.6.8 with a mock server implementing the Qwen3 protocols, driven through the real framework machinery (AudioEmitter, RecognizeStream, the retry loop) and through a real AgentSession with a real-time audio sink. Covered:

TTS: token-by-token push_text, socket reuse without config resend, keepalive on a parked socket, barge-in discarding the socket, transient error retried / persistent error propagating, word timestamps rebased across sentence boundaries
STT: handshake shape, start_of_speech → interim → final → end_of_speech, one-shot recognize(), partials disabled via interim_results=False
AgentSession: TTS audio out with reuse across turns and interrupt() mid-playout; STT mic audio in surfacing interim + final user turns over consecutive VAD-bounded turns

I'm happy to contribute those as pytest suites if you'd like them in-tree — I left them out to keep the diff focused and avoid adding scripts your CI would try to collect.

Live validation

Since opening this, both adapters have been run end to end against real Baseten
deployments of the same model-registry trusses they target (Qwen3-ASR streaming
on RTX Pro 6000, Qwen3-TTS Base on RTX Pro 6000), using real speech with ground
truth rather than synthetic audio.

STT — a mu-bench en-US utterance, streamed at 100ms frames:

partial: I want to get a higher limit on my
FINAL:   Hi, I want to get a higher limit on my credit card. | lang: en
truth:   Hi. I want to get a higher limit on my credit card.

100% word overlap, both through Qwen3STT.stream() directly and through a real
AgentSession (user_input_transcribed, 6 interims + 1 final). Confirms the
handshake, the base64 append frames, type: "transcription" parsing, the
language_code: "English" -> en mapping, and clean termination on the
commit-triggered final.

TTS — voice.list returns {"voices": [], ...} on the Base checkpoint,
confirming it ships no built-in speakers; voice.add cloning from a 14s
reference works; synthesis returns real 24kHz PCM; the second turn reuses the
warm socket (TTFA 1081ms vs a cold first turn).

Round trip — feeding the live TTS output back into the live STT transcribes
at 100% word overlap, so the synthesized audio is genuinely intelligible speech
and not just well-formed bytes.

One operational note worth stating: on a cold replica the first synthesis
exceeded the 60s session timeout and was retried by the framework before
succeeding. That is cold-start behavior rather than an adapter issue, but
production voice agents should keep min_replica >= 1.

ruff check and ruff format are clean.

Baseten hosts Qwen3-ASR Streaming and Qwen3-TTS alongside the Whisper and Orpheus models this plugin already supports, but they speak different wire protocols, so the existing STT/TTS classes cannot reach them: - STT sends raw binary PCM and reads `message_type`/`transcript`; Qwen3-ASR takes base64 audio in OpenAI-realtime `input_audio_buffer.append` frames and replies with `type: "transcription"` / `segments[].text`. - TTS posts `{prompt, voice, ...}` (or a WS init frame plus an `__END__` sentinel); Qwen3-TTS uses `session.config` -> `input.text` -> `input.done`, where `input.done` is a flush rather than a close. Adds `Qwen3STT` and `Qwen3TTS` as separate classes so the existing Orpheus and Whisper paths are untouched. Both accept `model_endpoint`, `model_id`, or `chain_id` with the same precedence as `STT`. Qwen3TTS keeps one warm WebSocket across turns, since the session config is sticky and re-dialing would add a connect plus a config round trip to every agent response. Parked sockets are kept off the server's 30s idle timeout with an empty `input.done` flush (a protocol ping proves the socket is alive, not the session). Interrupted sockets are discarded rather than parked, because closing the socket is what stops in-flight generation. Qwen3-TTS Base ships no built-in speakers, so `voice` names a registered clone; `register_voice`/`list_voices` helpers are exported for managing them. Both support optional word-level timestamps via `TimedString`.

CLAassistant · 2026-08-04T22:25:21Z

All committers have signed the CLA.

- Import TimedString directly instead of guarding it behind a try/except. The fallback assigned None to a name mypy treats as a type, and in-tree the dependency is always current — every other plugin imports it unconditionally. - Drop the hand-rolled language-name table in favor of LanguageCode, which already normalizes names to codes ("English" -> "en"). Only Cantonese and Filipino are left untouched by it, so those stay as explicit overrides (Cantonese in particular is a distinct Qwen3-ASR language, not a zh variant). - Annotate the json.loads/dict.get returns in the voice-management helpers.

- Qwen3STT sent `input_audio_buffer.commit` twice at end of input. `end_input()` pushes a flush sentinel *and then* closes the channel, so the sentinel branch committed and the trailing block committed again with no audio in between, making the server answer a spurious empty turn. Track whether the last action was a commit, the way the TTS side already does. Covered by a regression test (verified failing before the fix). - Use BASETEN_MODEL_ENDPOINT in Qwen3STT rather than inventing BASETEN_STT_ENDPOINT. STT, TTS and Qwen3TTS all read the documented variable, so only Qwen3STT diverged — and resolve_endpoint claims to mirror STT. - Warn when an endpoint is plaintext ws:// to a non-loopback host. The API key travels in an Authorization header and the audio is unencrypted, so this is worth flagging; ws:// stays permitted for local proxies and tests. - Add Google-style docstrings to the new public classes and constructors, per CONTRIBUTING (pdoc3 generates the API reference from them).

Qwen3TTS._keepalive_loop held _warm_lock while awaiting _empty_flush, which sends input.done and waits for session.done. _acquire() needs the same lock on the hot path for every turn, so a reply that began mid-keepalive stalled for the rest of that round trip — measured at 1.2s against a deliberately slow server, on a warm pool whose whole purpose is to cut startup latency. Take the socket out of the slot before flushing and re-park it after. A turn arriving during the flush now finds an empty slot and dials its own socket instead of blocking, and the invariant that a turn and the keepalive never share a socket is preserved by construction rather than by the lock. If a turn parked its own socket meanwhile, the keepalive's is closed as surplus.

Baseten's streaming transcription API has accepted a `language_options` list since Whisper runtime v0.5.0, letting detection be scoped to the languages an agent actually supports. The plugin only ever sent a single `audio_language`, so users had to choose between a fixed tag that mistranscribes the other language and `auto`, which detects across all 99 and is unreliable on the one- to two-second utterances typical of telephony. The field is only added to the handshake when non-empty: `StreamingWhisperInput` uses `extra="forbid"`, so unconditionally sending it would break anyone on a runtime older than v0.5.0. Verified against a live Whisper Large V3 Turbo streaming deployment: the handshake with `language_options: ["en", "de"]` is accepted and transcribes normally, while a deliberately misspelled field name closes the socket with 1011 — confirming both the name and that rejection is real rather than silent.

- Replace the hand-rolled Qwen3ChunkedStream with the framework's `_synthesize_with_stream()`. Mine passed the caller's conn_options straight to the inner stream, so a persistent failure retried (max_retry+1)^2 times — nine attempts at the default — and it never forwarded USERDATA_TIMED_TRANSCRIPT, silently dropping the word timings the class advertises via aligned_transcript. Four other streaming-only plugins already use this helper. - Close the keepalive's in-flight socket when the task is cancelled. The previous commit deliberately removed the socket from the warm slot before flushing (to keep the lock off the hot path), which meant aclose() during a flush cancelled the only reference and leaked the connection. - Anchor Qwen3STT timings to the session clock. Segment and word times were reported socket-relative, so after a reconnect they jumped back toward zero. The framework grows `start_time_offset` for exactly this, and the existing Whisper STT already applies it. Regression tests added for the timing offset and for one-shot synthesis (asserts a single attempt and that timed transcripts survive). Not changed: `resolve_endpoint` still warns rather than rejects plaintext ws:// to non-loopback hosts. Hard-failing would break local proxy and test setups, and the existing TTS/STT in this plugin accept ws:// with no check at all, so the warning is already stricter than the status quo.

…plies `_control` parsed `(await ws.receive()).data` as JSON without inspecting the frame type. aiohttp gives `.data` as None on CLOSED, the close code on CLOSE, and an exception on ERROR, so a deployment that drops the connection after the auth handshake produced a bare TypeError instead of the tailored errors `register_voice`/`list_voices` raise. The rest of this module already switches on `msg.type` before parsing; this brings the helpers in line. A dropped connection now reports: unexpected reply to 'voice.list': CLOSE Still not changed: plaintext ws:// to a non-loopback host warns rather than raises. Rejecting it outright would break local proxy and test setups, and the pre-existing TTS/STT in this plugin accept ws:// with no check at all.

iancarrasco-b10 · 2026-08-05T01:02:47Z

+from .log import logger
+
+_TRUSS_URL_TEMPLATE = "wss://model-{model_id}.api.baseten.co/environments/production/websocket"
+_LOOPBACK = {"localhost", "127.0.0.1", "::1"}


ooc what is reason for this

It's the exemption list for the plaintext-ws:// warning further down in resolve_endpoint:

if endpoint.startswith("ws://") and urlparse(endpoint).hostname not in _LOOPBACK: logger.warning("endpoint %r is plaintext ws://: the Baseten API key and all audio " "will be sent unencrypted. Use wss:// for any non-local host.", endpoint)

We send Authorization: Api-Key ... on that connection, so a ws:// endpoint to a remote host puts the key and the audio in cleartext. Loopback is exempt because local proxies and the tests legitimately use ws://127.0.0.1 and nothing leaves the machine.

Warning rather than rejecting was deliberate — hard-failing would break those local setups, and the existing TTS/STT in this plugin accept ws:// with no check at all today.

Fair callout though: it was wedged between the two URL templates with no context, which is exactly why it read as arbitrary. Moved it next to the logic it serves and added a comment in 843eac0.

Answers review feedback: _LOOPBACK sat between the two URL templates with no context, reading as arbitrary. Group it with the warning it serves and say what it is for — hosts exempt from the plaintext-ws:// warning, because local proxies and tests legitimately use ws://127.0.0.1.

- Feed the TTS stall watchdog from incoming audio. `_SESSION_DONE_TIMEOUT` was only refreshed by the sender finishing and by `session.done`, but the server sends `session.done` once the *whole* utterance is synthesized — so a reply taking longer than 60s was aborted mid-sentence on a healthy socket actively delivering PCM, and because audio had already been pushed the framework refuses to retry. Setting progress on each binary frame keeps the timeout a genuine idle watchdog. (This is what I saw on a cold replica during live testing and wrongly wrote off as cold-start noise.) - Only open an STT turn on actual words. An empty final — Silero closing a turn on noise with nothing recognized — emitted START_OF_SPEECH and END_OF_SPEECH with no transcript, and under turn_detection="stt" that commits the user's turn and makes the agent answer silence. Both covered by regression tests verified failing beforehand: the long-turn one raised APITimeoutError, the empty-turn one emitted a spurious start_of_speech/end_of_speech pair.

The keepalive pulls the parked socket out of the warm slot before its round trip. A turn starting in that window dials its own socket, and on completion `_release` parks it but skips scheduling a keepalive, because the old task is still inside `_empty_flush` and therefore not `done()`. The old loop then saw `surplus` and returned, leaving the freshly parked socket with nothing refreshing it. Not a failure — `_CONFIGURED_TTL` (25s) is below the server's 30s idle timeout, so `_acquire` discards the stale socket instead of using a dead one — but the warm-socket path this class exists for was silently off until the next `_release` happened to observe the task as done. Drop the redundant socket and keep looping instead of returning; the loop re-reads `self._warm` each iteration, so it picks up whichever socket is currently parked. Regression test verified failing beforehand: the second connection received 0 keepalives before the fix and 1 after.

The previous fix taught the surplus branch to keep looping, but left the same hazard in its sibling: a failed flush returned unconditionally. The loop drops `_warm_lock` for the round trip, so a turn can park a fresh socket at any point, and `_release` cannot schedule a keepalive for it while this task is still running (not `done()`). Returning strands that socket until the next release happens to observe the task as done. Rather than patch the third exit, make the invariant structural: the loop now has no early returns at all and only ends on shutdown. "Nothing parked", "the parked socket is closed", and "the flush failed" all `continue`, and the next iteration re-reads `self._warm`. The task idles cheaply when there is nothing to ping, and `aclose()` still cancels it. Regression test verified failing beforehand: with the failure branch returning, the socket parked by a concurrent turn received 0 keepalives; it now receives them normally.

tinalenguyen

hi, thanks for the PR! i wonder if we could fold in Qwen3TTS and Qwen3STT into the existing baseten.STT and baseten.TTS classes based on the models specified. cartesia does something similar as well for their STT and it feels clean, what do you think? example snippet

…lasses Per review: select the protocol with `model` rather than exposing separate Qwen3STT/Qwen3TTS classes, following the Cartesia STT pattern of one public class dispatching to per-protocol stream implementations. stt=baseten.STT(model="qwen3-asr", model_id=...) tts=baseten.TTS(model="qwen3-tts", model_id=..., voice=...) `STTModels`/`TTSModels` join `LLMModels` in models.py. The Qwen3 protocol code becomes an internal `_Qwen3Backend` plus its stream class in each qwen3_* module; `STT`/`TTS` own the public surface, capabilities, and stream selection. `Qwen3STT`, `Qwen3TTS`, `Qwen3SpeechStream` and `Qwen3SynthesizeStream` are no longer exported. Orpheus and Whisper paths are untouched. Several parameters legitimately want a different default per model, so they became NotGivenOr and resolve after the model is known: `language` (en / auto), `partial_transcript_interval_s` (1.0 / 0.5), `vad_min_silence_duration_ms` (300 / 500), `vad_speech_pad_ms` (30 / 100), `voice` (tara / required), and `show_word_timestamps` — which defaults off for qwen3-asr because its aligner is opt-in on the deployment, and defaulting it on would have advertised an `aligned_transcript` capability the server may not honour.

Folding left `_streams` annotated as the local `SpeechStream` on the Whisper path and as the base `stt.SpeechStream` on the qwen3-asr path, and the `update_options` fan-out called a method only the Whisper stream defines. Use the base type throughout and narrow before the fan-out. A qwen3-asr stream can never reach that loop (the method returns earlier for that model), but the isinstance check states it rather than relying on it.

`STT.update_options` early-returns for qwen3-asr, and forwarded only language, vad_threshold and vad_min_silence_duration_ms. `vad_speech_pad_ms` is a real Qwen3-ASR handshake field that the constructor already accepts for that model, so runtime changes to it were silently discarded. Forward it, and warn for the two options that genuinely do not apply to this backend (`language_options`, `buffer_size_seconds`) rather than dropping them without a word.

A-K-Erol requested a review from a team as a code owner August 4, 2026 22:25