Skip to content

Releases

livekit-agents@1.8.13

Choose a tag to compare

@github-actions github-actions released this 18 Nov 21:41
9ce5a1e

New providers and models

  • VLM Run: new livekit-plugins-vlmrun plugin for vision-language chat and System One decisions, including qwen/qwen3.8-27b.
  • AssemblyAI STT 3.7 Realtime Agent: supported through LiveKit Inference as assemblyai/assemblyai-stt-3-7-realtime-agent.
  • Sarvam: STTRealtime now supports saaras:v4.

Breaking changes

  • Boson AI: replace livekit-plugins-boson with livekit-plugins-boson-ai, import livekit.plugins.boson_ai, and change the [boson] extra to [boson-ai]. The avatar plugin is unchanged.
  • Slng: the STT/TTS plugin and [slng] extra have been removed; migrate to another provider.
  • Gladia: remove AudioEnergyFilter imports and the energy_filter constructor argument. All audio is now forwarded, preserving transcript timestamps; quiet periods no longer trigger filter-driven recording completion.
  • GPT Live: inference HTTP refusals now raise APIStatusError rather than APIConnectionError. Update exception handlers accordingly.

Behavior changes

  • OpenAI decisions: direct DecisionModel() requests now default to gpt-6-luna; the inference name remains openai/gpt-6-luna-decisions.
  • Gemini Live: inference sessions now retain Gemini’s own turn detection to avoid missing replies.
  • LiveKit Inference: STT, TTS, and eligible realtime sessions now hand off during gateway drains. Failed WebSocket handshakes expose quota details and respect terminal retry denials. Explicit provider pinning now warns of removal on March 1, 2027; omit it to allow routing and fallback.
  • Python: package metadata now allows CPython 3.15; some plugins remain capped below it.

This release also improves channel memory cleanup, avatar receiver lifecycle, Lokutor connection recovery, Anthropic judge requests, and TTS markup handling.

What's Changed

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.12...livekit-agents@1.8.13

livekit-agents@1.8.12

Choose a tag to compare

@tinalenguyen tinalenguyen released this 13 Nov 17:37
3e51568

Delegation recovery

You can recover caller answers that GPT-Live transcribes but never delegates. After a quiet grace period, a detector checks missed turns and starts backend delegation when needed, with a note for answers spoken over the agent.

GPTLiveModel(delegation_recovery=True)

New providers and models

  • Lokutor: Conv 2.0 STT and Versa 2.0 TTS across nine languages.
  • xAI: xai/stt-2 through Inference, with word alignment for adaptive interruption.
  • Google: Gemini 4 Carbon, Gemini 4 Hydrogen (also through Inference), and gemini-decision-1 through the Decisions API.
  • Perplexity: Flash and standard Decider v1.3, v1.2, and latest aliases.
  • AssemblyAI: assemblyai-stt-3-7-realtime-agent streaming STT.
  • Baseten: streaming-v2 transcription protocol.
  • Turn detector: updated local v1-mini model with turn-length input.

Breaking changes

  • GPT-Live: delegation_created listeners now also receive backend delegations; filter mode="responses" where needed.
  • NVIDIA: TTS output is restricted to LINEAR_PCM; update other encoding configurations.

Behavior changes

  • Dependencies: Google now requires google-genai>=3.2, Phonic requires phonic>=0.33.4, and local inference requires >=0.3.0,<0.4.
  • Jobs: uncaught entrypoint failures report failure; job-triggered session closes report JOB_SHUTDOWN.
  • Manual turns: committed snapshots survive clears and subsequent VAD activity.
  • ElevenLabs: dialogue models now honor similarity_boost.

Interruptions, tool delivery, handoffs, telemetry authentication, speech connections, and answering-machine detection are more reliable.

What's Changed

  • fix(telemetry): keep LiveKit Cloud auth when OTEL_EXPORTER_OTLP_HEADERS sets Authorization by @prassu017 in #8407
  • feat(inference): Add xai/stt-2 to the STT models by @u9g in #8409
  • feat(google): bump google-genai to >=2.33.0 and support mid-session context_update by @manishkjs1 in #8264
  • fix(ipc): report failed job entrypoints as failed by @davidzhao in #8445
  • feat(aws): cache the conversation with cache_messages by @apoplexi24 in #8439
  • feat(gradium): keyword boosting in STT by @davencyw in #8425
  • fix(observability): keep Tagger outcome consistent when using add()/remove() by @MohammadHijjawi97 in #7490
  • fix(ipc): close the session as JOB_SHUTDOWN when the job shuts down by @regalmoix in #8039
  • Add OSS Scanner build image and threat model by @erikhortsch in #8326
  • feat(examples): add the four homepage demo agents by @ShayneP in #8216
  • feat(gradium): word timestamps in TTS by @davencyw in #8435
  • fix(duplex): route a call that arrives after an interrupt to its own generation by @prassu017 in #8462
  • fix(voice): reject a handoff to an agent that is already running by @longcw in #8487
  • fix(voice): keep the session busy while a reply is still owed by @longcw in #8459
  • fix(voice): return to listening after a forced interrupt of unplayed speech by @longcw in #8488
  • fix: correctly mark job failed by @davidzhao in #8490
  • update livekit sdk to 1.2.1 by @longcw in #8412
  • fix: snapshot manually committed user turns by @chenghao-mou in #7811
  • perf(ipc): preload the lazily-resolved livekit.agents.inference exports by @u9g in #8485
  • fix: preserve manual commits during vad activity by @chenghao-mou in #7812
  • feat(inference): add utterance_end_ms to DeepgramOptions by @adrian-cowham in #8498
  • fix(soniox): honor the language passed to STT.stream() by @anubhavlal07 in #8471
  • fix(utils): ConnectionPool.prewarm warns instead of raising without a running loop by @neil-sarvam in #8378
  • feat(zoom-ai): add turn_detection="none" (commit on LiveKit VAD flush) by @cdliao in #8356
  • docs: add Rime Prism to inference SUPPORTED REALTIME MODELS list by @detail-app[bot] in #8423
  • fix(voice): send a returned tool result when the interruption timeout ends its wait by @TANGBUDU in #8492
  • fix(phonic): reconnect after abnormal disconnects by @arunwpm-work in #8501
  • feat(gpt-live): add delegation recovery by @longcw in #8404
  • tests: speed up the unit suite and fold per-PR test files by @longcw in #8510
  • feat: add Atmee avatar plugin by @JonasDaWi in #7424
  • fix(aws): fall back to auto tool_choice for Claude Sonnet 5.5 on Bedrock by @kimnamu in #7631
  • feat(google): add DecisionModel for Gemini's Decisions API by @tinalenguyen in #8517
  • feat(google): add gemini-4-carbon to ChatModels by @tinalenguyen in #8500
  • feat(elevenlabs): send similarity and clone_strength to dialogue models by @ethankraus-elyos in #8481
  • Harden NVIDIA Speech STT and TTS support by @Yashi11 in #6706
  • feat(eot): add agent context, the mini model, and turn length by @chenghao-mou in #8113
  • fix(amd): settle uncertain when the callee hangs up while listening by @sarankumar1325 in #8242
  • examples(homepage): upgrade demo agents to livekit-agents 1.8.11 by @ShayneP in #8518
  • feat(baseten): add model="streaming-v2" STT for Baseten's Real-time transcription API (v2) by @peirangong-baseten in #8175
  • feat(google): add gemini-4-hydrogen to ChatModels by @tinalenguyen in #8526
  • feat: add Lokutor STT and TTS plugin by @danivs10 in #7256
  • fix(elevenlabs): close idle text-to-dialogue sockets before the server does by @longcw in #8519
  • fix(elevenlabs): retry a failed stream on a fresh websocket by @GoGennady in #8465
  • feat(anam): add region and region_policy to SessionOptions by @vishal27shetty in #8533
  • feat(inference): add google/gemini-4-hydrogen to LLM models by @tinalenguyen in #8527
  • feat(assemblyai): add assemblyai-stt-3-7-realtime-agent streaming model by @dlange-aai in #8525
  • Add Boson realtime plugin by @helin-mktech in #6340
  • feat(perplexity): add DecisionModel for Perplexity's Decisions API by @brady-pplx in #8528
  • livekit-agents@1.8.12 by @github-actions[bot] in #8539

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.11...livekit-agents@1.8.12

livekit-agents@1.8.11

Choose a tag to compare

@github-actions github-actions released this 05 Nov 19:48
296f28b

Responses API through LiveKit Inference

You can now use the Responses API with your LiveKit Inference credentials, including structured output and web search. REST is the default; opt into WebSocket transport to reuse connections across turns.

from livekit.agents import inference

llm = inference.responses.LLM("openai/gpt-4.1", use_websocket=True)

New providers and models

  • Zoom AI: Zoom Scribe STT with live streaming and per-utterance recognition.
  • Kalpa: kalpa-tts-1 streaming TTS in English and Hindi.
  • Rime: STT and Prism speech-to-speech through the standard realtime interface; both require Python 3.11+, and STT requires the stt extra.
  • Fish Audio: transcribe-stream-1 realtime STT with word timings and recognition context.
  • OpenAI: gpt-realtime-translate through the beta translation interface, with gpt-realtime-whisper input transcription.
  • Google: gemini-3.5-live-translate-preview translation through Gemini API and Vertex AI.
  • Respeecher: public Swedish, Brazilian Portuguese, German, and Arabic TTS models.

Security

  • Respeecher: moves WebSocket API keys from request URLs to handshake headers, keeping credentials out of URLs.

Behavior changes

  • Inference Responses: now defaults to each known model’s lowest reasoning effort; set reasoning=None to retain the provider default.
  • STT turn detection: late finals after speech resumes join the resumed turn, which waits for STT’s next end-of-speech event.
  • Uninterruptible speech: user transcripts are now retained without interrupting playback or generating a reply.

This release also improves realtime history and reconnection, task handoffs, STT timing, transcript synchronization, VAD updates on Python 3.14, worker cleanup, and multichannel TTS fallback.

What's Changed

  • fix(realtime): keep item ids within OpenAI's 32-char limit by @russellmartin-livekit in #7958
  • docs(mistralai): add missing store option to LLM options example by @detail-app[bot] in #8219
  • feat(soniox): pass client_reference_id to the TTS websocket config by @grace0950 in #8236
  • fix(thunderphone): use _shorten_id after the openai_utils rename by @chenghao-mou in #8229
  • test(drive_thru): replace the eval tests with simulation scenarios graded on the final order by @u9g in #8176
  • fix(voice): suspend an AgentTask's awaiter in the run instead of unwatching it by @u9g in #7794
  • fix(voice): don't fail the run with a task error its awaiting tool handled by @osekkat in #8269
  • Sunset Sarvam TTS model bulbul:v2 and remove bulbul:v3-beta by @dhruvladia-sarvam in #7056
  • feat(rime): add Coda custom_lexicon option by @saad-py in #8245
  • fix(voice): end a turn on a late STT final after a later END_OF_SPEECH by @acharan-tech-200037 in #7739
  • update livekit sdk/api to 1.2/1.2.2 by @davidzhao in #8299
  • fix(voice): keep the user turn in the chat context when the current speech cannot be interrupted by @valemicolgarcia in #7922
  • chore: remove dead AgentsConsole members from cli.py by @detail-app[bot] in #8277
  • fix(worker): close resources when startup fails before run() completes by @BlueX888 in #7942
  • fix(deepgram): guard empty languages list in multilingual STT decoding by @Speedyzyzz in #7976
  • fix(vad): resize the speech buffer by copying so it works on Python 3.14 by @RitikaxG in #8162
  • fix(tools): omit cancel_task from duplicate reject/confirm when not cancellable by @ninglinLiu in #8270
  • reduce status update interval to 1.5s by @davidzhao in #8150
  • fix(openai): end a failed realtime response without an unretrieved exception by @zek in #8218
  • fix(voice): ignore a repeated flush of the same audio segment by @sehaj23 in #8316
  • fix(telemetry): keep the stall sampler out of linecache by @zek in #8226
  • fix: resolve a failed local turn-detector prediction to the positive default by @chenghao-mou in #8171
  • feat(rime): STT Support by @augustohalvarez in #8013
  • chore(krisp): bump livekit-plugins-krisp-internal to 0.2.3 by @lukasIO in #8250
  • feat(mistralai): expose reasoning effort on the LLM by @muzman123 in #8285
  • fix(respeecher): use WebSocket auth headers and add public models by @mitrushchienkova in #8286
  • feat(speechmatics): support 8kHz audio and emit_sentences config by @LArmstrongDev in #7952
  • chore(deps): update dependency langgraph to v1.2.14 [security] by @renovate[bot] in #8254
  • feat: add Zoom AI STT plugin by @cdliao in #7708
  • feat(cambai): add speaking_rate option to TTS by @nihaalnz in #8324
  • update dependencies by @davidzhao in #8325
  • feat(openai): gpt-realtime-translate as a beta TranslationModel by @tinalenguyen in #8105
  • fix(stt): anchor transcript timestamps on the session input start by @chenghao-mou in #7795
  • fix: report speech end timestamps across stt plugins by @chenghao-mou in #7796
  • refactor(openai): Move the Responses engine into livekit-agents by @adrian-cowham in #8188
  • test(openai): Add a live probe for multi-message Responses replies by @adrian-cowham in #8209
  • feat(inference): Add inference.responses.LLM by @adrian-cowham in #8210
  • feat(inference): WebSocket transport for inference.responses.LLM by @adrian-cowham in #8211
  • feat(inference): update_options and response_format for inference.responses.LLM by @adrian-cowham in #8212
  • docs(examples): Show inference.responses.LLM in basic_agent by @adrian-cowham in #8213
  • fix(inference): Don't retry terminal quota denials on inference.responses by @adrian-cowham in #8268
  • fix: drop self from the static type of bound function tools by @u9g in #8336
  • fix(openai): stop resending the previous output when chaining Responses requests by @longcw in #8085
  • fix(google): start a fresh Gemini session on a config change by @longcw in #7944
  • fix(voice): don't hide IGNORE_ON_ENTER tools after on_enter returns by @BhaskarKapri07 in #8280
  • fix(inference): don't crash on tool_call arguments that arrive before a name by @AKomplished-bug in #8249
  • feat(gradium): optional VAD-based start of speech in STT by @davencyw in #8247
  • fix(room_io): date a run's end at submission, not after a stalled capture by @u9g in #8168
  • fix(aws): hold on to the user-text and deferred-recycle tasks by @Rehansanjay in #7052
  • fix(voice): let only the STT end a turn the STT reopened by @chenghao-mou in #8318
  • fix(transcription): stop speaking-rate bursts from blocking the event loop by @longcw in #7938
  • fix(duplex): keep the words still waiting when the output stream ends by @basnijholt in #8301
  • fix(inference): Send agent_context only to STT models that read it by @adrian-cowham in #8365
  • feat(google): Gemini 3.5 Live Translate as a beta TranslationModel by @tinalenguyen in #8106
  • feat(openai): Add GPT-Live Decisions support by @shawnfeldman in #8338
  • feat(fishaudio): add realtime STT by @cshape in #8357
  • feat(inference): reasoning= for inference.LLM and inference.responses.LLM by @adrian-cowham in #8358
  • feat(inference): default inference.responses.LLM to the model's lowest reasoning effort by @adrian-cowham in #8359
  • fix(fireworksai): honor stream language override by @JY-M666 in #8390
  • feat(examples): support Tavus alongside LemonSlice in the avatar example by @ggarber in #8131
  • feat(voice): send session_closed to the remote session when the session closes by @u9g in #8099
  • fix(aws): honor language passed to STT.stream() by @pcbeingused333 in #8374
  • fix(tts): preserve channels when resampling fallback audio by @feiiiiii5 in #8393
  • feat(rime): add Prism through the standard realtime interface by @naszzz in #7856
  • fix(realtime): end a f...
Read more

livekit-agents@1.8.10

Choose a tag to compare

@github-actions github-actions released this 29 Oct 17:00
ff962a0

Recording bitrate control

You can use constant-bitrate recording to avoid misleading duration estimates in players such as QuickLook, or match Egress’s 128 kb/s encoding. RecorderIO now lets you set the bitrate and encoding mode; defaults are unchanged.

RecorderIO(agent_session=session, bit_rate=128_000, constant_bit_rate=True)

New providers and models

  • Reson8: new STT plugin for batch and realtime transcription, including server-side turn detection.
  • Mistral Voxtral 3: adds voxtral-mini-transcribe-3, voxtral-small-transcribe-3, voxtral-mini-transcribe-realtime-3, and voxtral-mini-tts-3, plus 36 TTS voice presets.
  • Inworld: adds inworld-tts-2-flash through the plugin and LiveKit Inference.
  • AssemblyAI: adds typed LiveKit Inference TTS support for assemblyai and assemblyai/assemblyai-tts-1.

Breaking changes

  • Mistral: api_mode is now ignored; migrate Chat Completions usage to Conversations. For recognize(), explicitly select a batch model; replace removed known IDs voxtral-mini-latest and voxtral-mini-2507. Without VAD or server-side turn detection, standalone realtime streams need stream.flush().

Behavior changes

  • Mistral: now requires mistralai>=3.5.0,<4.0.0; defaults move to Voxtral 3 STT/TTS and mistral-small-latest. Silero is no longer loaded automatically.
  • TTS defaults: AssemblyAI now uses Audrey; Inworld uses inworld-tts-2.
  • Streaming STT: detects stalled connections and limits retries without processing progress so fallback can proceed.
  • Worker health: returns 503 before registration and after disconnections exceed the 30-second grace period.

STT flushing, language overrides, token accounting, translation startup, avatar errors, and noise cancellation are more reliable.

What's Changed

  • fix(google): count Gemini thinking tokens in LLM completion_tokens by @ninglinLiu in #7891
  • fix(hume): reject instant_mode without a voice in update_options by @ShivamSharma43 in #8090
  • fix(tokenize): keep runs like ?! and !!! in one sentence by @ANDROIDHASSAN in #8116
  • fix(gladia): require translation target languages in update_options by @ShivamSharma43 in #8091
  • fix(tests): replace TCP port check with HTTP readiness probe in livekit_server fixture by @Lothnic in #6094
  • fix(fal, mistralai): don't persist the per-call language passed to recognize() by @ShivamSharma43 in #8054
  • fix(stt): stop per-call language leaking in openai/elevenlabs, forward deepgram options to open streams by @Atharvsinh-codez in #7828
  • fix(worker): return 503 from health until RegisterWorkerResponse by @ninglinLiu in #8093
  • fix(inference): detect a silently dropped Inference STT socket instead of hanging by @regalmoix in #8023
  • feat(openai): add TTS sample rate option by @rosetta-livekit-bot[bot] in #8122
  • fix(llm): drop the dangling openai.responses entry from _provider_format.all by @ShivamSharma43 in #8008
  • fix(anthropic): send user as metadata.user_id by @omChauhanDev in #7970
  • feat(voice): add bit_rate and constant_bit_rate options to RecorderIO by @ANDROIDHASSAN in #7740
  • Detect stalled STT connections and limit recovery retries by @davidzhao in #8123
  • fix(beta): stop unset_track from stalling TranslationRoomIO startup by @RaphaelFakhri in #8115
  • feat(assemblyai): default the TTS voice to Audrey by @dlange-aai in #8132
  • feat(reson8): add STT plugin with server-side turn detection by @phillip-kil in #7187
  • Add docstrings to the xAI realtime model by @aryeila in #8141
  • fix(deepgram): Drop the language hint the model cannot take, and add it for nova-3 multi by @adrian-cowham in #8079
  • feat: send up existing jobs during worker connection migration by @davidzhao in #8135
  • fix(bithuman): don't retry 4xx when starting a session, and keep the provider's error by @sidxh in #7500
  • feat(inference): Type and gate language_hint for Deepgram Nova-3 by @adrian-cowham in #8072
  • docs(plugins): fix environment variables in the neuphonic and rtzr READMEs by @ninglinLiu in #8151
  • fix(baseten): flush buffered audio with end_audio when Whisper STT input ends by @A-K-Erol in #8144
  • chore(examples): remove provider-specific examples by @tinalenguyen in #8154
  • feat(inworld): add inworld-tts-2-flash and move the default off TTS-1.5 by @shawnfeldman in #8047
  • synthesia: optionally start the avatar in the background (wait_for_avatar=False) by @atharvapatil-synth in #8146
  • fix(assemblyai): don't cap language_codes at 10 on universal-3-6-pro by @tinalenguyen in #8178
  • feat(simli): send Trinity-2 session options and the plain face ID by @Antonyesk601 in #8174
  • docs(microsoft-ai): drop stale smoke script references by @detail-app[bot] in #8182
  • feat(mistralai): add Voxtral 3 TTS support by @jeanprbt in #8164
  • feat(mistralai): add Voxtral 3 STT support by @jeanprbt in #8136
  • refactor(mistralai): use the Conversations API only for the LLM by @jeanprbt in #8169
  • feat(inference): Type AssemblyAI TTS models and options by @tinalenguyen in #8181
  • feat(mistralai): expose the Conversations API store option on the LLM by @tinalenguyen in #8187
  • chore(deps): update dependency nltk to v3.10.4 [security] by @renovate[bot] in #8186
  • chore(krisp): bump livekit-plugins-krisp-internal to 0.2.2 by @lukasIO in #8198
  • livekit-agents@1.8.10 by @github-actions[bot] in #8206

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.9...livekit-agents@1.8.10

livekit-agents@1.8.9

Choose a tag to compare

@github-actions github-actions released this 24 Oct 02:35
d4de578

Decisions through LiveKit Inference

You can evaluate named decisions with LiveKit credentials instead of a provider key. inference.DecisionModel batches evaluations and reports refusals per decision without discarding the other answers. Access requires a project feature flag.

model = inference.DecisionModel("openai/gpt-6-luna-decisions")
response = await model.evaluate(
    input=chat_ctx,
    decisions={"handoff": llm.Probability("The caller requests a human.")},
)

Custom LLM request attributes

You can attach prompt metadata directly to the provider’s llm_request span from your custom llm_node, without a custom OpenTelemetry processor. Attributes stay scoped to that node’s calls.

async def llm_node(self, chat_ctx, tools, model_settings):
    gen_ai.set_llm_request_attributes(
        {"langfuse.observation.prompt.name": "support-agent"}
    )
    async for chunk in Agent.default.llm_node(
        self, chat_ctx, tools, model_settings
    ):
        yield chunk

Silent tool progress

You can record background-tool progress without prompting a spoken response. Final background-tool results can now trigger follow-up tool calls.

await ctx.update("checking the fare rules", silent=True)

New providers and models

  • ThunderPhone: new realtime speech-to-speech plugin for saved-agent and inline sessions, including bolt.
  • OpenAI: Decisions API support directly and through Inference with openai/gpt-6-luna-decisions.
  • TypeSafe Jev and LiveKit Clef Omni: decision evaluation through LiveKit Inference.
  • Microsoft-Decision-1: batched decisions through Microsoft Foundry.
  • AssemblyAI assemblyai-tts-1: streaming TTS with word timestamps and six languages.
  • Keyframe persona-2.0-live: speech-timed avatar expressions.
  • Fish Audio drama-3: Inference TTS with free-form expressive steering.
  • Speechify simba-3.5: streaming synthesis with timestamps.
  • Google gemini-3.8-flash: added to supported chat models and selected by default.

Security

  • Tool errors: raw arguments and validation details no longer bypass PII filtering through log message bodies.
  • Simli: startup error logs no longer expose the session token.
  • RoomIO: configured participant identity now restricts chat before linking and during reconnection, preventing another participant from starting a turn.

Breaking changes

  • Rime: WebSocket v1 requires explicit model; add it if you relied on URL-path detection.
  • Gradium: passing both voice and voice_id now raises ValueError; choose one.
  • Inworld: max_connections and idle_connection_timeout are deprecated and ignored; TTS now uses one WebSocket per instance.

Behavior changes

  • Adaptive interruptions: errors now fall back to VAD without reconnecting by default.
  • STT flushing: supported engines now flush at manual and detected turn boundaries; buffering custom nodes must disable manual_flush.
  • Gradium: the default voice is now Everly; explicit voice selections are honored.
  • Anthropic: forced tool choices become auto for Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1, so tool use is not guaranteed.
  • Inworld: inworld-tts-2 uses server-side sentence splitting by default; client-side splitting defaults to 250 characters.
  • Dependencies: OpenAI requires SDK 3.28+, Phonic requires phonic>=0.32.36, and the framework requires livekit-protocol>=1.1.28.

This release also improves reliability across realtime history, tool handoffs, task shutdown, TTS retries, avatar interruptions, worker prewarming, and session reporting.

What's Changed

  • fix(workflows): support turn_handling in warm transfers by @JonasDaWi in #7847
  • feat(room_io): warn when agent and client both apply noise cancellation by @lukasIO in #7848
  • docs: stop flagging LiveKit identifiers as PII in Devin review by @chenghao-mou in #7849
  • fix(inference): send keyterms to speechmatics/linden-1 by @shawnfeldman in #7853
  • feat(inference): add DecisionModel for the gateway's /v1/decisions by @shawnfeldman in #7855
  • docs(inference): Deepgram mip_opt_out is always true on the gateway by @shawnfeldman in #7860
  • fix(elevenlabs): offset text-to-dialogue alignment by chunk start by @satheeshk50 in #7784
  • prewarm Resource.create to avoid blocking runloop during start by @davidzhao in #7872
  • fix(voice): skip audio turn handling warnings in text simulations by @u9g in #7878
  • fix(gradium): identify the STT plugin to the API with x-api-source: livekit-plugin by @hhugo-gradium in #7873
  • docs: add Google realtime models to inference SUPPORTED REALTIME MODELS list by @detail-app[bot] in #7861
  • docs: add the shared LiveKit sections to AGENTS.md by @JustinScitech in #7885
  • fix(anthropic): fall back to auto tool_choice for Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1 by @omChauhanDev in #7511
  • feat(deepgram): support speed parameter in Flux TTS by @73bruno in #7483
  • fix(llm): drop nonexistent name from all by @BlueX888 in #7894
  • Clarify what "item" is in logged warning by @kath0la in #7934
  • Send lk.agent.session messages with one-shot send_bytes instead of stream_bytes by @1egoman in #7751
  • fix(cli): apply connect credentials to the agent server by @BlueX888 in #7929
  • feat(telemetry): let llm_node set attributes on the llm_request span by @rrfunde in #7827
  • fix(simli): retry session start and stop the error handler from crashing by @sidxh in #7498
  • feat(soniox): report output tokens in RECOGNITION_USAGE for token-based billing by @Panmax in #6435
  • fix(cartesia): expand TTS language support and add WebSocket timestamp params by @Namit1867 in #7754
  • fix(voice): keep ToolError detail out of log message bodies by @Li-john1021 in #7804
  • fix(llm): keep a nullable enum's null out of the shared $defs entry by @ANDROIDHASSAN in #7760
  • fix(llm): render missing {} and formatted template placeholders as empty by @ANDROIDHASSAN in #7783
  • fix(tts): don't repeat the reply when a FallbackAdapter stream retries by @RachelWanggg in #7840
  • fix(ipc): drop exited idle processes from the warmed queue by @RGB-loop in #7864
  • fix(bey): use the current session endpoint and send 16 kHz audio by @niqodea in #7877
  • fix(bey): don't retry 4xx when starting a session, and keep the provider's error by @sidxh in #7496
  • fix: do not reconnect the adaptive interruption stream by @chenghao-mou in #7923
  • chore: remove unused BoundedDict.set_or_update method by @detail-app[bot] in #7568
  • feat(openai): add DecisionModel for OpenAI's Decisions API by @tinalenguyen in #7882
  • fix(proto): stringify map values in encode_by_name by @u9g in #7962
  • fix(rime): require explicit v1 models and anonymize endpoints by @naszzz in #7964
  • fix(meta): don't delay Muse STT audio by the websocket connect time by @DEVJHAWAR11 in #7890
  • fix(voice): stop wait_for_idle from spinning through a speech delay by @longcw in #7965
  • fix(voice): preserve tool outputs across concurrent realtime context updates by @Epsilon006 in #7905
  • fix: honor participant identity for chat before linking by @chenghao-mou in #7435
  • fix(tokenize): don't hold a CJK reply until the end in word streams by @dev-S-t in #7669
  • fix(hume, simli): make every all entry resolve by @omChauhanDev in #7981
  • fix(inference): emit every parallel tool call sent in one chunk by @pcbeingused333 in #7887
  • fix(realtime): treat credit_balance_exhausted as a fatal OpenAI error by @jipeng6036-del in #7819
  • fix(gradium): identify the TTS plugin to the API with x-api-source (1/4) by @hhugo-gradium in #7815
  • fix(voice): add say() text to the realtime chat context when it is spoken by @xiaoyuCR7 in #7947
  • fix(gradium): stop voice_id default from overriding an explicit voice (1/2) by @hhugo-gradium in #7813
  • fix(room_io): size trailing silence by num_channels by @Hugo-DDT in #7968
  • fix: prevent stale avatar clear timeouts from finishing new replies by @chenghao-mou in #7980
  • feat(microsoft-ai): add DecisionModel for Microsoft-Decision-1 by @tinalenguyen in #7883
  • fix: keep sarvam stt streams open across flushes by @chenghao-mou in https://github.com/livekit/a...
Read more

livekit-agents@1.8.8

Choose a tag to compare

@github-actions github-actions released this 15 Oct 06:10
31c6026

GPT-Live through LiveKit Inference

You can now run GPT-Live with your LiveKit credentials instead of a separate OpenAI key. inference.DuplexModel handles its protocol, and you can adopt it directly with a model string:

session = AgentSession(llm="openai/gpt-live-1")

New providers and models

  • OpenAI GPT-Live 1: now available through LiveKit Inference.
  • Google Gemini Live: adds Inference model-string support for google/gemini-3.8-live, google/gemini-3.8-live-extended-thinking, and google/gemini-live.
  • xAI Grok Voice 3.0: replaces Think Fast 2.0 in the Inference and xAI plugin model lists.
  • TypeSafe Jev: a new decision-model plugin supports jev-1.13.0 and OpenRouter’s Decisions API for background probability, choice, and score evaluations without delaying replies.

Breaking changes

  • OpenAI realtime: inference.RealtimeModel no longer subclasses the plugin’s openai.realtime.RealtimeModel; update isinstance checks. Incidental wildcard exports such as realtime_model.logger are no longer available.
  • GPT-Live voices: cedar replaces the unsupported aster voice; update voice selections.

Behavior changes

  • LiveKit Inference: retired GPT Realtime 1.5/2 and Grok Voice Think Fast 2.0 IDs remain accepted with deprecation warnings. Realtime inference_class is deprecated and no longer sent.
  • Google Gemini: 3.x reply instructions now use wrapped user turns.
  • xAI: hosted sessions now emit one final user transcript per turn.

XML-aware TTS chunking, realtime startup retries, turn-audio flushing, and interruption handling are more reliable.

What's Changed

  • fix(tokenize): ignore stray closing tags in the XML depth check by @ANDROIDHASSAN in #7742
  • feat(inference, google): run GPT-Live as inference.DuplexModel, wrap Gemini 3.x reply instructions, update the realtime model list by @russellmartin-livekit in #7673
  • feat(agents): add DecisionModel for background decisions by @tinalenguyen in #7714
  • feat(typesafe): add TypeSafe Jev decision model plugin by @tinalenguyen in #7756
  • livekit-agents@1.8.8 by @github-actions[bot] in #7844

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.7...livekit-agents@1.8.8

livekit-agents@1.8.7

Choose a tag to compare

@github-actions github-actions released this 14 Oct 04:29
43ca0f8

Realtime E2E latency

You can now measure realtime replies from the last detected user speech to first assistant playback when you configure local VAD. Read the latency from assistant message metrics:

@session.on("conversation_item_added")
def on_message(event):
    if event.item.type == "message" and event.item.role == "assistant":
        print(event.item.metrics.get("e2e_latency"))

New providers and models

  • Claude Opus 5, Opus 5.5 and Sonnet 5.5: model support through the Anthropic SDK upgrade.

Breaking changes

  • Telemetry: fallback attributes lk.fallback.label and lk.fallback.index move to adapter request spans; update trace queries for the flattened hierarchy.
  • Anthropic: custom SDK HTTP clients must use httpx2, not httpx.
  • OpenTelemetry: replace the removed Events API with LogRecord(event_name=...); custom Logger subclasses must implement enabled().

Behavior changes

  • Dependencies: now requires openai>=3.19,<4, mcp>=2.2,<3, websockets>=17.1,<17.2, and OpenTelemetry >=1.45,<1.46; Anthropic requires >=1.8,<2, and turn detection requires onnxruntime>=1.30,<1.31.
  • TLS: contexts are cached per trust configuration; restart workers after replacing a CA bundle at the same path.
  • Tools: tools still running at interruption timeout now continue asynchronously and deliver deferred results.
  • Google: instruction-free realtime replies are allowed on immutable-context models; unavailable timing is reported as unknown.
  • ElevenLabs: STT client errors now retain their status and fail without retries.

Recording shutdown, VAD resizing, interruption teardown, chat summaries, structured outputs, multilingual sentence splitting, and tool discovery are more reliable.

What's Changed

  • fix(deps): update third party deps by @renovate[bot] in #5489
  • fix(deps): update third party deps (major) by @renovate[bot] in #6268
  • fix(upliftai): declare the python-socketio dependency by @zaidqureshi2 in #7567
  • fix(google): only block generate_reply when instructions passed to immutable-context model by @d42f in #5715
  • flatten fallback/stream adapter spans, name model on attempts by @davidzhao in #7723
  • fix(elevenlabs): fail fast on 4xx errors in STT recognize() by @pradneshfernandez in #7701
  • fix(http_context): build the TLS context once per process, not once per job by @zdurm in #7617
  • fix(ivr): reimplement TfidfLoopDetector with numpy to remove scikit-learn dependency by @Mugunth007 in #6867
  • fix: preserve nullable literal response schemas by @liangzhechen655 in #7715
  • fix: discard resampler history when resetting audio buffer by @liangzhechen655 in #7713
  • soniox: add reduce_silence option to TTS by @razvanvoica in #6798
  • fix(google): report unknown timing for results without word offsets by @15054538509 in #7687
  • fix: measure e2e latency for realtime agents by @chenghao-mou in #7719
  • Only send modern transcriptions to client sdks which can auto back convert modern -> legacy transcriptions internally by @1egoman in #7240
  • fix: reference real tool name in ToolProxyToolset description and errors by @BlueX888 in #7734
  • fix(voice): don't hang or buffer audio when the recorder encoder fails by @ANDROIDHASSAN in #7741
  • fix(vad): clamp cursor when resizing speech buffer by @gudcks0305 in #7566
  • fix(tokenize): split sentences on Arabic, Urdu and Devanagari terminators by @ahmad65saad in #7509
  • fix(interruption): tolerate websocket teardown race on session.close write by @BhaskarKapri07 in #6893
  • fix(telemetry): count a watchdog starved after it woke inside the stall window by @zdurm in #7612
  • fix(llm): keep earlier summaries when summarizing chat context again by @RaphaelFakhri in #7721
  • feat(deepgram): make Flux audio chunk duration configurable by @dnaumenko in #7205
  • feat(llm): report prompt-cache write tokens as cache_creation_tokens by @adrian-cowham in #7710
  • docs: fix broken DeepWiki badge in README by @longcw in #7770
  • fix: split underscore terms in BM25 description and param tokens by @BlueX888 in #7733
  • fix(voice): answer every tool return after ctx.update() by @longcw in #7777
  • fix(voice): release tools still running when an interrupted speech times out by @longcw in #7759
  • ci: draft release notes in the release PR by @longcw in #7779
  • fix: preserve reply_required for invalid tool results by @chenghao-mou in #7434
  • feat(worker): add preload_modules to forkserver preload by @tanish111 in #7761
  • fix: flush buffered tts audio before playback drains by @chenghao-mou in #7692
  • feat(llm): add CacheBreakpoint content marker for OpenAI prompt caching by @adrian-cowham in #7459
  • feat(inference): send prompt cache breakpoints for GPT-5.6 and later by @adrian-cowham in #7461
  • feat(openai): send prompt cache breakpoints for GPT-5.6 and later by @adrian-cowham in #7464
  • feat(voice): end the instructions message with a cache breakpoint each turn by @adrian-cowham in #7475
  • feat(llm): add Instructions(dynamic=...) for per-call text behind a cache breakpoint by @adrian-cowham in #7478
  • feat(openai): send prompt cache breakpoints from the Responses LLM by @adrian-cowham in #7656
  • feat(inference): add openai/gpt-live-transcribe STT by @shawnfeldman in #7479
  • feat(fishaudio): support drama-3 model by @cshape in #7799
  • gemini 4 argon by @tinalenguyen in #7790
  • feat(inference): add language_hint to DeepgramFluxOptions by @adrian-cowham in #7797
  • fix(rime): use model-specific WebSocket endpoints by @naszzz in #7766
  • livekit-agents@1.8.7 by @github-actions[bot] in #7792

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.6...livekit-agents@1.8.7

livekit-agents@1.8.6

Choose a tag to compare

@tinalenguyen tinalenguyen released this 09 Oct 22:17
294de27

What's Changed

  • fix(sarvam): close the streams the STT collects by @dhruvladia-sarvam in #7058
  • fix(aws): keep realtime session active after stream restart by @uuzzrm in #6811
  • fix(telemetry): format a loop stall report without reading source files by @zdurm in #7616
  • update deps by @davidzhao in #7655
  • fix(deps): bump bithuman in uv.lock for Windows wheel support by @abhijeetgaur-dev in #7559
  • fix(voice): keep preemptive generation when only non-ASCII punctuation changes by @ahmad65saad in #7507
  • fix(ipc): allow opting out of the local EOT weights in the preload by @Zaygal in #7580
  • fix(openai): prewarm Responses API connections by @nikhiljangid120 in #7584
  • feat(speechify): stream synthesize(), reuse the HTTP connection, add prewarm() by @vitalragaz in #7543
  • fix(tests): pin toxiproxy to v2.12.0 by @1egoman in #7676
  • fix(inference): keep temperature/top_p for gpt-5.1+ at reasoning_effort none by @adrian-cowham in #7650
  • fix(ipc): skip the EOT preload in spawned job processes by @longcw in #7665
  • fix(voice): enforce max_tool_steps on the realtime tool loop by @longcw in #7555
  • fix(mcp): tell the server when a tool call is cancelled by @longcw in #7542
  • fix(voice): store only the spoken part of a reply when the TTS fails by @longcw in #7670
  • fix(llm): diff a chat context in O(n log n) when its item ids are unique by @zdurm in #7626
  • fix(ipc): let one flag skip the whole local-inference preload by @longcw in #7681
  • fix(ipc): close the job loop when the runner exits by @zdurm in #7630
  • fix(gpt-live): remember the mute across a reconnect by @Zaygal in #7675
  • fix(tts): keep TTS.sample_rate in sync when update_options changes it by @Rehansanjay in #7135
  • bithuman: fix package description casing and add bitHuman docs URL by @sgu-bithuman in #7677
  • fix(voice): skip the AEC warmup in simulations by @chenghao-mou in #7674
  • fix: migrate soniox websocket authentication by @chenghao-mou in #7698
  • feat(sarvam): add bulbul:v4-flash TTS by @dhruvladia-sarvam in #7335
  • feat(nabrah): add Nabrah Arabic STT plugin by @MagdiWaleed in #7337
  • feat(cerebras): add qwen-3.8-27b to CerebrasChatModels by @abhijeetgaur-dev in #7680
  • docs(microsoft_ai): correct MAI STT/TTS availability in README by @tinalenguyen in #7704
  • livekit-agents@1.8.6 by @github-actions[bot] in #7707

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.5...livekit-agents@1.8.6

livekit-agents@1.8.5

Choose a tag to compare

@tinalenguyen tinalenguyen released this 06 Oct 02:06
76de175

What's Changed

  • fix: support gemini-3.8-live and warn on api mismatches by @chenghao-mou in #7587
  • feat: add sticky llm fallback routing by @chenghao-mou in #7589
  • chore: remove unused ATTR_START_TIME, ATTR_END_TIME, and ATTR_SESSION_OPTIONS telemetry constants by @detail-app[bot] in #6934
  • docs: update LiveKit Inference OpenAI model list for newer GPT-5.x variants by @detail-app[bot] in #7588
  • docs(bithuman): bring the plugin README and docstrings up to date by @sgu-bithuman in #7570
  • fix(google): send the call id with tool responses on Vertex AI by @lucasolinas in #7591
  • fix(cli): preserve Unicode logs on Windows CP1252 by @Luke-salinero in #7594
  • fix(llm): keep non-ascii text readable in mcp results and task group tool description by @avionicharshit-byte in #7565
  • chore(examples): use AssemblyAI universal-3-6-pro by @dlange-aai in #7578
  • fix(google): don't send a locally made call id back to the server by @davidzhao in #7610
  • fix(openai): replay function calls and their outputs on a realtime reconnection by @zdurm in #7576
  • fix(soniox): widen final span to all timed tokens, never regress end_time by @rkfshakti in #6976
  • chore(deps): update dependency pyjwt to v2.15.0 [security] by @renovate[bot] in #7563
  • chore(deps): update astral-sh/setup-uv action to v10 by @renovate[bot] in #7048
  • fix(tavus): don't retry 4xx when calling the Tavus API, and keep the provider's error by @sidxh in #7494
  • fix(telemetry): free the sampled threads' frames when a loop stall sample returns by @zdurm in #7613
  • fix: avoid duplicate token usage on llm fallback spans by @chenghao-mou in #7532
  • feat: correlate telemetry spans with the conversation by @chenghao-mou in #7533
  • feat: include tool names in telemetry result parts by @chenghao-mou in #7534
  • fix(elevenlabs): prewarm websocket connection by @nikhiljangid120 in #7586
  • fix(elevenlabs): re-arm response timeout after audio arrives by @davidzhao in #7627
  • feat: record LLM input as deltas on trace spans by @davidzhao in #7605
  • fix(voice): re-watch a completed AgentTask's awaiter in the active run by @u9g in #7641
  • feat(boson-avatar): add avatar plugin by @warlockee in #7019
  • feat(aws): add Nova Sonic 2.5 support by @samx18 in #7505
  • livekit-agents@1.8.5 by @github-actions[bot] in #7642

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.4...livekit-agents@1.8.5

livekit-agents@1.8.4

Choose a tag to compare

@tinalenguyen tinalenguyen released this 01 Oct 22:51
e684c23

What's Changed

  • feat(smallestai): add TTS continuations protocol support by @harshitajain165 in #7111
  • baseten: inline mid-conversation instructions for Qwen/Qwen3.8-27B by @henrytang-baseten in #7388
  • fix: drain inference STT transcripts before closing by @rosetta-livekit-bot[bot] in #7115
  • feat(azure): Add Azure Voice Live Realtime API support by @007DXR in #4693
  • Revert "worker: read agent name from livekit.toml as last fallback (#7295)" by @u9g in #7452
  • docs: versions change only in release PRs by @u9g in #7457
  • fix(workflows): forward Twilio CallToken during warm transfers by @piyush-gambhir in #7309
  • feat: record stt event metadata on user turns by @chenghao-mou in #7473
  • feat(deepgram): support speed parameter in TTS by @73bruno in #7480
  • feat(assemblyai): default to universal-3-6-pro by @dlange-aai in #7518
  • fix(aws): omit temperature/top_p for OpenAI GPT-5.6 and GPT-6 on Bedrock by @kimnamu in #7414
  • fix(aws): stop logging "unknown chunk type" for Bedrock reasoning deltas by @kimnamu in #7416
  • fix(aws): don't retry Bedrock 4xx errors such as ValidationException by @kimnamu in #7418
  • feat(elevenlabs): support eleven_v4 and eleven_v4_turbo by @kraenhansen in #7519
  • fix(gladia): store region in STT.update_options by @RaphaelFakhri in #7523
  • updated smallestai stt adapter by @hamees-sayed in #7379
  • fix(llm): block unguarded mutations on read-only chat contexts by @BlueX888 in #7445
  • fix(voice): treat a non-list record_keyterms argument as no change by @Lesereingrape in #7468
  • fix(stt): honor APIError.retryable in recognize() by @yaosongding in #7474
  • fix(llm): make ChatContext.truncate() honor a non-positive max_items by @sxh313 in #7367
  • fix(voice): keep text_transforms.replace single-pass across chunks by @sxh313 in #7471
  • fix(background_audio): detach agent_state_changed before closing the player by @maktoufzied in #7488
  • fix(voice): don't hold TTS text behind a * or _ that can never pair up by @avionicharshit-byte in #7520
  • feat: add Microsoft AI speech plugin by @mirimadahmed in #7378
  • microsoft_ai: build the TTS voice ID from a voice name and model by @tinalenguyen in #7572
  • feat(openai): add the gpt-5.6 models to the model registry by @tinalenguyen in #7549
  • (dep): bump livekit to 1.1.20 by @tinalenguyen in #7460
  • livekit-agents@1.8.4 by @github-actions[bot] in #7575

New Contributors

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.8.3...livekit-agents@1.8.4