Agent transitions `thinking → speaking` after user has already transitioned to `speaking` (stale reply plays over new user turn)

[miguelmoralai]

Summary

We are reliably reproducing a race on livekit-agents ~= 1.5.2: the agent enters speaking and begins TTS playout even though the UserStateChangedEvent for listening → speaking has already fired milliseconds earlier. The stale reply gets played over the user's new turn, causing both sides to talk at once and typically resulting in a bad interruption/EOT resolution.

We shipped a workaround on our side that works, but we believe the SDK should guard this internally — once the session has observed the user transitioning into speaking, it shouldn't let a reply that is still in thinking (no audio emitted yet) promote to speaking.

The bug is visible when turn detection is driven by both VAD and STT.

Reproduction — real call logs

In this call the LLM generation for the previous user turn had just started when the user began a new turn. The SDK still flipped the agent to speaking:

02:43:58.700  Agent state changed: listening → thinking
02:43:59.094  VAD speech detected (triggering_probability=0.90)
02:43:59.095  User state changed: away → speaking
02:43:59.144  VAD waiting for EOS (probability=0.92, min_silence_required=1.10s)
02:43:59.264  Agent state changed: thinking → speaking   ← should not happen
02:43:59.299  ElevenLabs TTS request completed

Between the user going speaking (02:43:59.095) and the agent going speaking (02:43:59.264) there is ~170 ms — plenty of time for the SDK to observe the user state transition and cancel the pending generation.

A full chat history + traces + logs bundle from one of these calls is available on request — we didn't want to paste customer data in a public issue.

Our workaround

In our user_state_changed handler, when the new state is speaking and session.agent_state == "thinking", we call session.interrupt() to cancel the in-flight generation before it can promote to speaking. The committed user turn's on_end_of_turn path then regenerates naturally against the new transcript.

Guardrails we had to add:

• Skip when current_speech.num_steps > 1 — otherwise we cancel the follow-up generation that runs after a tool call (e.g. agent handoff).
• Skip when current_speech.allow_interruptions is False — SpeechHandle.interrupt(force=False) raises RuntimeError for non-interruptible speeches (our initial greeting, mandatory compliance messages, etc.).
• Wrap session.interrupt() in try/except RuntimeError as defence in depth for non-interruptible speeches sitting in the scheduling queue.

How often we see it

We added a metric (agent.thinking_interrupted_by_user) that increments every time our workaround fires. Across 72 completed calls with ~51s average duration, the metric fired 127 times — so this is not a rare edge case, it fires multiple times per call in some conversations. Since shipping the workaround we haven't seen the bug in call reviews.

Why we think this belongs in the SDK

• Every agent hitting this race has to reimplement the same user_state_changed listener with the same guardrails (tool-call follow-ups, non-interruptible speeches). That logic is easy to get wrong.
• The SDK already knows the user's state and the agent's state — it is in a much better position than application code to break the tie atomically.
• Semantically, if the user is already in speaking when a reply is still in thinking (nothing audible yet), promoting it to speaking is never the right call.

Suggested fix direction

Inside the session scheduler, before allowing a speech to transition from thinking → speaking, check whether the user is currently in speaking state. If yes, cancel the pending speech (same semantics as session.interrupt() does today) rather than starting TTS playout. Respect allow_interruptions=False and the tool-call follow-up case the same way we had to externally.

If maintainers agree with this direction, happy to open a PR.

Versions

livekit-agents ~= 1.5.2 (with deepgram/soniox/silero/turn-detector extras)
livekit-api >= 1.1.0, < 2

[longcw]

do you mean the agent interrupted the user? a minimal reproducible example and the full logs around that time should be helpful for us to understand the issue.

[miguelmoralai]

Yes — the agent ends up speaking while the user is already in speaking state. Whether you call that "interrupting" is a naming question, but the outcome is the same: the agent starts TTS on top of the user.

Here's a fresh minimal repro (same bug, different call). The turns involved happen between 00:47 and 00:50 of the call:

00:47  User   Okay, no, no, no, no.
00:50  Agent  Vale, si prefieres no continuar, no
00:50  User   No hace falta. 678.

Corresponding log window:

17:20:08.766  User state changed: speaking → listening (spoke for 2.0s)
17:20:08.769  livekit.agents: stt end of speech received while user is speaking, resetting vad
17:20:08.801  Agent state changed: listening → thinking
17:20:09.299  VAD speech detected (triggering_probability=0.78)
17:20:09.299  User state changed: listening → speaking
17:20:09.423  Agent state changed: thinking → speaking        ← ~124 ms after user went speaking
17:20:09.628  ElevenLabs TTS request completed

The user briefly paused mid-sentence (normal silence inside a turn), STT fired an end-of-speech, the session flipped to thinking, and ~500 ms later the user resumed — VAD picked it up and the SDK logged User state changed: listening → speaking at 17:20:09.299. Despite that, the session still promoted its pending speech to speaking at 17:20:09.423, 124 ms after the user state change was already observed. TTS playout then started over the user's continuation.

Full session for maintainer visibility:

• Session ID: RM_WvB2nZqdJ5ZZ
• LiveKit Cloud: https://cloud.livekit.io/projects/p_2tsga5unx60/sessions/RM_WvB2nZqdJ5ZZ/observability?mode=logs

On frequency: this is the dominant failure mode for mid-sentence pauses in our traffic. The metric we added (agent.thinking_interrupted_by_user, which fires every time our workaround cancels one of these stale replies) showed 127 hits across 72 calls (~51s average duration). So it fires more than once per call on average, and it's specifically triggered by normal in-turn silences — exactly when the user shouldn't be getting interrupted.

[longcw]

thanks for sharing the logs.

I see the issue now, user state changed to speaking when VAD detected speech duration > min_speech_duration (default 0.05s), and the VAD will pause/interrupt the agent when speech duration > InterruptionOptions.min_duration (default 0.5s).

That's why the agent speaking still started after 124ms the user state changed to speaking. One thing we can do is to pause the agent speech once the user state is speaking (so the threshold is 0.05s instead of 0.5s), if it's a false positive, the agent speech can still be resumed.

[miguelmoralai]

Thanks, that matches what we see. Your proposal would fix it cleanly.

To confirm the mechanism: user state → speaking is VAD-driven via min_speech_duration = 0.05s, but the session only pauses agent speech once VAD duration crosses InterruptionOptions.min_duration = 0.5s. That ~450ms gap is exactly where the stale reply promotes from thinking to speaking.

What we do in our workaround: when user state → speaking fires and the agent is still in thinking, we cancel the pending turn. We only act while the agent is in thinking — never once it's already in speaking — so we avoid cutting off agent speech that's already being played out. The normal interruption path handles that case.

[longcw]

What we do in our workaround: when user state → speaking fires and the agent is still in thinking, we cancel the pending turn. We only act while the agent is in thinking

Yeah I think it makes sense. The only risk is when you interrupt a speech in thinking state, if the user speaking is a false positive and never trigger a new agent reply, the agent will be idle until next user input.

[miguelmoralai]

Good call — we hit that exact risk. Our workaround: arm a safety-net after the interrupt. If STT emits an empty final transcript (explicit "no speech" signal) we regenerate. A 3s backstop covers the case where STT emits nothing at all. Real speech (any non-empty transcript) disarms it. Would be great to see this logic land in the SDK.