fix(interruption): don't drop committed audio at the transport send boundary

[toubatbrian]

Note

Stacked on #2114. This branch is rebased on top of brian/fix-recorder-segment-ordering-deadlock
(#2114), so the intended merge order is main → #2114 → #2116 → #2117. GitHub's stacked-PR
feature refuses to change the base of a PR that is already part of a stack
(Cannot change the base branch because the pull request is part of a stack), so this PR still
reports main as its base and its diff therefore also shows #2114's recorder commits.
Review only the agents/src/inference/interruption/* and agents/src/voice/events.ts changes here;
the recorder changes belong to #2114.

What

createWsTransport's transform re-read the overlap state immediately before writing to the socket:

if (reconnecting) await reconnecting;

// Only forwards buffered audio while overlap speech is actively on.
const state = getState();
if (!state.overlapSpeechStartedAt || !state.overlapSpeechStarted) return;

That check is redundant with the one in interruption_stream.ts, which already only cuts and enqueues a slice while overlapSpeechStarted is true. The transport's copy runs after an await, and three things can clear the flag inside that window — the overlap-speech-ended and agent-speech-ended sentinels, and bargein_detected (via setState({ overlapSpeechStarted: false })). When that happens, audio the pipeline had already committed to sending is dropped and never counted in numRequests, which is the only code path that can produce numRequests: 0 and force the hardcoded backchannel fallback.

This removes the send-time gate. The upstream buffering stage stays the single place that decides whether a slice belongs to an overlap.

This is not a confirmed reproduction of the reported "adaptive interruption always classifies as backchannel" bug. That bug could not be reproduced on main in a live session. The ordering hazard here is real and is the only known path to numRequests: 0, but note that during a multi-second overlap the transport sends a slice every 100 ms, so this can realistically drop only the trailing slice(s) at an overlap boundary — it does not on its own explain an overlap that made zero requests. Treat it as a correctness fix and a Python-alignment fix, not as a proven root cause.

Why removal, not a narrower fix

Python's send_task has no equivalent gate — it sends every slice the buffering stage hands it and gates only on the receive side:

async for audio_data in input_ch:
    ...
    await self._num_requests.increment()
    created_at = perf_counter_ns()
    header = struct.pack("<Q", created_at)
    await ws.send_bytes(header + audio_data.tobytes())

The gate is not load-bearing:

• Late responses are already handled. Both bargein_detected and inference_done are guarded by state.overlapSpeechStarted && overlapSpeechStartedAt !== undefined in handleMessage, so audio sent after an overlap closed cannot produce a verdict.
• No stale-cache hazard. A send outside an overlap leaves an unresolved cache entry, which the timeout loop would otherwise trip on. But every new send is necessarily preceded by an overlap-speech-started sentinel, which calls cache.clear() — so a stale entry can never be the oldest unanswered one at the next real send.
• Cost is one or two extra inference frames per overlap, which Python already accepts.

The WebSocket not connected backstop inside sendAudioData is untouched.

Observability

A backchannel verdict was previously indistinguishable from "we never asked the model."

• The no-result case moves from debug to warn and carries overlap duration, numRequests, buffered samples, agentSpeechStarted and agentEnded, so a user hitting the fallback sees it in normal logs.
• Every verdict logs probability, isInterruption and numRequests at debug, once per overlap, so "model scored it low" is separable from "the model was never asked."
• OverlappingSpeechEvent is exported by name from voice/events.ts. The fields already reached user code — agent_activity.ts forwards the event object verbatim to AgentSessionEventTypes.OverlappingSpeech — but the type could not be imported to write a typed handler.

No threshold knob was added; the threshold is a server-side decision.

Test coverage

There was no test anywhere covering the audio-to-verdict path, which is why this class of bug is currently unfalsifiable. interruption_pipeline.test.ts drives real room-rate 48 kHz audio through AudioRecognition into a mocked gateway and asserts on classification outcome: requests get sent during an overlap, a bargein verdict becomes isInterruption: true and reaches onInterruption, and repeated overlaps in one agent turn keep working.

Two tests target this fix specifically:

• sends a slice committed during an overlap that ends while a reconnect is in flight — parks the transport on await reconnecting with a slice in hand, ends the overlap, then completes the reconnect. This is the exact hazard and it is red before the change (expected +0 to be 1).
• forwards a slice regardless of the overlap flag at send time — pins the send-side contract directly at the transport seam.

Red before, green after:

 × ... > sends a slice committed during an overlap that ends while a reconnect is in flight
   → expected +0 to be 1
 × ... > forwards a slice regardless of the overlap flag at send time
   → expected +0 to be 1
 Tests  2 failed | 3 passed (5)

 ✓ ... > sends a slice committed during an overlap that ends while a reconnect is in flight 107ms
 ✓ ... > forwards a slice regardless of the overlap flag at send time 34ms
 Tests  5 passed (5)

Test plan

• pnpm build exits 0
• npx vitest run --dir agents/src — 110 files, 1463 passed
• pnpm lint (no new findings on touched files)
• pnpm format:check
• pnpm api:check — fails identically on unmodified main (The "export * as ___" syntax is not supported yet), pre-existing and unrelated

Made with Cursor

[changeset-bot]

🦋 Changeset detected

Latest commit: 537d374

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[toubatbrian]

Closing this one because the mechanism it fixes was shown to be unreachable.

The change removes the send-time overlap gate in ws_transport.ts on the reasoning that overlap-speech-ended, agent-speech-ended or bargein_detected can clear overlapSpeechStarted inside the await reconnecting window and discard a slice the buffering stage had already committed to. Instrumenting both ends of the pipe — slices cut by the buffering stage versus slices written to the socket — and running the pipeline with the gate present and with it removed:

Scenario	Gate present	Gate removed
Realistic track cadence (10ms frames, 8s overlap)	59 cut / 59 sent / 0 dropped	59 cut / 59 sent / 0 dropped
Maximum burst (no inter-frame delay, 1s overlap)	59 cut / 59 sent / 0 dropped	59 cut / 59 sent / 0 dropped
bargein_detected clearing the flag mid-stream	3 cut / 3 sent / 0 dropped	3 cut / 3 sent / 0 dropped

Zero drops either way. The reason is structural: transform() only parks on await reconnecting, and reconnecting is only set by reconnectWatcher, which only fires on reconnectEvent — reachable solely through InterruptionStreamBase.updateOptions(), whose only caller is the public AdaptiveInterruptionDetector.updateOptions(). Nothing in the framework calls that, so the gate can only drop audio if an application calls updateOptions() while an overlap is open.

The overlapGeneration tagging goes with it: it only becomes necessary once the gate is removed, so it fixes a mis-attribution the same PR introduces.

The reported symptom that motivated this work turned out to be #2123 — the Krisp plugin pulled in a second copy of @livekit/rtc-node through createRequire, so frames failed instanceof AudioFrame and the classifier received zero audio.

Two pieces of this branch are worth keeping and have been carried into #2126: the end-to-end interruption_pipeline.test.ts harness (minus the interruption transport send gate describe block) and the OverlappingSpeechEvent type re-export in voice/events.ts. The recorder commits were superseded by #2114, which merged a superset.

Closing in favour of #2126.

[toubatbrian]

Redirect, and a correction: the actual change in this PR is not being shipped. Only the test harness and the OverlappingSpeechEvent type re-export were salvaged, into #2131.

The reasoning for removing the ws_transport.ts overlap gate was that overlap-speech-ended, agent-speech-ended or bargein_detected could clear the flag inside the await reconnecting window and discard a slice the buffering stage had already committed to. Instrumenting both ends of the pipe — slices cut versus slices written to the socket — under three scenarios, with the gate and without:

Scenario	Gate present	Gate removed
Realistic track cadence (10ms frames, 8s overlap)	59 cut / 59 sent / 0 dropped	59 cut / 59 sent / 0 dropped
Maximum burst (no inter-frame delay, 1s overlap)	59 cut / 59 sent / 0 dropped	59 cut / 59 sent / 0 dropped
bargein_detected clearing the flag mid-stream	3 cut / 3 sent / 0 dropped	3 cut / 3 sent / 0 dropped

Zero drops either way, and the reason is structural: transform() only parks on await reconnecting, which is only ever set via InterruptionStreamBase.updateOptions() — nothing in the framework calls it. The gate can only drop audio if an application calls updateOptions() while an overlap is open.

The follow-up overlapGeneration tagging is excluded for the same reason: it exists to stop a slice sent after its overlap closed from being charged to the next overlap's numRequests, which can only happen once the gate is gone. It fixes a mis-attribution this PR introduces.

This was written while every barge-in was being classified as a backchannel. That turned out to be #2123 — the Krisp plugin loading a second copy of @livekit/rtc-node, so instanceof AudioFrame failed and the classifier was fed zero audio. This PR was chasing that symptom down a path that was never broken.