fix(room_io): stop a finished interruption from dropping the next reply's audio - #2132
Conversation
🦋 Changeset detectedLatest commit: 35dc9e0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
4367a3e to
db8aabf
Compare
db8aabf to
82e5534
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (!this.segmentOpen) { | ||
| this.segmentOpen = true; | ||
| // An interruption raised before this segment began belongs to a speech that is already | ||
| // over. interruptedFuture stays resolved until the next flush, so without this snapshot | ||
| // every frame of the new speech bails at the gate below and the reply is lost. | ||
| this.segmentInterruptCount = this.interruptCount; | ||
| } |
There was a problem hiding this comment.
🟡 Abandoning an open audio segment leaves stale interruption tracking that can silently drop the next reply
The new per-segment flag that decides whether a fresh reply re-reads the interruption counter (segmentOpen at agents/src/voice/room_io/_output.ts:434-440) is cleared only when a segment is flushed, not when a segment is abandoned, so after an abandoned segment the next reply reuses the old segment's interruption snapshot.
Impact: In the rare case where an audio capture fails and its segment is abandoned, a following reply that gets paused can have its remaining audio dropped at the pause gate while still being reported as fully spoken — the exact class of silent audio loss this change set out to fix.
Flag desync between segmentOpen and the base _capturing latch
flush() clears both the base _capturing latch (via super.flush()) and the new segmentOpen flag (agents/src/voice/room_io/_output.ts:523-524). The base class exposes a second segment-close path, abandonOpenSegment() (agents/src/voice/io.ts:227-229), which resets _capturing only. ParticipantAudioOutput does not override it to also reset segmentOpen.
abandonOpenSegment() is invoked on the wrapped output (the ParticipantAudioOutput) by the recorder at agents/src/voice/recorder_io/recorder_io.ts:727 when a downstream capture failed. After that call, the base treats no segment as open (so the next captureFrame counts a new base segment), but segmentOpen is still true, so captureFrame skips the this.segmentInterruptCount = this.interruptCount snapshot for the new segment. If interruptCount advanced in the meantime, interruptCount > segmentInterruptCount holds for the new reply's frames, and any frame parked at the pause gate bails — dropping the reply's audio while playback accounting still treats it as delivered.
To keep the two flags symmetric, segmentOpen should be reset wherever _capturing is (i.e. an abandonOpenSegment() override).
Prompt for agents
The new segmentOpen flag in ParticipantAudioOutput (agents/src/voice/room_io/_output.ts) mirrors the base AudioOutput._capturing latch: both indicate an open segment and both must be cleared when a segment closes. flush() correctly clears both (super.flush() clears _capturing, and the added line clears segmentOpen). However the base class has a second segment-close path, abandonOpenSegment() (agents/src/voice/io.ts), which clears _capturing only. This path is reachable on ParticipantAudioOutput: recorder_io.ts calls this.nextInChain.abandonOpenSegment() on the wrapped audio output after a downstream capture failure. Because ParticipantAudioOutput does not override abandonOpenSegment() to also reset segmentOpen, the next captureFrame will count a new base segment but skip re-snapshotting segmentInterruptCount = interruptCount. If interruptCount advanced between segments, the new reply's frames satisfy interruptCount > segmentInterruptCount and get discarded at the pause gate while still counted as played. Fix by overriding abandonOpenSegment() in ParticipantAudioOutput to call super.abandonOpenSegment() and set this.segmentOpen = false, keeping segmentOpen and _capturing in sync across all segment-close paths.
Was this helpful? React with 👍 or 👎 to provide feedback.
82e5534 to
df471b0
Compare
toubatbrian
changed the title
toubatbrian
changed the title
clearBuffer() resolved an interruptedFuture that was only reset by the *next* segment's flush(), so for the whole of the following reply the signal still described an interruption that was already over. Any frame parking at the pause gate in that window was discarded — frames of a reply that had nothing to do with that interruption. Snapshot the interrupt count when a segment opens and bail only for interruptions raised at or after that point. Consult the snapshot on every frame, not only on frames that parked at a closed gate. cancelSpeechPause un-gates the sink as soon as the handle is interrupted, but the interrupted reply's forwardAudio loop runs for another event loop turn with seconds of TTS audio still to drain. Those frames found the gate open and took the unchecked path to the wire. Co-authored-by: Cursor <cursoragent@cursor.com>
df471b0 to
35dc9e0
Compare
toubatbrian
changed the title
|
Closing: this came out of a misdiagnosis on my side. The reported symptom — the agent's audio and its transcript disagreeing after a barge-in, with the lag compounding over a conversation — is caused by the inference gateway answering a single Measured on its own, #2146's branch contains only The defect it fixes is real and has its own red/green test, and the branch is not deleted. If it ever shows up in practice it is worth reproposing on its own merit rather than as a barge-in fix. |
1 participant
Second of four: #2131 → #2132 → #2136 → #2142. Base: #2131 — review only the top commit; the diff against
mainincludes #2131.Problem
ParticipantAudioOutput'sinterruptedFutureis resolved byclearBuffer()but only reset inwaitForPlayout(). Between those two points the signal is stale-but-armed, and every frame that parks at the pause gate in that window is discarded — including frames belonging to the next reply, which has nothing to do with the interruption that set the flag.Reachable on defaults: with
resumeFalseInterruption: trueandfalseInterruptionTimeout: 2000, the reply after a barge-in loses almost all of its audio while the session still reports it as fully spoken.Fix
Scope the gate to the segment being captured.
clearBuffer()bumps a monotonic count, the first frame after a flush boundary snapshots it, and a frame bails only when the count has advanced past its own segment's snapshot. A finished interruption can no longer reach into the next reply.Testing
false_interruption_audio_loss.test.tsruns a realAgentSessionagainst a realParticipantAudioOutputand counts frames at theAudioSourceboundary: the reply after a barge-in delivers 2 of 20 frames onmain, 20 of 20 here. Plus two_output.test.tsunit tests that also fail onmain.Python parity
Cannot happen in Python — there is no pause gate on
capture_frame, and bothclear_bufferandflushbail onif not self._pushed_duration, so a stale signal is self-healing after at most one frame. JS broke that symmetry by leavingclearBuffer()unguarded whileflush()stayed guarded.Scope — read this before merging
Written while investigating a live report of agent audio running behind the transcript after a barge-in. That report is now traced elsewhere: the gateway emits multiple
donemessages persession.flushon some TTS paths and the client returns on the first, truncating the reply and handing a still-streaming socket back to the pool. Fixes are in flight separately (#2144, #2146).#2124, which this closes, is a distinct symptom with its own reproduction — a whole reply of which the user hears the first fraction of a second. The report above still reproduces with the whole stack applied, so do not merge this as a fix for it.
Closes #2124.