fix(transcription): stop the synchronizer reporting the previous reply's transcript after an interrupt - #2141
Closed
toubatbrian wants to merge 1 commit into
Closed
fix(transcription): stop the synchronizer reporting the previous reply's transcript after an interrupt#2141toubatbrian wants to merge 1 commit into
toubatbrian wants to merge 1 commit into
Conversation
…fter an interruption An interrupted reply whose playback_finished arrives before its text forwarding flushes leaves a rotated segment that looks text-ended while carrying no audio. The next reply's first chunk mistook it for a segment awaiting an in-flight finish and queued it, but nothing could ever settle an entry that owes no finish. From then on every real finish was consumed to settle the segment queued a turn earlier, so each reply reported the previous reply's transcript and the lag never recovered for the rest of the session. Only queue a rotated-out segment when it genuinely still owes a playback finish. Co-authored-by: Cursor <cursoragent@cursor.com>
🦋 Changeset detectedLatest commit: a136f8f The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
toubatbrian
changed the title
toubatbrian
changed the title
Contributor
Author
|
Closing: this came out of a misdiagnosis on my side. The reported symptom — the agent's audio and its transcript disagreeing after a barge-in, with the lag compounding over a conversation — is caused by the inference gateway answering a single Measured on its own, #2146's branch contains only The defect it fixes is real and has its own red/green test, and the branch is not deleted. If it ever shows up in practice it is worth reproposing on its own merit rather than as a barge-in fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SegmentSynchronizercan get permanently stuck one segment behind. After an interrupt, every later reply is reported, committed to the chat context, and shown to the user with the previous reply's text while a different reply is the one actually being spoken; late in a session the reported text degrades further, to empty.This is a text-path defect — the audio is correct throughout, only the transcript is wrong. It is not the production report of agent audio falling behind the transcript (see Scope).
Cause
On a barge-in a reply's
playback_finishedusually arrives before its text-forwarding task flushes. The finish rotates the segment, so the lateflush()ends text input on the new impl instead. The next reply's first text chunk sees that impl markedtextInputEndedand queues it in_pendingRotatedSegments, assuming its playback finish must still be in flight — but that impl never carried audio and is owed no finish, so nothing settles it. From then on each real finish is consumed settling the segment queued a turn earlier, which re-queues the current one and makes the off-by-one self-sustaining.Fix
Add
SegmentSynchronizerImpl.owesPlaybackFinished(carried audio downstream, not yet marked finished) and only enqueue a rotated-out segment when it is true. That is exactly the precondition the queue's own comment asserts, so the genuine interrupt-plus-fast-next-reply race it was built for still enqueues.Testing
Two regression tests, red before / green after. Voice suite green at 566 tests. The three existing
_pendingRotatedSegmentstests pass unchanged — notably the one covering the case the queue was introduced for.Scope — read this before merging
Written while investigating a live report of agent audio running behind the transcript after a barge-in. That report is now traced elsewhere: the gateway emits multiple
donemessages persession.flushon some TTS paths and the client returns on the first, truncating the reply and handing a still-streaming socket back to the pool. Fixes are in flight separately (#2144, #2146). The two defects share a vocabulary and are opposites: this one misreports text against correct audio, that one plays wrong audio against a correct transcript.Also independent of the interruption stack (#2131 → #2142) — reproduces on
mainregardless of which of those is checked out. Python's_SyncedAudioOutput.on_playback_finishedonly ever reads the current impl and so cannot report another segment's text; this is a JS-only divergence.