fix(inference): treat a TTS gateway done as a generation boundary - #2146
fix(inference): treat a TTS gateway done as a generation boundary#2146toubatbrian wants to merge 1 commit into
done as a generation boundary#2146Conversation
🦋 Changeset detectedLatest commit: 21e00a1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c78345f73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const unplayedMs = | ||
| firstEmittedAt === undefined | ||
| ? 0 | ||
| : Math.max(0, emittedAudioMs - (Date.now() - firstEmittedAt)); | ||
| return Math.min(DRAIN_IDLE_TIMEOUT, unplayedMs); |
There was a problem hiding this comment.
When a gateway streams a generation at roughly playback speed, emittedAudioMs grows at the same rate as Date.now() - firstEmittedAt, so unplayedMs is zero or nearly zero when done arrives. The next read therefore times out immediately instead of covering the documented inter-generation gap, finalizes after the first generation, and can recycle the socket while later audio is still coming. The new test sends each generation as a synchronous burst, so it does not exercise this common streaming cadence.
Useful? React with 👍 / 👎.
| if (wasDraining) { | ||
| await finalize(); | ||
| return; |
There was a problem hiding this comment.
session.closed
When the gateway sends session.closed after a done, finalize() sets sessionDrained, causing the surrounding cleanup to skip pool.remove(ws) and withConnection to return this websocket to the pool. The server-side session is explicitly closed, however, and a pooled connection does not send another session.create; the next synthesis therefore sends its transcript and flush to a dead logical session and may wait for the receive timeout before retrying. The current reply can finish normally here, but this websocket must not be reused.
Useful? React with 👍 / 👎.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
9c78345 to
f854d37
Compare
Several gateway providers answer one `session.flush` with more than one generation, splitting at roughly 1kB of text or 35s of audio and sending a `done` after each. A probe of `inworld/inworld-tts-2` saw 6 `done` events for a single flush, the first covering 39.4s of the 201.0s the session went on to produce. Stopping at the first `done` cut replies off after 20-80% of their audio while the transcript committed all of it, and released the websocket into the ConnectionPool mid-synthesis so the next reply spoke the previous one's leftover audio. The FIFO session is never reset, so each reply queued behind the last one's unsynthesized text and the lag compounded. Keep reading after `done` and end the flush only once the session has stayed quiet. The wait is capped by how much synthesized audio is still unplayed, so it is hidden behind playout and collapses to near zero on replies too short for the gateway to split. A terminal event that arrives during that wait is judged on what it actually proves. `session.closed` still ends the reply normally, but its socket is evicted instead of pooled: `session.create` is only ever sent when a socket is opened, so a pooled socket would keep the closed session and the next reply would stall inside it. An `error` no longer resolves the reply, because a `done` is only a candidate end and a provider that fails while preparing the next generation has left the reply unfinished; a genuinely finished reply is protected from that retry by the idle timeout, which resolves the run before a later error can be read. The cap on the wait is a known limitation: a gateway streaming at roughly playback speed leaves nothing buffered, so the wait collapses to about zero and the first `done` ends the flush. It is documented and pinned by a test rather than lengthened — the bound is what keeps this off the turn latency path, and the providers that do split a flush run far faster than realtime. Co-authored-by: Cursor <cursoragent@cursor.com>
f854d37 to
21e00a1
Compare
|
Closing: the multi- The gateway's own design notes treat Against that, the cost is real: it diverges from Python, which breaks on the first #2144 is unaffected and still wanted — a session dropped mid-synthesis being reported as a completed reply is an independent defect, and fixing it restores Python parity rather than breaking it. Reopen if it turns out multiple |
1 participant
Stacked on #2144, which gates socket reuse on having seen a
done; this PR changes what adonemeans. Review only this commit.Problem
Several gateway providers answer one
session.flushwith more than one generation, splitting atroughly 1kB of text or 35s of audio and sending a
doneafter each. A probe ofinworld/inworld-tts-2got 6dones for one flush, the first covering 39.4s of the 201.0s thesession went on to produce.
Stopping at the first
donecut every reply longer than one generation off after 20-80% of itsaudio while the transcript committed all of it, and released the socket into the
ConnectionPoolmid-synthesis, so the next reply spoke the previous one's leftover audio. Thesession is FIFO and never reset, so each reply queued behind the last one's unsynthesized text
and the lag compounded.
Fix
doneis a generation boundary: keep reading, and end the flush only once the session has gonequiet. A terminal event arriving during that wait is judged on what it proves —
session.closedstill ends the reply but evicts the socket (session.createis only sent whena socket opens, so a pooled socket would keep the closed session), and an
errorfails theattempt, because a
donenever promised the reply was complete.Testing
A fake gateway splitting one flush into three
done-terminated generations under one sessionid: on
mainthe reply speaks 1 of 3 and the next reply speaks the first reply's secondgeneration. End to end, the barge-in scenario goes from 6 of 10 replies never spoken and the
transcript 26.8s ahead of the audio to 0ms of lag and 1-2 never spoken, in 3 of 3 runs.
Known limit
The wait is capped by how much synthesized audio is still unplayed, so it hides behind playout
and costs no turn latency. A provider streaming at ~1x realtime has nothing buffered, so the
wait collapses to zero and truncation persists there. Pinned by a test rather than lengthened:
the providers that do split a flush run far faster than realtime.
agent-gateway#1105fixes theInworld path server-side.
Python parity
inference/tts.pyhas the identical defect —end_input(); breakon the firstdone— andneeds the same change with the grace in seconds. Filed separately.