fix(voice): rebuild playback bridge after idle to stop slow new turns (#1816)

* fix(voice): rebuild playback bridge after idle to stop slow new turns

Assistant playback routes through a MediaStreamAudioDestinationNode ->
HTMLAudioElement bridge so a selected output device can be honored. That
bridge is a live playout path, and reusing the element for a new turn after
it sat idle between turns made the new turn resume at the wrong rate (audible
as slow-motion that re-converges over the turn).

Tear the bridge down and rebuild it once it has fully drained and been idle
past a short threshold, so each turn plays through a fresh element. The
scheduledSources.size === 0 guard keeps this from firing mid-turn.

Adds regression tests covering within-turn reuse, after-idle rebuild, and
sub-threshold reuse.

* fix(voice): keep idle-rebuild working after a playback interrupt

#stopPlayback reset #lastPlaybackEnd to null, which disabled the post-idle
bridge rebuild for the next turn (the guard requires a non-null value). An
interrupt leaves the bridge idle just like a natural drain, so interrupt ->
long pause -> new turn could reproduce the slow playback. Record the audio
clock time at interrupt as the idle start instead, so the next turn still
rebuilds. Adds a regression test for the interrupt-then-idle path.

Author: cjol

.changeset/voice-playback-bridge-idle-rebuild.md

@@ -0,0 +1,5 @@
+---
+"@cloudflare/voice": patch
+---
+
+Fix assistant speech playing back slow on a new turn after an idle gap. `VoiceClient` routes playback through a `MediaStreamAudioDestinationNode` -> `HTMLAudioElement` bridge, and reusing that element for a fresh burst after it had been idle between turns made the new turn resume at the wrong rate (audible as slow-motion that re-converges to normal over the turn). The bridge is now torn down and rebuilt once it has fully drained and been idle past a short threshold, so each turn plays through a freshly created element. Rebuilds never happen mid-turn, since chunks within a turn keep at least one source scheduled on the playback cursor.

packages/voice/src/tests/voice-client.test.ts

@@ -732,3 +732,99 @@ describe("VoiceClient gapless playback", () => {
     expect(sources[1].startedAt).toBe(2);
   });
 });
+
+describe("VoiceClient playback bridge reuse", () => {
+  // The audible symptom (a new turn plays back slow, then re-converges) lives
+  // inside the HTMLAudioElement's playout and is not observable from JS, so we
+  // cannot assert playback speed. Instead we lock in the mechanism the fix
+  // relies on: a bridge that has gone idle between turns is rebuilt, not reused,
+  // and is never rebuilt mid-turn.
+  function pcm16Chunk(): ArrayBuffer {
+    // 1600 samples of 16-bit PCM = 0.1s at 16kHz
+    return new ArrayBuffer(1600 * 2);
+  }
+
+  function startPcm16Call(): { transport: MockTransport; client: VoiceClient } {
+    const transport = new MockTransport();
+    const client = new VoiceClient({ agent: "test-agent", transport });
+    client.connect();
+    transport.receive(
+      JSON.stringify({ type: "audio_config", format: "pcm16" })
+    );
+    return { transport, client };
+  }
+
+  it("reuses the playback bridge for consecutive chunks within a turn", async () => {
+    const { transport } = startPcm16Call();
+    audioContext.currentTime = 5;
+
+    transport.receive(pcm16Chunk());
+    transport.receive(pcm16Chunk());
+    await waitForSourceCount(2);
+
+    // Chunks within a turn stay scheduled ahead on the cursor, so the bridge
+    // must not be torn down between them.
+    expect(audioContext.mediaStreamDestinationCount).toBe(1);
+    expect(audioElement.playCount).toBe(1);
+  });
+
+  it("rebuilds the playback bridge for a new turn after it has gone idle", async () => {
+    const { transport } = startPcm16Call();
+
+    // Turn 1: one chunk ending at t=5.1.
+    audioContext.currentTime = 5;
+    transport.receive(pcm16Chunk());
+    const [first] = await waitForSourceCount(1);
+    expect(audioContext.mediaStreamDestinationCount).toBe(1);
+
+    // Turn 1 finishes playing and the bridge drains.
+    first.stop();
+
+    // A gap longer than the idle threshold passes before the next turn.
+    audioContext.currentTime = 5.5; // 0.4s past the last chunk's end (5.1)
+    transport.receive(pcm16Chunk());
+    await waitForSourceCount(2);
+
+    // The idle bridge is torn down and a fresh one is built for turn 2.
+    expect(audioContext.mediaStreamDestinationCount).toBe(2);
+    expect(audioElement.playCount).toBe(2);
+  });
+
+  it("rebuilds the playback bridge for a new turn after an interrupt and idle gap", async () => {
+    const { transport } = startPcm16Call();
+
+    // Turn 1 starts playing, then is interrupted mid-playback (abrupt stop,
+    // not a natural drain).
+    audioContext.currentTime = 5;
+    transport.receive(pcm16Chunk());
+    await waitForSourceCount(1);
+    expect(audioContext.mediaStreamDestinationCount).toBe(1);
+    transport.receive(JSON.stringify({ type: "playback_interrupt" }));
+
+    // A gap longer than the idle threshold passes before the next turn.
+    audioContext.currentTime = 6; // 1s after the interrupt
+    transport.receive(pcm16Chunk());
+    await waitForSourceCount(2);
+
+    // The bridge went idle at the interrupt, so the next turn must rebuild it.
+    expect(audioContext.mediaStreamDestinationCount).toBe(2);
+    expect(audioElement.playCount).toBe(2);
+  });
+
+  it("reuses the playback bridge when the next turn follows within the idle threshold", async () => {
+    const { transport } = startPcm16Call();
+
+    audioContext.currentTime = 5;
+    transport.receive(pcm16Chunk());
+    const [first] = await waitForSourceCount(1);
+    first.stop();
+
+    // Next chunk arrives only 0.1s after the previous chunk's end (< 0.3s).
+    audioContext.currentTime = 5.2;
+    transport.receive(pcm16Chunk());
+    await waitForSourceCount(2);
+
+    expect(audioContext.mediaStreamDestinationCount).toBe(1);
+    expect(audioElement.playCount).toBe(1);
+  });
+});

packages/voice/src/voice-client.ts

@@ -296,6 +296,10 @@ export class VoiceClient {
   #isScheduling = false;
   #scheduledSources = new Set<AudioBufferSourceNode>();
   #playbackCursor = 0;
+  // End time (on the AudioContext clock) of the most recently scheduled chunk.
+  // Used to detect when the playback bridge has been idle across the gap
+  // between turns so it can be rebuilt (see #playAudio).
+  #lastPlaybackEnd: number | null = null;
   #playbackElement: HTMLAudioElement | null = null;
   #playbackDestination: MediaStreamAudioDestinationNode | null = null;
   #playbackDestinationPromise: Promise<AudioNode> | null = null;
@@ -905,6 +909,28 @@ export class VoiceClient {
       }
       if (generation !== this.#playbackGeneration) return;
 
+      // Assistant playback is routed through a MediaStreamAudioDestinationNode
+      // -> HTMLAudioElement bridge (so a selected output device can be honored
+      // via HTMLMediaElement.setSinkId). That bridge is a live playout path:
+      // when it is reused for a new turn after sitting idle through the gap
+      // between turns, the element resumes the fresh burst at the wrong rate and
+      // the new turn plays back noticeably slow (it then re-converges to normal
+      // over the turn). A freshly created element does not show this, so once
+      // the bridge has fully drained and been idle past a short threshold, tear
+      // it down; #getPlaybackDestination rebuilds a fresh element for this turn.
+      // The scheduledSources.size === 0 guard means this never fires mid-turn:
+      // chunks within a turn are scheduled ahead on the cursor, keeping at least
+      // one source live until the turn finishes.
+      const BRIDGE_IDLE_REBUILD_SECONDS = 0.3;
+      if (
+        this.#playbackElement &&
+        this.#scheduledSources.size === 0 &&
+        this.#lastPlaybackEnd !== null &&
+        ctx.currentTime - this.#lastPlaybackEnd > BRIDGE_IDLE_REBUILD_SECONDS
+      ) {
+        this.#closePlaybackOutput();
+      }
+
       const destination = await this.#getPlaybackDestination(ctx);
       if (generation !== this.#playbackGeneration) return;
 
@@ -930,6 +956,7 @@ export class VoiceClient {
       // a periodic click during agent speech.
       const startAt = Math.max(ctx.currentTime, this.#playbackCursor);
       this.#playbackCursor = startAt + audioBuffer.duration;
+      this.#lastPlaybackEnd = this.#playbackCursor;
       source.start(startAt);
     } catch (err) {
       console.error("[VoiceClient] Audio playback error:", err);
@@ -973,6 +1000,15 @@ export class VoiceClient {
     this.#isPlaying = false;
     this.#isScheduling = false;
     this.#playbackCursor = 0;
+    // An interrupt (barge-in or a user transcript) stops playback abruptly, so
+    // the bridge goes idle from now, not from the cut chunk's scheduled end.
+    // Record the current audio-clock time as the idle start so the next turn's
+    // rebuild check in #playAudio still fires after interrupt -> long pause ->
+    // new turn. Nulling this here would disable that check and let the slow-
+    // playback symptom reappear on the post-interrupt turn.
+    this.#lastPlaybackEnd = this.#audioContext
+      ? this.#audioContext.currentTime
+      : null;
   }
 
   // --- Mic capture ---