Skip to content

feat(voice): add output device selection - #1711

Merged
threepointone merged 5 commits into
mainfrom
audio-picker
Jun 12, 2026
Merged

feat(voice): add output device selection#1711
threepointone merged 5 commits into
mainfrom
audio-picker

Conversation

@cjol

@cjol cjol commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This PR adds output device selection for @cloudflare/voice playback and wires the voice-agent example to use it. Fixes #1550.

Why

Users need call audio to follow the speaker selected in their app UI instead of always playing through the system default output. The SDK previously played decoded audio directly to AudioContext.destination, which left consumers no path to apply HTMLMediaElement.setSinkId().

The implementation routes SDK playback through a hidden HTMLAudioElement, then applies sink selection to that element. This avoids relying on AudioContext.setSinkId(), which has weaker cross-browser support, while keeping playback alive on the default or previous output when sink selection is unsupported or rejected.

Public API Surface

  • Adds outputDeviceId?: string to VoiceClientOptions, inherited by useVoiceAgent() options:
    const client = new VoiceClient({
      agent: "my-voice-agent",
      outputDeviceId: selectedSpeaker.deviceId
    });
    
    const voice = useVoiceAgent({
      agent: "my-voice-agent",
      outputDeviceId
    });
  • Adds VoiceClient.setOutputDevice(outputDeviceId?: string): Promise<void> to update the active client without reconnecting the call.
  • Adds VoiceClient.outputDeviceError and an outputdeviceerror event for non-fatal speaker-routing failures.
  • Adds outputDeviceError to useVoiceAgent() so React apps can render speaker-routing warnings separately from call/connection errors.
  • Treats undefined and "default" as default output.

Architectural Changes

Playback now flows through AudioBufferSourceNode -> MediaStreamAudioDestinationNode -> hidden HTMLAudioElement instead of connecting assistant audio directly to AudioContext.destination. The hidden media element becomes the output-routing boundary for browser sink selection. If that setup fails, playback falls back to AudioContext.destination.

useVoiceAgent() applies outputDeviceId changes to the existing VoiceClient, so changing speakers does not reconnect the WebSocket or restart the call.

Output-device failures are intentionally separated from the global error channel because playback can continue even when routing to a selected speaker fails. Apps can show outputDeviceError near a speaker picker while keeping global error reserved for call, mic, connection, and server failures.

Reliability and Browser Behavior

  • Serializes playback-output initialization so startCall() pre-warm and early audio chunks share one hidden media element setup.
  • Guards teardown during async media-element startup so ended calls do not orphan audio routing.
  • Handles rapid setOutputDevice() calls by preserving the latest requested device even when setSinkId() promises resolve out of order.
  • Continues playback on the default or previous output when setSinkId() is unsupported or rejected.
  • Documents browser support caveats and the fact that output device labels may be blank until microphone permission is granted.

Code Changes

  • Refactored VoiceClient playback output management and cleanup.
  • Added sink selection, runtime output switching, and dedicated output-device warning state in VoiceClient.
  • Updated useVoiceAgent() to apply output device changes without reconnecting and expose outputDeviceError.
  • Added an output picker to the voice-agent example call controls with inline routing warnings.
  • Updated package README, docs, and example README for output-device selection.
  • Added unit and React hook tests for sink routing, fallback behavior, no-reconnect updates, warning separation, race handling, and cleanup.
  • Added a changeset for @cloudflare/voice.

Validation

  • pnpm --filter @cloudflare/voice test
  • pnpm --filter @cloudflare/voice test:react
  • pnpm run build
  • pnpm run check

Open in Devin Review

@changeset-bot

changeset-bot Bot commented Jun 9, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d7a8c82

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@cloudflare/voice Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new

pkg-pr-new Bot commented Jun 9, 2026

Copy link
Copy Markdown

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1711

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1711

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1711

create-think

npm i https://pkg.pr.new/create-think@1711

hono-agents

npm i https://pkg.pr.new/hono-agents@1711

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1711

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1711

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1711

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1711

commit: d7a8c82

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

Open in Devin Review

Comment thread packages/voice/src/voice-client.ts
cjol and others added 5 commits June 12, 2026 09:23
Separate non-fatal speaker routing failures from the global voice error channel so applications can keep call failures distinct from output selection warnings. This adds dedicated client and React state for output device errors, routes the example warning beside the speaker picker, and documents the new behavior.

Also harden playback output setup against async races: rapid sink switches now preserve the latest requested device, concurrent playback initialization shares one media element setup, and teardown during startup no longer leaves orphaned audio routing. Expanded tests cover unsupported sink selection, out-of-order switches, setup/teardown races, and React hook propagation.

Co-authored-by: Cursor <cursoragent@cursor.com>
@threepointone
threepointone merged commit a3a8d83 into main Jun 12, 2026
4 checks passed
@threepointone
threepointone deleted the audio-picker branch June 12, 2026 08:58
@github-actions github-actions Bot mentioned this pull request Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[voice] Output device routing (setSinkId / outputDeviceId) for Voice playback

2 participants