Experimental Audio Support

[adamshiervani]

Captures audio from the host's HDMI source via a UAC1 USB gadget, encodes it server-side (G.722 / PCMU fallback), and pipes it to the browser over the existing WebRTC session.

Decisions

• Off by default. New Settings → Audio → Enable Audio (badge: Experimental) writes to device config; toggling the switch reloads the page so the next WebRTC handshake renegotiates cleanly. Frontend only renders the <audio> element when the device says audio is on, so Firefox's "block audio" policy doesn't fire on a silent stream.
• Single pion Session carries both tracks. Audio is negotiated inside the same offer/answer exchange — no separate signaling path, no second peer connection, no extra dependencies. The video path is untouched when audio is disabled.
• One canonical MediaStream on the client. ontrack ignores event.streams[0] and accumulates every incoming track into one stream stored in the RTC store. Robust to answers that omit a=msid for audio (which Firefox would otherwise route into a synthetic per-track stream that wipes out the video).

Mechanisms added

• ALSA capture (internal/audio/). Cgo-loaded libasound via dlopen so the regular Go build works without ALSA headers in the sysroot; periodic blocking reads, no zero-fill on short reads.
• G.722 + G.711 µ-law encoders in pure Go (internal/audio/g722.go, g711.go), browser-negotiated based on the offer SDP. 48 kHz stereo → 8 kHz / 16 kHz mono with a single-divide downsample.
• Reconnect logic. After 5 consecutive non-idle read errors the capture closes and reopens with exponential backoff (100 ms → 2 s cap), re-resolving the ALSA card index on each attempt so USB re-enumeration with a shifted card number is handled for free.
• WebRTC playout-delay header extension (internal/playoutdelay/). Stamps min=max=0 on every outgoing RTP packet for both video and audio. Chrome's adaptive jitter buffer is one-way — it grows under jittery input (e.g. variable-size H.264 frames during high-motion host content) and never shrinks back, leaving Playback Delay stuck at hundreds of ms until reload. The extension is an authoritative sender-side override that pins the receiver at the floor; audio is registered too because Chrome's AV-sync layer pulls video up to whatever the audio jitter buffer is.

Tests

End-to-end via Playwright against a real device + remote host: plays a 997 Hz tone on the USB ALSA sink and verifies getStats() inbound audio energy on the browser. Folded into make test_e2e under the existing remote-agent project.

---

Note

High Risk
Adds a new audio capture/encode pipeline (ALSA via cgo/dlopen) and modifies WebRTC negotiation/session lifecycle, which are latency- and stability-sensitive paths that can impact streaming reliability across browsers and devices.

Overview
Adds experimental, opt-in host audio streaming over the existing WebRTC connection, including device-level config (AudioEnabled) and new JSON-RPC endpoints (getAudioConfig/setAudioConfig) plus a Settings page to toggle it (forces reload for renegotiation).

Implements an ALSA capture stack in internal/audio (cgo libasound loaded via dlopen, plus Go G.711 µ-law and G.722 encoders) and a new audio.go capture loop with backoff/reopen logic and safe teardown (stopAudioIfOwner). WebRTC now always offers an audio m-line, conditionally attaches an outgoing audio track based on config+offer codec, adds a playout-delay RTP header-extension interceptor for both audio and video, and updates the UI to use a separate <audio> element and a canonical MediaStream that accumulates tracks (with mediaStreamTrackVersion for consumers).

Extends the USB gadget defaults to include a UAC1 audio function and expands e2e coverage: the remote-agent can list ALSA devices and play/stop a test tone, while Playwright adds an end-to-end spec that verifies inbound audio stats and improves remote-agent deploy caching.

^{Reviewed by Cursor Bugbot for commit c443959. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 330f617. Configure here.}