Barge-in with headphones - #6
Merged
Merged
Conversation
WebRTC APM won't build on macOS (Linux-only prctl.h), so use speexdsp's EchoCanceller — cross-platform (Mac + Pi), optional dep (falls back to the half-duplex gate if the system lib is absent). - voice_aec.py: canceller factory + pure helpers (16k resample, 10ms reframing), env-tunable echo tail (ARIA_AEC_TAIL). speexdsp is optional via available(). - voice_live.py: AEC mode default when speexdsp present — mic stays open, each 10ms frame echo-cancelled against the matching played reference, so you can talk over her without her hearing herself. --duplex (headphones, raw) and --no-aec (gate) fallbacks. Render references captured in the player; render_q cleared on barge-in. - tests for the pure helpers (192 total pass). Acoustic quality is manual-test only. Assisted-by: Claude Code (Opus 4.8)
She kept hearing herself because the previous design ran mic and speaker as SEPARATE streams and paired them through a queue — variable/empty reference, so speexdsp's adaptive filter never converged and echo leaked. Collapse to ONE duplex sd.Stream at 16 kHz: each callback plays one frame and captures one frame in lockstep, giving a constant aligned mic↔speaker delay the echo canceller can actually learn. Gemini's 24 kHz output is resampled to 16 kHz for playback (and reused as the cancellation reference). Gate (--no-aec) and headphone (--duplex) paths preserved. Assisted-by: Claude Code (Opus 4.8)
The speexdsp binding exposes only EchoCanceller, not the residual-suppression preprocessor, so it leaks enough echo to self-trigger Gemini's VAD (she answers herself). Demote it to an opt-in --aec experiment. Default is now the half-duplex gate (mutes mic while she speaks → no self-loop, no barge-in); --duplex (headphones) gives true barge-in. Real speaker barge-in needs OS/WebRTC-grade AEC (browser getUserMedia in the eventual app, or WebRTC APM on the Pi). Assisted-by: Claude Code (Opus 4.8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Barge-in in duplex mode with speakers