fix(transcription): transcribe mic+system downmix in Both mode#5
Open
farce1 wants to merge 1 commit into
Open
Conversation
In the default "Both" capture mode only the microphone was sent to ASR; system/loopback audio (remote meeting participants) was recorded to the OGG but never transcribed, so remote speakers were missing from the transcript. Downmix mic+system to mono for the transcription stream via mix_to_mono so remote participants appear in the transcript. Averaging keeps the signal in range (no clipping); Mic mode is unchanged (empty system passes mic through) and System mode already transcribes loopback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In the default "Both" capture mode (
audio_sourcedefaults to\"both\"), the mixer sent only the microphone to ASR —system/loopback audio was mixed into the OGG recording but never into the transcription stream (audio/mod.rs). Result: in remote/hybrid meetings, every remote participant was missing from the transcript (their audio arrives via system/loopback). In-person meetings were unaffected (the mic catches the room).Fix
mixer::mix_to_mono(mic, system)— averages the two sources into one mono stream (averaging keeps the signal in range; no clipping), tolerates length mismatch (pads the shorter with silence), and passes the mic through unchanged when there's no system audio.mix_to_mono(mic, latest_system)to transcription. Mic mode is byte-identical (itslatest_systemis always empty → mic pass-through); System mode already transcribed loopback.Diarization still separates speakers from the OGG, so the richer transcript and speaker labels stay coherent.
Verification (TDD)
mix_to_mono(5 cases: pass-through, clip-safe average, phase cancellation, shorter-system padding, empty mic).clippy --all-targets --all-features -- -D warnings✅; fullcargo test→ 5 new tests pass, only the pre-existingmodel_archive_consts_tests(REPLACE_WITH_release placeholders) fail, unrelated to this change.Known limitation (pre-existing, not introduced here)
The mixer pairs each mic chunk with the latest system chunk (crude sync, same as the existing OGG stereo mix) and assumes both sources share a sample rate. Overlapping speech is downmixed (Whisper handles it imperfectly). These are inherited from the current mixing path; this PR strictly adds remote-speaker coverage without changing that behavior.
mainis red on three maintainer-only release blockers (SHA256/pubkeyREPLACE_WITH_placeholders, uncommittedvendor/vc_redist.x64.exe, and a separate frontend failure). Verified locally with a temporary untracked vendor placeholder. Do not merge until release prep is complete. Staged Track-A fix from the Handy vs openNotes transcription review; code-only, no new deps.