feat: speaker diarization with voice-print identification - #3
Merged
Conversation
Add optional speaker diarization via sherpa-onnx (pyannote segmentation + wespeaker embeddings, ONNX on CPU, fully local) behind a Diarizer port, plus voice-print enrolment so SPEAKER_00 labels become real names. - domain stays pure: SpeakerTurn, VoicePrint, assign_speakers, rename_speakers, cosine_similarity/match_speakers, longest_turn_per_speaker are plain Python with no numpy and no sherpa - sherpa_onnx is imported in adapters/ only, so the engine stays swappable through the port - --diarize turns off silence removal and denoising: silenceremove shifts the timeline away from the transcript, and dynaudnorm/afftdn degrade the speaker embeddings - diarization always runs on the exact file that was transcribed - vox speakers add/list stores voice prints in ~/.vox/voiceprints.json, --identify then renames labels on every later video - outputs carry the speaker: [NAME] in SRT, per-speaker blocks in TXT, "speaker" field in JSON - worker threads default to cpu_count - 2, floored at 2 Verified end to end on a two-voice dialogue: turns alternate correctly, enrolment then identification maps both speakers to their real names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uz3rnGtYHs6YcBNr6MkY1z
--identify was pointless: when no voice is enrolled the use case returns early without extracting a single embedding, so running identification unconditionally costs nothing. Whenever voices are known, matching speakers are now renamed automatically. Replaced by --no-identify for the rare case of wanting raw SPEAKER_xx labels back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uz3rnGtYHs6YcBNr6MkY1z
--speakers is no longer needed. The diarizer now over-segments on purpose (clustering threshold 0.05), embeds the longest turn of each resulting label, and merges the excess labels by picking the count with the best silhouette score. Measured on two-voice dialogues: both are resolved to exactly 2 speakers with the right alternation, including the pair of near-identical synthesised voices that a single fixed threshold could never separate. - clustering, silhouette, estimate_speaker_count, merge_labels and relabel_turns are pure domain, plain Python, no numpy and no sherpa - AutoSpeakerCountDiarizer implements the Diarizer port by composition, so it is unit-tested entirely with fakes - cost stays low: one embedding per label (a handful), not per turn (628 on a one-hour file) - --speakers still forces a count and then skips estimation entirely Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uz3rnGtYHs6YcBNr6MkY1z
On a real 36-minute recording the over-segmenting pass produced 290 labels, not the handful the design assumed. Embedding all of them and running the cubic clustering once per candidate count took ~45 minutes. The probe now samples the 24 longest turns regardless of how many labels came out, estimates the count from those, and re-runs diarization with it. Cost no longer depends on the label count. Measured on that file: 45 min -> 7.3 min (4.9x real time). The phone conversation, previously collapsed onto a single speaker, is now split across two voices. Still open: the estimate returns 2 speakers where the recording holds 3-4, so distinct people still share a label. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uz3rnGtYHs6YcBNr6MkY1z
Sampling the globally longest turns concentrated on whichever speaker talks most: on a file where one voice dominates, all 24 sampled turns belonged to it and the count collapsed to 1. The probe now takes the longest turn of each label first, then keeps the longest 24 of those. Also adds pick_speaker_count, which can accept a finer split whose silhouette stays within a tolerance of the best score. It defaults to 1.0, i.e. the plain maximum, because lowering it is not justified yet: at 0.70 a real 3-speaker recording did report 3, but the third label held 2 turns out of 609 while the two genuinely distinct people stayed merged, and 2-speaker files drifted to 3-4. Right number, wrong reason. Verified: every 2-speaker file now returns 2, including the one that previously returned 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uz3rnGtYHs6YcBNr6MkY1z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional speaker diarization — who said what — plus voice-print enrolment so
SPEAKER_00labels become real names.Usage
Outputs carry the speaker:
[Coco]in SRT, per-speaker blocks in TXT,"speaker"field in JSON.Engine choice
sherpa-onnx(2 MB + 16 MB native lib, one transitive dependency) rather thanpyannote.audio(22 direct dependencies, including torch, Lightning, matplotlib and three OpenTelemetry packages). It runs the same models as pyannote 3.1 — pyannote 3.0 segmentation and wespeaker resnet34-LM embeddings — but through ONNX Runtime.The deciding factor is that this project is Apple Silicon only: there is no CUDA, and MPS is unreliable for pyannote, so torch would run on CPU anyway — all of its weight, none of its benefit. Everything here is local and free: no API, no key, no per-hour cost.
Architecture
sherpa_onnxis imported inadapters/only.models/,ports/anduse_cases/contain zero references to it, so swapping the engine means writing one adapter and changing one line in_build_use_case.The interesting logic is pure domain, in plain Python with no numpy:
assign_speakers(overlap-weighted turn assignment),rename_speakers,cosine_similarity/match_speakers,longest_turn_per_speaker.Two non-obvious fixes
--diarizedisablessilenceremoveandafftdn/dynaudnorm. Silence removal physically shifts the audio timeline out of sync with the transcript, and dynamic normalisation flattens exactly the cues that separate voices.Verification
Two-voice dialogue with known ground truth: turns alternate correctly (A-B-A-B), and after enrolling both voices, identification maps them to the right names. Speaker separation measured at 0.94/0.92 within a speaker versus 0.71/0.65 across speakers, hence the 0.8 match threshold.
256 tests green, ruff clean, written test-first throughout.
Known limits
--speakers Nwhen you know it. A silhouette-based automatic estimate is the planned follow-up.vox channel(batch) does not expose--diarizeyet.SherpaVoicePrintExtractor.extractdecodes the whole file per call — fine for enrolment, needs a single decode before batch use.Based on
feat/models-command-and-ciso the diff shows only this work.🤖 Generated with Claude Code