Skip to content

Auditorium v1.39.0 — one track per speaker

Latest

Choose a tag to compare

@Redrum624 Redrum624 released this 06 Sep 14:58
· 3 commits to main since this release

Separate Voice now hears the people in the room: it counts the speakers, shows
you the count before anything lands, and gives you one full-length track per
speaker plus Backing.

Added

  • Separate Voice lands one track per speaker. Pipeline → Separate Voice
    is the same command, and when more than one person is talking it goes one
    step further. It runs three stages instead of one: the existing HT-Demucs
    separation, then speaker segmentation (pyannote-segmentation-3.0 over 10 s
    windows shifted by 1 s — SEG_WINDOW / SEG_SHIFT in
    electron/diarizeHost.cjs) and speaker embedding (WeSpeaker ResNet34-LM with
    per-utterance mean subtraction) in a new utility process, then clustering and
    assembly in the renderer. Then it stops: a confirmation step shows the
    count it found, each speaker's speech time and share, and the memory a
    landing would take. Nothing lands until you press Land, and changing the
    count re-groups the measurements already taken — instantly, with no second
    model run. Landing produces <name> — Speaker 1 … Speaker N plus
    <name> — Backing. A count of one lands exactly the Voice + Backing result it
    always did. The two models are one 32.5 MB download (5,992,913 B
    segmentation + 26,530,550 B embedder = DIARIZE_MODEL_BYTES 32,523,463 in
    src/services/diarizeService.ts), offered beside the 166 MB separation model.
    The grouping policy is fixed and in code: threshold 0.55, minimum cluster
    size 4, minimum speaker share 5 %, at most 6 speakers
    (DIARIZE_THRESHOLD / MIN_CLUSTER_SIZE / MIN_SPEAKER_SHARE in
    src/dsp/diarization.ts, MAX_SPEAKERS in src/dsp/speakerClustering.ts).
    New: src/dsp/diarization.ts, src/dsp/spanMask.ts,
    electron/diarizeHost.cjs, electron/diarizeManager.cjs,
    src/services/diarizeService.ts, scripts/diarize-bench.cjs,
    scripts/fetch-diarization-assets.cjs; changed:
    src/components/Dialogs/SeparateDialog.tsx, src/services/stemLanding.ts,
    electron/main.cjs, electron/preload.cjs, src/App.tsx.
  • A speaker track is the whole voice with the other people silenced, and the
    landing carries no exact-sum claim.
    Each speaker track is the full-length
    voice stem with everything outside that speaker's turns taken to silence, and
    every kept turn faded in and out over 10 ms so its edges do not click.
    Where two people talk at once, that audio is written into both their
    tracks. So speakers + Backing does not add back up to the source — unlike
    Separate into Stems (bit-exact) and the one-voice landing (within float32
    rounding, worst 4.32e-7). The dialog says so before you land and prints no
    exactness note afterwards; the Backing on its own is unchanged and still adds
    back. The full write-up is in docs/KNOWN_LIMITATIONS.md under A speaker
    split is a COUNT that was right four times out of four, and nothing finer
    .
  • What the count rests on — and what it does not. From
    docs/bench/diarize-bench-baseline.json (written by
    scripts/diarize-bench.cjs): on the four test recordings the counts came out
    2 / 2 / 2 / 4 against a truth of 2 / 2 / 2 / 44 of 4 — in both
    measured conditions, --direct (16 kHz speech straight into the speaker step)
    and --full-chain (what the tool actually does: HT-Demucs first, then the
    Vocals stem). Audio-anchored consistency was 100 / 100 / 96.5 / 100 % direct
    and 100 / 91.9 / 96.8 / 100 % through the chain. That is the whole of the
    evidence, and the same file says what it is not: four recordings, about
    162 s in total; the truth shipped with them is a speaker count per
    file and nothing finer, so there is no diarization error rate — a row can
    have the right count with the wrong turns, and nothing here measures who spoke
    when. One of the four is the only four-speaker case and it is Mandarin read
    by an English-trained embedder
    (WeSpeaker VoxCeleb ResNet34-LM). Overlapping
    speech is near-absent in all four, so overlap handling is untested end to
    end, and recordings with many short turns or heavy crosstalk were not in the
    set. The count is a control you can overrule for exactly that reason, and the
    panel prints the limit beside it.

Changed

  • The Separate Voice time estimate and its progress bar carry the measured
    stage rates.
    Both seeds were re-derived from
    docs/bench/diarize-bench-baseline.json on an idle machine — segmentation
    10 ms and embedding 75 ms per audio second (MEASURED_SEGMENT_MS_PER_S
    and MEASURED_EMBED_MS_PER_S, src/services/diarizeService.ts), replacing 8
    and 55, which came from an early spike that had timed a different embedder
    (CAM++, which this feature does not ship). The weighted bar now hands Demucs
    88.6 % of the run instead of 91.3 % (stageWeights(), computed from those
    two seeds and MEASURED_REALTIME_FACTOR = 1.52 in
    src/services/stemService.ts). The estimate assumes an otherwise idle
    machine.
    Its Demucs term is the conservative end of the app's own two stem
    measurements, while this bench measured the stem stage at 1.97–2.07× realtime
    (msPerAudioSecond.stem 483.2–507.2 across the four full-chain rows), so on
    an idle machine the panel reads long — the safe direction. On a busy one
    it reads short: docs/KNOWN_LIMITATIONS.md records the same bench run beside
    a full test suite putting the chain at 1,150–1,230 ms per audio second, about
    40 % over what the panel predicts. The per-stage spread through the chain
    is in the baseline: segmentation 6.0–11.9 ms and embedding 39.9–110.1 ms per
    audio second. Affects: src/services/diarizeService.ts.
  • The speaker landing's memory gate prices the Backing document too. A
    landing allocates N speaker documents plus a full-length Backing, so the
    panel quotes and refuses on (N + 1) × the document size. At 15 minutes of
    44.1 kHz stereo one document is 317.5 MB, so two speakers cost 952.6 MB (not
    635.0 MB) and three cost 1.3 GB; a landing above 1.2 GB
    (SPEAKER_LANDING_BUDGET_BYTES, src/services/stemLanding.ts) is refused
    with the figure in hand rather than landed short. None of these documents has
    been written to disk, so a project save writes every one of them into the
    project file — export the speakers you want and close the rest. Affects:
    src/components/Dialogs/SeparateDialog.tsx, src/services/stemLanding.ts.

Verify your download (see SHA256SUMS.txt):

CertUtil -hashfile "Auditorium Setup 1.39.0.exe" SHA256