DiariZen Explained

A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline

Paper · Notebook · Cheatsheets · HTML Preview · Original DiariZen

This repository explains every step of the leading open-source state-of-the-art system.

---

Overview

DiariZen (Han et al., ICASSP 2025) is the leading open-source state-of-the-art speaker diarization system at the time of writing this repository. It combines a structurally pruned WavLM-Large encoder, a Conformer backend with powerset classification, and VBx clustering to answer who spoke when in multi-speaker audio.

Despite its strong performance, DiariZen's implementation is spread across multiple repositories and frameworks, making it difficult to understand, reproduce, or extend as a whole. This repository fills that gap.

We decompose the pipeline into 7 functional blocks, provide a modular Python implementation of each stage, annotated visualisations of intermediate outputs, and a Jupyter notebook that runs the complete pipeline end-to-end on a real multi-speaker recording from the AMI Meeting Corpus.

Note: This repository is a tutorial companion to the original DiariZen system. All credit for the DiariZen architecture belongs to Han et al. (ICASSP 2025).

---

Pipeline at a Glance

#	Block	What it does	Input	Output
1	Audio loading & sliding window	Load WAV, apply overlapping windows	WAV file	(10, 1, 256000)
2	WavLM feature extraction	Pruned WavLM-Large encoder + learned layer sum	(10, 1, 256000)	(10, 799, 1024)
3	Conformer + powerset	4-layer Conformer + 11-class powerset head	(10, 799, 1024)	(10, 799, 11)
4	Segmentation aggregation	Overlap-add + median filter + speaker count	(10, 799, 11)	(10, 799, 4) + (1521, 1)
5	Speaker embeddings	wespeaker ResNet34 + overlap-exclusion mask	(10, 799, 4)	(10, 4, 256)
6	VBx clustering	LDA + AHC/PLDA + VB-HMM refinement	(10, 4, 256)	(10, 4)
7	RTTM output	Reconstruct + binarize + write RTTM	(10, 4)	.rttm file

---

Quickstart

1. Clone this tutorial repository

git clone https://github.com/nikhilraghav29/diarizen-tutorial.git
cd diarizen-tutorial

2. Clone the original DiariZen repository

The tutorial blocks depend on DiariZen's source code. Clone it alongside this repository:

git clone https://github.com/BUTSpeechFIT/DiariZen.git
cd DiariZen
git submodule init
git submodule update

3. Set up the conda environment

conda create --name diarizen python=3.9
conda activate diarizen
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 \
      pytorch-cuda=12.1 "mkl<2024.1" -c pytorch -c nvidia -c defaults
pip install -r requirements.txt && pip install -e .
cd pyannote-audio && pip install -e ".[dev,testing]" && cd ..

The complete environment specification is provided in environment_diarizen.yml. To recreate it exactly:

conda env create -f environment_diarizen.yml
conda activate diarizen

Key packages in the verified working environment:

Package	Version
Python	3.9.25
PyTorch	2.1.2
torchaudio	2.1.2
torchvision	0.16.2
CUDA	12.1
pyannote-audio	3.1.1
speechbrain	1.1.0
numpy	1.26.4
scipy	1.13.1
einops	0.8.2
accelerate	1.6.0

4. Copy the tutorial blocks into your DiariZen directory

cp -r /path/to/diarizen-tutorial/blocks/ /path/to/DiariZen/blocks/
cp /path/to/diarizen-tutorial/DiariZen_pipeline.ipynb /path/to/DiariZen/

5. Download the pretrained models

The models are downloaded automatically from HuggingFace on first run. Set your cache directory before running:

export HF_HOME=/path/to/your/hf/cache

Models downloaded on first run:

• BUT-FIT/diarizen-wavlm-large-s80-md (~278 MB) — segmentation model
• pyannote/wespeaker-voxceleb-resnet34-LM (~27 MB) — speaker embedding model

Note on model licence: The pretrained DiariZen model weights are released under CC BY-NC 4.0 — research and academic use only. See the DiariZen model licence for full details before use.

6. Run the pipeline notebook

cd /path/to/DiariZen
conda activate diarizen
jupyter notebook DiariZen_pipeline.ipynb

Select the diarizen kernel and run the cells top to bottom.

---

Repository Structure

diarizen-tutorial/
│
├── blocks/                          # Modular pipeline functions
│   ├── __init__.py
│   ├── pipeline_loader.py           # Load pipeline once, pass to all blocks
│   ├── block1_windowing.py          # run_block1() → dict
│   ├── block2_wavlm.py              # run_block2(b1, pipeline) → dict
│   ├── block3_conformer.py          # run_block3(b2, pipeline) → dict
│   ├── block4_aggregation.py        # run_block4(b3, pipeline) → dict
│   ├── block5_embeddings.py         # run_block5(b4, pipeline) → dict
│   ├── block6_clustering.py         # run_block6(b5, pipeline) → dict
│   └── block7_rttm.py               # run_block7(b6, pipeline) → dict
│
├── cheatsheets/                     # Visual one-page summaries for each block
│   ├── diarizen_full_pipeline_overview.svg
│   ├── block1_sliding_window_cheatsheet.svg
│   ├── block2_wavlm_feature_extraction_cheatsheet.svg
│   ├── block3_conformer_powerset_cheatsheet.svg
│   ├── block4_segmentation_aggregation_cheatsheet.svg
│   ├── block5_speaker_embeddings_cheatsheet.svg
│   ├── block6_vbx_clustering_cheatsheet.svg
│   └── block7_reconstruct_rttm_cheatsheet.svg
│
├── example/                         # Test audio
│   └── EN2002a_30s.wav              # 30s excerpt from AMI Meeting Corpus
│
├── run_block1.sh  ...  run_block7.sh  # SLURM scripts for GPU cluster
├── DiariZen_pipeline.ipynb          # End-to-end Jupyter notebook
├── DiariZen_pipeline.html           # Pre-rendered notebook (no setup required)
├── environment_diarizen.yml         # Complete conda environment specification
├── CITATION.cff                     # Citation metadata
└── README.md

---

Block-by-Block Walkthrough

Block 1 — Audio Loading & Sliding Window

Loads the WAV file, forces mono, and applies a sliding window using torch.unfold().

Key parameters (from config.toml):

• seg_duration = 16s — each chunk covers 16 seconds
• segmentation_step = 0.1 — step is 10% of duration = 1.6s → 90% overlap
• Last incomplete chunk is zero-padded to 16s

Output: (10, 1, 256000) — 10 chunks × 1 channel × 256,000 samples

---

Block 2 — WavLM Feature Extraction

Passes each chunk through the structurally pruned WavLM-Large encoder.

Key ideas:

• WavLM-Large pruned from 316M → 63M parameters (80% sparsity)
• 25 layer representations extracted (1 CNN + 24 transformer)
• SUPERB-style learned weighted sum collapses 25 layers → 1
• CNN downsamples 320×: 256,000 samples → 799 frames at ~50 fps

Output: (10, 799, 1024) — 10 chunks × 799 frames × 1024-dim

---

Block 3 — Conformer + Powerset Classification

Applies the Conformer encoder and powerset classification head.

Data flow:

(10,799,1024) → proj(1024→256) → lnorm → 4×ConformerBlock → classifier(256→11) → LogSoftmax → (10,799,11)

Powerset encoding — predicts one of 11 speaker combinations per frame:

Class	Meaning
0	Silence
1–4	Single speaker (Speaker 1, 2, 3, or 4)
5–10	Two-speaker overlap (all 6 combinations of {s1,s2,s3,s4})

Output: (10, 799, 11) log-softmax scores + (10, 799, 4) binary multilabel

---

Block 4 — Segmentation Aggregation

Aggregates 10 overlapping chunk predictions into one continuous segmentation.

Key operations:

1. get_segmentations() — overlap-add averaging (each interior frame averaged over up to 10 chunks)
2. median_filter(1,11,1) — 220ms temporal smoothing
3. speaker_count() — instantaneous active speaker count → (1521, 1)

Statistics on the test recording:

• 8.0% silence, 64.0% single speaker, 27.9% overlap

---

Block 5 — Speaker Embedding Extraction

Extracts a 256-dim speaker embedding per (chunk, local speaker) pair.

Key design: Overlap exclusion masking — frames where 2+ speakers are active are zeroed out before pooling, ensuring embeddings represent clean single-speaker speech.

Model: pyannote/wespeaker-voxceleb-resnet34-LM — ResNet34 trained on VoxCeleb

Output: (10, 4, 256) — L2-normalised, NaN where speaker is inactive in that chunk

---

Block 6 — VBx Clustering

Assigns each local speaker in each chunk to a global speaker identity.

Two-stage process:

1. AHC + PLDA — LDA projection (256→128) + pairwise PLDA scoring + agglomerative clustering (ahc_threshold=0.6)
2. VB-HMM EM — Variational Bayes refinement (Fa=0.07, Fb=0.8, max_iters=20)

Output: (10, 4) hard cluster IDs — integer global speaker ID, -2 = inactive

---

Block 7 — Reconstruct + RTTM Output

Converts cluster assignments → diarization annotation → RTTM file.

Steps:

1. reconstruct() — map local→global IDs, OLA aggregate → (1521, 4)
2. Binarize(onset=0.5) — threshold → pyannote Annotation
3. rename_labels() — 0,1,2,3 → SPEAKER_00, SPEAKER_01, ...
4. to_rttm() — write .rttm file

Result on EN2002a_30s.wav:

SPEAKER EN2002a_30s 1  0.792 12.820 <NA> <NA> SPEAKER_03 <NA> <NA>
SPEAKER EN2002a_30s 1 23.453  6.940 <NA> <NA> SPEAKER_02 <NA> <NA>

4 speakers, 13 segments detected in the 30-second AMI meeting excerpt.

---

Design Philosophy

The pipeline is loaded once and passed to each block — avoiding redundant model reloading across stages. Each block function accepts the previous block's output dictionary and returns a new dictionary carrying all keys forward:

from blocks.pipeline_loader   import load_pipeline
from blocks.block1_windowing  import run_block1
from blocks.block2_wavlm      import run_block2
from blocks.block3_conformer  import run_block3
from blocks.block4_aggregation import run_block4
from blocks.block5_embeddings  import run_block5
from blocks.block6_clustering  import run_block6
from blocks.block7_rttm        import run_block7

pipeline = load_pipeline(diarizen_root, hf_cache)

b1 = run_block1(audio_path, diarizen_root, verbose=True)
b2 = run_block2(b1, pipeline, output_dir="sandbox_outputs/block2")
b3 = run_block3(b2, pipeline, output_dir="sandbox_outputs/block3")
b4 = run_block4(b3, pipeline, output_dir="sandbox_outputs/block4")
b5 = run_block5(b4, pipeline, output_dir="sandbox_outputs/block5")
b6 = run_block6(b5, pipeline, output_dir="sandbox_outputs/block6")
b7 = run_block7(b6, pipeline, session_name="EN2002a_30s",
                rttm_out_dir="sandbox_outputs/block7")

---

Tested Environment

All experiments were conducted on the following setup. Other configurations may work but have not been verified.

Component	Configuration
GPU	NVIDIA H200 NVL (150 GB VRAM)
System RAM	256 GB
Disk space	~10 GB (models + outputs)
CUDA	12.1
Python	3.9.25
PyTorch	2.1.2
OS	Ubuntu 24.04

---

Citation

If you find this tutorial useful in your research or teaching, please consider citing the accompanying paper:

@misc{raghav2025diarizen,
  author    = {Nikhil Raghav},
  title     = {{DiariZen} Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline},
  year      = {2025},
  publisher = {arXiv},
  url       = {https://arxiv.org/abs/2604.21507}
}

Please also cite the original DiariZen system:

@inproceedings{han2025leveraging,
  title     = {Leveraging Self-Supervised Learning for Speaker Diarization},
  author    = {Han, Jiangyu and Landini, Federico and Rohdin, Johan and
               Silnova, Anna and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
  booktitle = {Proc. ICASSP},
  year      = {2025}
}

---

Related Work

• DiariZen — the original system (Han et al., ICASSP 2025)
• pyannote-audio — the speaker diarization framework
• SC-pNA — self-tuning spectral clustering for diarization (Raghav et al., ICASSP 2025)
• wespeaker — the speaker embedding toolkit
• dscore — DER scoring tool

---

License

The tutorial code in this repository is licensed under the Apache License 2.0. The pretrained DiariZen model weights, downloaded separately from HuggingFace, are released by the original authors under CC BY-NC 4.0 and are restricted to research and academic use. Please review the DiariZen model licence before use.

---

Contact

For any questions or if you would like to share some feedback: raghav.nikhil29@gmail.com · nikhil.raghav.92@tcgcrest.org · Open an issue