4.0.4 Training fine-tuning documentation

[WGS-note]

Hello pyannote team, Hello everyone !

First of all, thank you very much for open-sourcing this excellent project.

I have integrated and decoupled pyannote.audio 4.0.4 into my own speech processing pipeline, and I would like to ask for some advice regarding customization and fine-tuning.

My questions are:

1. What is the recommended way to fine-tune pyannote.audio models? For example, if I want to fine-tune speech segmentation, speaker change detection, or overlap detection, what training pipeline should I follow?

2. In overlapping speech scenarios, what approaches would you recommend to obtain better segmentation results? Are there specific strategies, models, or post-processing methods that can improve overlap handling?

Thank you very much for your time and help.

Billions of thanks.

[LuShadowX]

The short answer to (1) is that there isn't a current recipe to follow — both places the docs send you are pinned to older majors, and the APIs they use were removed in 4.0. So the first thing worth knowing is what to translate.

Why the existing tutorials don't run on 4.0.4

• README → tutorials/training_a_model.ipynb pins pyannote.audio==3.1.1 and imports OverlappedSpeechDetection
• FAQ "How can I improve performance?" → tutorials/adapting_pretrained_pipeline.ipynb pins pyannote.audio==2.1.1 and uses Segmentation(..., loss="bce", vad_loss="bce")
• tutorials/overlapped_speech_detection.ipynb pins 3.1.1 and is built entirely on the removed task

Mapping the differences, from the 4.0.0 changelog:

2.1 / 3.1	4.x
tasks.Segmentation	tasks.SpeakerDiarization
tasks.OverlappedSpeechDetection	removed — "part of SpeakerDiarization task"
vad_loss=	removed (BREAKING)
loss=	deprecated, warns, has no effect
max_num_speakers=	deprecated → max_speakers_per_chunk
pytorch_lightning	lightning
use_auth_token=	token=
pyannote-audio-train CLI	removed

1. Fine-tuning segmentation on 4.x

The structure of the 2.1 recipe still holds — only the task class and its arguments changed:

from pyannote.audio import Model
from pyannote.audio.tasks import SpeakerDiarization
from pyannote.database import registry, FileFinder

registry.load_database("database.yml")
dataset = registry.get_protocol(
    "MyCorpus.SpeakerDiarization.MyProtocol", {"audio": FileFinder()}
)

model = Model.from_pretrained("pyannote/segmentation-3.0", token=True)

task = SpeakerDiarization(
    dataset,
    duration=model.specifications.duration,
    # match the checkpoint you are adapting, otherwise the classifier is resized
    max_speakers_per_chunk=len(model.specifications.classes),
    max_speakers_per_frame=model.specifications.powerset_max_classes,
    batch_size=32,
)
model.task = task
model.prepare_data()
model.setup()

Those two specifications fields are the ones that matter: classes is ["speaker#1", ..., "speaker#N"] with N == max_speakers_per_chunk, and powerset_max_classes is max_speakers_per_frame. Reading them off the checkpoint keeps the powerset head the same shape, which is what you want when adapting rather than retraining from scratch.

Training itself is unchanged apart from the import — lightning instead of pytorch_lightning, Adam at 1e-4, and monitor, direction = task.val_monitor for ModelCheckpoint / EarlyStopping. Setting cache= on the task is worth it for anything large; metadata caching is where the 4.0 training speedup comes from.

2. Overlapping speech

There is no separate overlap task or pipeline to tune any more — 4.0 removed both, and overlap is handled inside SpeakerDiarization by the powerset formulation (Plaquet & Bredin, "Powerset multi-class cross entropy loss for neural speaker diarization", Interspeech 2023). The model predicts one class per combination of speakers rather than independent per-speaker sigmoids, so simultaneous speech is represented directly instead of being reconstructed by thresholding.

The knob is max_speakers_per_frame. It defaults to 2, meaning the powerset covers up to two simultaneous speakers; raising it to 3 lets the model represent three-way overlap, at the cost of a larger output space (num_powerset_classes is the sum of binomials up to that value, so it grows quickly). If your data genuinely has three-way overlap and you have enough of it annotated, that is the parameter to raise — otherwise leaving it at 2 and fine-tuning on in-domain audio is the higher-yield move.

One caveat on all of the above: this is the API as it stands in the 4.x source, not a fine-tuning run I have benchmarked end to end — so treat the values as defaults to start from rather than tuned ones.

[WGS-note]

thanks for your reply ! In China, there is an old saying: "茅塞顿开".

Again, thank you very much for your reply and explanation.

I have another small question to ask.

As you mentioned, in pyannote.audio 4.x, the training data is based on RTTM annotations. The annotations need to provide information about who is speaking at each time segment and whether there is overlapping speech.

My understanding is that the training entry point is now the SpeakerDiarization task, instead of the previous Segmentation task. However, for adapting the speaker-diarization-community-1 model to my domain, the main component I need to fine-tune is the segmentation model checkpoint.

My current goal is to improve:

1. speech segmentation accuracy (more accurate speaker boundaries);
2. overlap handling in telephone conversations.

So I would like to confirm:

• Is fine-tuning the speaker-diarization-community-1/segmentation checkpoint with my own RTTM annotated dataset the recommended approach?
• After fine-tuning, can I directly replace the original segmentation checkpoint in the Community-1 pipeline while keeping the embedding and PLDA components unchanged?

My data consists mainly of Chinese telephone conversations, where the main challenges are short speaker turns, frequent interruptions, and overlapping speech.

Finally, thank you very much for your reply, and wish you all the best !

[LuShadowX]

Thanks for the reply, and for 茅塞顿开 — glad it helped. Happy to help with your issue.

Short answers: yes to both.

1. Fine-tuning the segmentation checkpoint is the right target

For your two goals it's the only component that matters. The segmentation model decides speaker boundaries and which frames contain overlap; the embedding and PLDA only decide who is who once boundaries exist. Sharper turn boundaries and better overlap in telephone audio are entirely a segmentation problem.

Load that checkpoint on its own with the subfolder argument:

from pyannote.audio import Model

model = Model.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    subfolder="segmentation",
    token=True,
)

Then attach the SpeakerDiarization task and train exactly as in my earlier message, reading max_speakers_per_chunk and max_speakers_per_frame off model.specifications so the powerset head keeps its shape.

2. You can swap it in and keep embedding + PLDA unchanged

The pipeline takes those three as independent arguments — this is the real signature in 4.0.4, unchanged in 4.0.7:

def __init__(
    self,
    legacy: bool = False,
    segmentation: PipelineModel = {"checkpoint": "pyannote/speaker-diarization-community-1",
                                   "subfolder": "segmentation"},
    segmentation_step: float = 0.1,
    embedding: PipelineModel = {"checkpoint": "pyannote/speaker-diarization-community-1",
                                "subfolder": "embedding"},
    embedding_exclude_overlap: bool = False,
    plda: PipelinePLDA = {"checkpoint": "pyannote/speaker-diarization-community-1",
                          "subfolder": "plda"},
    clustering: str = "VBxClustering",
    ...
)

segmentation goes through get_model, which accepts a Model instance, a path or hub id as a string, or a dict of from_pretrained kwargs. So you pass your checkpoint and leave the other two at their defaults:

from pyannote.audio.pipelines import SpeakerDiarization

pipeline = SpeakerDiarization(
    segmentation="/path/to/your/finetuned.ckpt",   # your model
    # embedding, plda, clustering left at community-1 defaults
    token=True,
)
pipeline.instantiate({
    "segmentation": {"min_duration_off": 0.0},
    "clustering": {"threshold": 0.6, "Fa": 0.07, "Fb": 0.8},
})

Build it this way rather than with Pipeline.from_pretrained, which would load the shipped config and its own segmentation checkpoint.

The part that's easy to miss

Those hyper-parameters are default_parameters() from the source — they were tuned for the shipped segmentation model. Once you replace it, they're no longer optimal, and on telephone data with short turns the clustering values are where I'd expect the biggest shift. Re-tune them on a held-out subset of your own data; keeping the fine-tuned model but the old parameters is the usual reason an adapted pipeline scores worse than the original.

Two smaller notes:

• The tunable set depends on your model. The pipeline checks specifications.powerset: powerset models expose only min_duration_off, non-powerset ones also expose threshold. If you fine-tune from the community-1 checkpoint you stay on the powerset branch.
• For frequent interruptions, segmentation_step (default 0.1, i.e. 10% of the model's window) controls how much consecutive windows overlap. Lowering it gives finer boundary resolution at the cost of more forward passes — cheap to try before touching the model.

Usual caveat: this is read from the 4.x source rather than a fine-tuning run I've benchmarked on Chinese telephone data, so treat the numbers as starting points. If you do run it, the DER before and after re-tuning would be genuinely useful to see here.

[WGS-note]

I got it ! thank you very much for your reply and explanation ! wish everything goes smoothly !

[LuShadowX]

Hey! Happy to help. Looks like you forgot to mark it as the answer. Guess it didn't solve your query. 😅
By the way, just a random thought. I've always admired China. I've heard people there are very intelligent and highly competitive, and from what I've seen, that definitely seems true. I also really like Chinese culture.
Would you mind being friends? Also, if you're ever working on a project related to LLMs or machine learning, count me in! I'm really interested in ML, so I'd love the opportunity to collaborate and learn. It would be great to work with someone from China.
See ya.