ValueError: inconsistent "classes" (is ['non_change', 'change'], should be: ['non_speech', 'speech'])

[danFromTelAviv]

Describe the bug
I'm trying to go through the diarization pipeline tutorial on my own data.

I am trying to run "apply" on my own data and model for speaker change detection.
I get an error that looks like it's trying to apply speech activity detection

ValueError: inconsistent "classes" (is ['non_change', 'change'], should be: ['non_speech', 'speech'])

To Reproduce
Steps to reproduce the behavior:

$ export EXP_DIR=tutorials/pipelines/speaker_diarization 
$ pyannote-audio scd  apply --step=0.1 --pretrained="<path to>/tutorials/models/speaker_change_detection/train/myData.SpeakerDiarization.general.train/validate_segmentation_fscore/myData.SpeakerDiarization.general.train" --subset=train ${EXP_DIR} myData.SpeakerDiarization.general

pyannote environment

$ pip freeze | grep pyannote
pyannote.core==4.1
pyannote.database==4.0.1
pyannote.metrics==3.0.1
pyannote.pipeline==1.5.2

Additional context
I only prepared a development set called "train" right now - so I'm running on that.
I successfully ran the SAD apply step before moving to SCD.

[hbredin]

It looks like your scd model has been trained with pyannote-audio sad train instead of pyannote-audio scd train.

To confirm this, can you please share the content of <path to>/tutorials/models/speaker_change_detection/train/myData.SpeakerDiarization.general.train/specs.yml?

[danFromTelAviv]

Thanks for getting back to me Herve.

I'm almost certain I trained with SCD - the content also looks like scd...

X:
  dimension: 1
task: frame-wise multi-class classification
y:
  classes:
  - non_change
  - change

[hbredin]

What about an existing metadata.yml file in EXP_DIR?

Did you first apply sad into the EXP_DIR and then try to apply scd into the same ÈXP_DIR directory?
This might explain the (not very clear indeed) error message you got.

[danFromTelAviv]

Thank you again for the effort!

There is no metadata.yml file directly in EXP_DIR
There is one in
{$EXP_DIR}/train/myData.SpeakerDiarization.general.train/validate_segmentation_fscore/myData.SpeakerDiarization.general.train/apply/0200

classes:
- non_change
- change
dimension: 2
duration: 0.0203125
start: 0.0
step: 0.0016875

When I trained the SCD I followed the scd finetune tutorial -

 export EXP_DIR=tutorials/models/speaker_change_detection

[danFromTelAviv]

I may have accidentally trained SAD there for a couple of epochs and then stopped it and trained a SCD. Is there a way to check - I don't remember.

[danFromTelAviv]

And is there a way to fix that if that's the case?

[hbredin]

I may have accidentally trained SAD there for a couple of epochs and then stopped it and trained a SCD. Is there a way to check - I don't remember.

If you could share the content of every .yml file in EXP_DIR, I might be able to help.
Otherwise, the only (bad) solution I can think of is starting again from scratch...

[danFromTelAviv]

Doesn't look like there's anything else... :/

I'll try to run it via code and put some breakpoints to see if I can figure out where it's straying.

Starting from scratch = open up a new folder called "speaker_change_detection_v2" and use it as an EXP_DIR or literally strain everything over (SAD, EMB...etc)?

Thanks for the quick responses!

[hbredin]

Starting from scratch = open up a new folder called "speaker_change_detection_v2" and use it as an EXP_DIR or literally strain everything over (SAD, EMB...etc)?

First option is enough: use a brand new SCD folder.

[danFromTelAviv]

nope - same warning remains.

I opened a new folder. trained an SCD there from scratch.
validated it.
then ran apply from the diarization EXP_DIR.
ValueError: inconsistent "classes" (is ['non_change', 'change'], should be: ['non_speech', 'speech'])

[danFromTelAviv]

Maybe I'm calling the model incorrectly?
Can you give me an example of what an "apply" command would look like for a locally trained model?

[hbredin]

See Application section of the tutorial:
https://github.com/pyannote/pyannote-audio/tree/develop/tutorials/models/speaker_change_detection#application

[danFromTelAviv]

I think I did that right...

Do I need to "apply" in the validate dir (as is the last step of this tutorial)? I skipped that step since the first step of the diarization pipeline is applying.

[hbredin]

Yes, apply it first in the validate directory and skip the Raw scores extraction part of the diarization pipeline tutorial.

Then, update the diarization pipeline configuration file to link to the ${VAL_DIR}/apply/{BEST_EPOCH} paths.
Someone (you? ;-)) should update the diarization pipeline tutorial to inform users about this feature.

[danFromTelAviv]

ok - I think that could work. I'll try it out and if it works I'll update my journey into the documentation.

[danFromTelAviv]

Just to let you know - I'm still running this process when I get a few minutes here and there.

One thing I'd like to edit already is every place where the export has "$ export varName = /path/to/some/where". It's stupid and minor but at least on my machine export doesn't work with spaces around the "=" sign.

[danFromTelAviv]

Looks like the apply locally trick worked! The diarization pipeline is training ...
I'll have time to update the documentation this weekend I believe.

[danFromTelAviv]

Hi Herve,

It's been a few months since I last did any PRs with github.
I forked, cloned the fork, opened a branch, made a couple of commits, pushed, added this repo as an "upstream" and tried to pull request.
I get a "no permissions" kind of error message.

Do you need to give me permission? am I doing something obvious wrong?

[hbredin]

Thanks for contributing to the library!
I believe you do not need to add the main repo as an "upstream".

Navigate to your own branch in your own repo and there should be a "Pull request" button somewhere on Github.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.