[WIP] Multilabel Detection #891

hadware · 2022-02-14T16:11:25Z

This is a new PR for the VTC feature, this time based on a cleaner implem. I'm making a new PR as to keep the former branch "clean" (and prevent any mishaps).

What is done:

renaming the SpeakerTracking task into a MultilabelDetection task
added MultilabelPipeline
update MultilabelFscore.report()
tested the new preprocessor

What's to be done:

re-test the new implem on our clinical data (as well as the child data from @MarvinLvn )
maybe a couple of unit tests (especially for the preprocessor)
maybe make the aggregated "multilabel" fscore duration-based instead of file-based

…efault config

# Conflicts: # pyannote/audio/pipelines/multilabel_detection.py

…sors # Conflicts: # pyannote/audio/cli/train_config/hydra/train.yaml

codecov · 2022-02-16T15:30:37Z

Codecov Report

Merging #891 (158e809) into develop (26cddd9) will decrease coverage by 1.09%.
The diff coverage is 6.01%.

❗ Current head 158e809 differs from pull request most recent head 4b4f249. Consider uploading reports for the commit 4b4f249 to get more accurate results

@@             Coverage Diff             @@
##           develop     #891      +/-   ##
===========================================
- Coverage    35.32%   34.23%   -1.10%     
===========================================
  Files           58       59       +1     
  Lines         3459     3584     +125     
===========================================
+ Hits          1222     1227       +5     
- Misses        2237     2357     +120

Impacted Files	Coverage Δ
pyannote/audio/pipelines/__init__.py	`0.00% <0.00%> (ø)`
pyannote/audio/pipelines/multilabel.py	`0.00% <0.00%> (ø)`
pyannote/audio/utils/metric.py	`0.00% <0.00%> (ø)`
pyannote/audio/utils/preprocessors.py	`0.00% <0.00%> (ø)`
pyannote/audio/tasks/segmentation/multilabel.py	`90.90% <70.00%> (ø)`
pyannote/audio/tasks/__init__.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26cddd9...4b4f249. Read the comment docs.

hadware · 2022-03-08T14:02:15Z

Alright, I've tested it again using the clinical data, the scores are analogous to what I got in the table in #694 . Marianne is going to test it on babytrain data, but she's currently on holiday (and for two weeks). If that's ok with you, i'd like to proceed to the part where you make sure that the code is mergeable, as I think that this should be good enough for a merge.

pyannote/audio/tasks/segmentation/voice_type_classification.py

…cstring tweaks.

hbredin · 2022-03-15T08:08:54Z

Seems to go in the right direction.
I will make a pass and update a few things myself.

In the meantime, could you add a notebook showing how to use this new objects to train a male/female/speech multi-label pipeline on top of Debug.SpeakerDiarization.Debug protocol available in tests/data/database.yml.

Basically, this would illustrate

the use of a preprocessor to convert Mxxxx labels to male and Fxxxx labels to female
the use of a VoiceTypeClassifierPreprocessor preprocessor to add the union of male and female as speech
a way to chain them
the new MultilabelDetection task
the new MultilabelDetectionpipeline

* rename MultilabelDetectionPipeline to MultilabelDetection * rename MultilabelFMeasure to MacroAverageFMeasure and move it to pyannote.audio.utils.metric * rename VoiceTypeClassifierPreprocessor to DeriveMetaLabels and move it to pyannote.audio.utils.preprocessors * make segmentation model mandatory and remove default parameters * add support for pipeline hook * rename chunk_labels to ordered_labels

hbredin

I spent some time looking at the code. Looks good!
I do have a few questions and would love an update on the notebook :) Does it run as it is right now (refering to the link you shared earlier)?

pyannote/audio/pipelines/multilabel_detection.py

hbredin · 2022-06-08T19:45:29Z

pyannote/audio/pipelines/multilabel_detection.py

+                for label in self._classes
+            }
+        )
+        # TODO: would it make sense to share min_duration_{on|off} between classes?


For this particular task that may lead to a huge hyperparameter search space, we could also force onset == offset (once again through a flag use_hysteresis).

hbredin · 2022-06-08T19:52:13Z

pyannote/audio/pipelines/multilabel_detection.py

+                for label in self._classes
+            }
+        )
+        # TODO: would it make sense to share min_duration_{on|off} between classes?


But, let's do that in a separate PR.

hbredin · 2022-06-08T19:56:49Z

pyannote/audio/tasks/segmentation/multilabel_detection.py

+            num_workers: int = None,
+            pin_memory: bool = False,
+            augmentation: BaseWaveformTransform = None,
+            metric: Union[Metric, Sequence[Metric], Dict[str, Metric]] = None,


I haven't thought a lot about the validation metric for this type of task.
It relies on AUROC right? But what does it mean for multilabel classification?

It does rely on AUROC I think. I don't have an answer to this. I just used the metric provided by the class without thinking a second about what I was doing. 🙃

pyannote/audio/tasks/segmentation/multilabel_detection.py

hbredin · 2022-06-16T15:13:03Z

FYI - I just released pyannote.pipeline 2.3 with support for ParamDict.

hadware · 2022-06-20T14:21:38Z

All right

https://colab.research.google.com/drive/1JX3x4t_QTHMr8hzjw3Em8eQBQnQ3UBgk#scrollTo=U9lwdYcTHyp_ is working fine as an example (with your input on using a subset of AMI). The inferred annotations look pretty good, and the tuning step amounts to a noticeable gain of bout 5% of IER.

If you think this is good enough, I can then re-add the comments and tutorial text from the original notebook (with a couple of tweaks) and we can call it a day. I'll also merge the latest commits from "upstream".

hadware · 2022-06-20T14:30:29Z

I'll also take care of the comments you submitted in #891 (review)

hbredin · 2022-06-20T14:58:42Z

All right

https://colab.research.google.com/drive/1JX3x4t_QTHMr8hzjw3Em8eQBQnQ3UBgk#scrollTo=U9lwdYcTHyp_ is working fine as an example (with your input on using a subset of AMI). The inferred annotations look pretty good, and the tuning step amounts to a noticeable gain of bout 5% of IER.

Awesome! I was able to run the whole thing successfully in 10min or so.
That is a great demo!

If you think this is good enough, I can then re-add the comments and tutorial text from the original notebook (with a couple of tweaks) and we can call it a day. I'll also merge the latest commits from "upstream".

Yes, please!

…ipeline

hadware · 2022-06-20T17:02:38Z

I re-added the text to the colab notebook, and added, implemented and tested the share_min_duration option

hbredin · 2022-06-30T12:22:28Z

Hey @hadware, would you mind adding the notebook to this PR?

This should go in the tutorials directory with a link added in the README "Documentation / Miscellaneous" section.

…ials/ folder. added entry in README.md.

hadware · 2022-06-30T17:45:10Z

Done.

hadware · 2022-06-30T17:45:40Z

Is there anything more needed to merge this?

hbredin · 2022-07-01T08:38:44Z

Hurrah! Looking forward to lots of new applications!

I removed the notebook because it felt like it was not clean enough.
Anyway, all notebooks should probably be updated to run on Google Colab with the new AMI Mini.
I'll open a dedicated issue.

Thanks again @hadware for the hard work and sorry for being such a pinailleur...

hadware · 2022-07-01T10:38:35Z

The pinailling is part of the fun 🥵

I'd like to pretrain a couple of models to host then on hugginface along with the others. What kind of classes/train datasets would you suggest? I was thinking about MALE/FEMALE using AMI/VoxCeleb, but that's it. Any other ideas?

hadware and others added 17 commits January 7, 2022 20:09

Re-added files from backup branch

5fe2153

Re-added to init

1d59d44

Re-added __init__ references, re-added VoiceTypeClassification.yaml d…

7ebbc74

…efault config

Merge branch 'develop' into feat/vtc

14998e6

Merge branch 'develop' into feat/vtc

fd1e3bc

Fixing Fscore metric, fixing MultilabelPipeline apply code

5cc74da

Merge remote-tracking branch 'origin/feat/vtc' into feat/vtc

98c97d2

Fixed multilabel pipeline apply method.

934cf01

Merge branch 'develop' into feat/vtc

ab7112e

Fixing imports

1f0c63e

Merge remote-tracking branch 'origin/feat/vtc' into feat/vtc

a5947c3

# Conflicts: # pyannote/audio/pipelines/multilabel_detection.py

Fixing imports (again)

b0ec1a2

Fixing imports (again^2)

a28bacb

Started re-factoring the VTC task into the speaker-tracking task.

e846109

Fixed VTC preprocessor

d569f2a

Added a VTC tutorial. The custom fscore metric should be good.

d4584c1

Merge remote-tracking branch 'bredin/develop' into feat-vtc-preproces…

e9df3f8

…sors # Conflicts: # pyannote/audio/cli/train_config/hydra/train.yaml

hadware and others added 3 commits February 17, 2022 16:11

Merge branch 'develop' into feat-vtc-preprocessors

5f52175

Fixed multilabel pipeline init.

deabd36

Merge branch 'develop' into feat-vtc-preprocessors

ebd12a3

hbredin added 2 commits March 13, 2022 11:38

Merge branch 'develop' into feat-vtc-preprocessors

5777043

Merge branch 'develop' into feat-vtc-preprocessors

796926a

hbredin reviewed Mar 14, 2022

View reviewed changes

pyannote/audio/tasks/segmentation/voice_type_classification.py Outdated Show resolved Hide resolved

hadware and others added 3 commits March 14, 2022 21:44

Documentation for the VoiceTypeClassifierPreprocessor. A couple of do…

5a26f17

…cstring tweaks.

Removed VoiceTypeClassification.yaml .

fd1acd7

Merge branch 'develop' into feat-vtc-preprocessors

be54c45

hbredin reviewed Jun 8, 2022

View reviewed changes

Merge branch 'develop' into feat-vtc-preprocessors

a51b244

Merge branch 'develop' into feat-vtc-preprocessors

345a262

hadware added 2 commits June 20, 2022 16:22

Merge branch 'develop' into feat-vtc-preprocessors

5fbcb33

Removed the multilabel_detection.ipynb notebook

4d0c36e

Added shared min_duration parameters to the multilabel_detection.py p…

5372c56

…ipeline

hbredin mentioned this pull request Jun 30, 2022

music activity detection? #1023

Closed

Re-added the ipynb version of multilabel_detection.ipynb to the tutor…

897d99f

…ials/ folder. added entry in README.md.

hbredin added 10 commits July 1, 2022 09:25

Update README.md

68e55de

Update and rename multilabel_detection.py to multilabel.py

39fff5a

Update __init__.py

b69cbd1

Update README.md

e49eada

Update __init__.py

2edf9aa

Update and rename multilabel_detection.py to multilabel.py

e7e18db

Update __init__.py

7dc6296

Update test_reproducibility.py

158e809

Update README.md

551b2c4

Delete multilabel_detection.ipynb

4b4f249

hbredin merged commit 1dac64a into pyannote:develop Jul 1, 2022

hbredin mentioned this pull request Jul 1, 2022

Useful multi-label models #1027

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Multilabel Detection #891

[WIP] Multilabel Detection #891

hadware commented Feb 14, 2022 •

edited

codecov bot commented Feb 16, 2022 •

edited

hadware commented Mar 8, 2022

hbredin commented Mar 15, 2022

hbredin left a comment

hbredin Jun 8, 2022

hbredin Jun 8, 2022

hbredin Jun 8, 2022

hadware Jun 20, 2022

hbredin commented Jun 16, 2022 •

edited

hadware commented Jun 20, 2022

hadware commented Jun 20, 2022

hbredin commented Jun 20, 2022

hadware commented Jun 20, 2022

hbredin commented Jun 30, 2022 •

edited

hadware commented Jun 30, 2022

hadware commented Jun 30, 2022

hbredin commented Jul 1, 2022

hadware commented Jul 1, 2022

[WIP] Multilabel Detection #891

[WIP] Multilabel Detection #891

Conversation

hadware commented Feb 14, 2022 • edited

codecov bot commented Feb 16, 2022 • edited

Codecov Report

hadware commented Mar 8, 2022

hbredin commented Mar 15, 2022

hbredin left a comment

Choose a reason for hiding this comment

hbredin Jun 8, 2022

Choose a reason for hiding this comment

hbredin Jun 8, 2022

Choose a reason for hiding this comment

hbredin Jun 8, 2022

Choose a reason for hiding this comment

hadware Jun 20, 2022

Choose a reason for hiding this comment

hbredin commented Jun 16, 2022 • edited

hadware commented Jun 20, 2022

hadware commented Jun 20, 2022

hbredin commented Jun 20, 2022

hadware commented Jun 20, 2022

hbredin commented Jun 30, 2022 • edited

hadware commented Jun 30, 2022

hadware commented Jun 30, 2022

hbredin commented Jul 1, 2022

hadware commented Jul 1, 2022

hadware commented Feb 14, 2022 •

edited

codecov bot commented Feb 16, 2022 •

edited

hbredin commented Jun 16, 2022 •

edited

hbredin commented Jun 30, 2022 •

edited