ESPnet-Spk part 3 - inference every epoch using EER #5314

Jungjee · 2023-07-20T19:03:39Z

This PR is a follow-up PR on #5258.
We only had training available, whereas through this PR, we finish the ESPnet-SPK basic pipeline.
Now, EER is calculated every epoch.

for more information, see https://pre-commit.ci

…aker2

codecov · 2023-07-20T21:34:24Z

Codecov Report

Merging #5314 (249bf6e) into master (353c01f) will decrease coverage by 0.09%.
The diff coverage is 58.58%.

@@            Coverage Diff             @@
##           master    #5314      +/-   ##
==========================================
- Coverage   76.21%   76.13%   -0.09%     
==========================================
  Files         669      672       +3     
  Lines       59566    59859     +293     
==========================================
+ Hits        45396    45571     +175     
- Misses      14170    14288     +118

Flag	Coverage Δ
test_integration_espnet1	`65.93% <ø> (ø)`
test_integration_espnet2	`48.02% <47.54%> (-0.01%)`	⬇️
test_python	`66.40% <33.12%> (-0.17%)`	⬇️
test_utils	`23.00% <ø> (-0.18%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet2/schedulers/cosine_anneal_warmup_restart.py	`17.30% <17.30%> (ø)`
espnet2/iterators/sequence_iter_factory.py	`85.39% <27.27%> (-8.20%)`	⬇️
espnet2/train/preprocessor.py	`44.83% <47.22%> (+0.75%)`	⬆️
espnet2/train/spk_trainer.py	`75.51% <75.51%> (ø)`
espnet2/utils/eer.py	`93.33% <93.33%> (ø)`
espnet2/spk/espnet_model.py	`89.06% <100.00%> (-2.75%)`	⬇️
espnet2/spk/loss/aamsoftmax.py	`100.00% <100.00%> (ø)`
espnet2/spk/loss/abs_loss.py	`100.00% <100.00%> (ø)`
espnet2/tasks/abs_task.py	`76.89% <100.00%> (+0.22%)`	⬆️
espnet2/tasks/spk.py	`92.85% <100.00%> (+0.06%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

…aker2

for more information, see https://pre-commit.ci

egs2/mini_an4/spk1/local/make_trial.py

Emrys365 · 2023-07-21T00:33:25Z

egs2/mini_an4/spk1/local/make_trial.py

+        lines = f.readlines()
+
+    joint_key = lines[0].strip().split(" ")[0] + "*" + lines[1].strip().split(" ")[0]
+    with open(os.path.join(sys.argv[2], "trial.scp"), "w") as f:


Maybe it's better to rename trial.scp to trial1.scp to make them more consistent.

Jungjee · 2023-07-21T14:40:36Z

Great! current version passes all CI.
@Emrys365, @sw005320, let's aim for merging this pr as well within today.

I will also add another one PR (will be small compared to this) to add some metric learning objective function (thinking of SimCLR style contrastive at the moment).

sw005320 · 2023-07-21T15:05:29Z

Can you fix the conflicts due to the merge of your previous PRs?

mergify · 2023-07-21T15:06:33Z

This pull request is now in conflict :(

Jungjee · 2023-07-21T15:08:34Z

Just resolved all conflicts! Now same to my local branch which passes CI tests

ftshijt

Great effort! some minor comments.

egs2/TEMPLATE/spk1/spk.sh

egs2/voxceleb/spk1/conf/tuning/train_RawNet3_sgdr.yaml

for more information, see https://pre-commit.ci

kan-bayashi · 2023-07-21T23:42:51Z

Hi @Jungjee, Could you minimize the parameters for mini_an4 as possible as you can ?
You can refer #5324

Jungjee · 2023-07-22T00:12:41Z

@kan-bayashi , the number of parameter of ESPnet-SPK's ci model was 0.1M and I further reduced it to 0.01M. Hope this is small enough. Let me know if you need further alterations.

espnet2/spk/loss/aamsoftmax.py

espnet2/train/spk_trainer.py

Co-authored-by: Wangyou Zhang <C0me_On@163.com>

for more information, see https://pre-commit.ci

espnet2/train/spk_trainer.py

sw005320

LGTM!

sw005320 · 2023-07-22T02:19:12Z

espnet2/train/preprocessor.py

This is more for a question.
Is there a way to keep the same random seed even in the multiple worker environment, so that we could reproduce the same augmented data?

Also, this extension is a very good example and I really appreciate both @Jungjee and @Emrys365’s efforts!

This is more for a question. Is there a way to keep the same random seed even in the multiple worker environment, so that we could reproduce the same augmented data?

I think it is handled for the sequence iterator by the worker_init_fn:
https://github.com/espnet/espnet/blob/master/espnet2/iterators/sequence_iter_factory.py#L13-L17

I see, thx!

…aker2

for more information, see https://pre-commit.ci

add epoch wise inference using EER

e884009

mergify bot added ESPnet2 README CI Travis, Circle CI, etc Installation labels Jul 20, 2023

pre-commit-ci bot and others added 6 commits July 20, 2023 19:04

[pre-commit.ci] auto fixes from pre-commit.com hooks

363b844

for more information, see https://pre-commit.ci

Merge branch 'master' into speaker2

f8dc909

make dummy test trial for ci

e5c4a6c

[pre-commit.ci] auto fixes from pre-commit.com hooks

c1b9937

for more information, see https://pre-commit.ci

make test_python work

91a777f

Merge branch 'speaker2' of https://github.com/Jungjee/espnet into spe…

6316eed

…aker2

sw005320 and others added 4 commits July 21, 2023 06:35

Merge branch 'master' into speaker2

d4238e4

make test_integration2 work

447eb83

Merge branch 'speaker2' of https://github.com/Jungjee/espnet into spe…

1833acb

…aker2

[pre-commit.ci] auto fixes from pre-commit.com hooks

04bb321

for more information, see https://pre-commit.ci

Emrys365 reviewed Jul 21, 2023

View reviewed changes

sw005320 self-requested a review July 21, 2023 00:35

Jungjee mentioned this pull request Jul 21, 2023

ESPNet-SPK part 2 - training #5258

Merged

mergify bot added the conflicts label Jul 21, 2023

Merge branch 'master' into speaker2

f0bacaa

mergify bot removed the conflicts label Jul 21, 2023

Emrys365 added 2 commits July 21, 2023 11:40

Add data augmentation for speaker taks

e91c0e3

Merge branch 'speaker2' of github.com:Jungjee/espnet into speaker2

5586713

ftshijt reviewed Jul 21, 2023

View reviewed changes

egs2/TEMPLATE/spk1/spk.sh Outdated Show resolved Hide resolved

egs2/voxceleb/spk1/conf/tuning/train_RawNet3_sgdr.yaml Outdated Show resolved Hide resolved

revoke undoed patch

3708b6f

Jungjee and others added 5 commits July 21, 2023 17:34

Merge branch 'master' into speaker2

920b51c

Add unit tests

dd51257

[pre-commit.ci] auto fixes from pre-commit.com hooks

b1088c1

for more information, see https://pre-commit.ci

Fix a bug in SPK preprocessor

37f4ab4

Merge branch 'master' into speaker2

19a09e0

fix minor bug, make ci model smaller

ba66a76

sw005320 marked this pull request as draft July 22, 2023 00:53

Jungjee marked this pull request as ready for review July 22, 2023 01:01

Jungjee closed this Jul 22, 2023

Jungjee reopened this Jul 22, 2023

Emrys365 reviewed Jul 22, 2023

View reviewed changes

espnet2/spk/loss/aamsoftmax.py Outdated Show resolved Hide resolved

Emrys365 reviewed Jul 22, 2023

View reviewed changes

espnet2/train/spk_trainer.py Outdated Show resolved Hide resolved

Jungjee and others added 3 commits July 21, 2023 21:45

Update espnet2/spk/loss/aamsoftmax.py

9470be5

Co-authored-by: Wangyou Zhang <C0me_On@163.com>

Update espnet2/train/spk_trainer.py

19d5d81

Co-authored-by: Wangyou Zhang <C0me_On@163.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

267bfb4

for more information, see https://pre-commit.ci

Emrys365 reviewed Jul 22, 2023

View reviewed changes

espnet2/train/spk_trainer.py Outdated Show resolved Hide resolved

Emrys365 reviewed Jul 22, 2023

View reviewed changes

espnet2/train/spk_trainer.py Outdated Show resolved Hide resolved

Jungjee and others added 2 commits July 21, 2023 22:05

reflect @Emrys365 comments

e5297a1

Merge branch 'master' into speaker2

caa2a73

sw005320 approved these changes Jul 22, 2023

View reviewed changes

sw005320 added the auto-merge Enable auto-merge label Jul 22, 2023

sw005320 added this to the v.202307 milestone Jul 22, 2023

Jungjee and others added 4 commits July 22, 2023 00:24

reflect local changes

12fea27

Merge branch 'speaker2' of https://github.com/Jungjee/espnet into spe…

9790e6d

…aker2

Merge branch 'master' into speaker2

9aaad7c

[pre-commit.ci] auto fixes from pre-commit.com hooks

249bf6e

for more information, see https://pre-commit.ci

mergify bot merged commit 8c264ff into espnet:master Jul 22, 2023
24 of 25 checks passed

Jungjee deleted the speaker2 branch August 3, 2023 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESPnet-Spk part 3 - inference every epoch using EER #5314

ESPnet-Spk part 3 - inference every epoch using EER #5314

Jungjee commented Jul 20, 2023

codecov bot commented Jul 20, 2023 •

edited

Emrys365 Jul 21, 2023

Jungjee commented Jul 21, 2023

sw005320 commented Jul 21, 2023

mergify bot commented Jul 21, 2023

Jungjee commented Jul 21, 2023

ftshijt left a comment

kan-bayashi commented Jul 21, 2023

Jungjee commented Jul 22, 2023

sw005320 left a comment

sw005320 Jul 22, 2023

sw005320 Jul 22, 2023

Emrys365 Jul 22, 2023

sw005320 Jul 22, 2023

ESPnet-Spk part 3 - inference every epoch using EER #5314

ESPnet-Spk part 3 - inference every epoch using EER #5314

Conversation

Jungjee commented Jul 20, 2023

codecov bot commented Jul 20, 2023 • edited

Codecov Report

Emrys365 Jul 21, 2023

Choose a reason for hiding this comment

Jungjee commented Jul 21, 2023

sw005320 commented Jul 21, 2023

mergify bot commented Jul 21, 2023

Jungjee commented Jul 21, 2023

ftshijt left a comment

Choose a reason for hiding this comment

kan-bayashi commented Jul 21, 2023

Jungjee commented Jul 22, 2023

sw005320 left a comment

Choose a reason for hiding this comment

sw005320 Jul 22, 2023

Choose a reason for hiding this comment

sw005320 Jul 22, 2023

Choose a reason for hiding this comment

Emrys365 Jul 22, 2023

Choose a reason for hiding this comment

sw005320 Jul 22, 2023

Choose a reason for hiding this comment

codecov bot commented Jul 20, 2023 •

edited