CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs #5183

popcornell · 2023-05-24T20:59:06Z

@Emrys365 does this fix it ?

codecov · 2023-05-24T21:33:14Z

Codecov Report

Merging #5183 (fc344e8) into master (0216f85) will increase coverage by 8.60%.
The diff coverage is n/a.

❗ Current head fc344e8 differs from pull request most recent head afeecc1. Consider uploading reports for the commit afeecc1 to get more accurate results

@@            Coverage Diff             @@
##           master    #5183      +/-   ##
==========================================
+ Coverage   65.94%   74.54%   +8.60%     
==========================================
  Files         640      640              
  Lines       57267    57267              
==========================================
+ Hits        37762    42688    +4926     
+ Misses      19505    14579    -4926

Flag	Coverage Δ
test_integration_espnet1	`66.28% <ø> (?)`
test_integration_espnet2	`47.58% <ø> (?)`
test_python	`65.28% <ø> (ø)`
test_utils	`23.28% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 107 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Emrys365 · 2023-05-24T23:37:47Z

I think we can ignore this issue. Because the default value of use_chime6_falign in egs2/chime7_task1/asr1/run.sh is 0. So the CHiME7_DASR_falign will not be used.

But probably some descriptions should be added to mention when we should set this argument to 1.

…1_diar # Conflicts: # egs2/chime7_task1/asr1/run.sh

for more information, see https://pre-commit.ci

…1_diar

for more information, see https://pre-commit.ci

popcornell · 2023-05-30T14:25:54Z

FYI changed the scope of this PR to be more broad.

adds support for evaluation set inference by fixing some minor things.
adds support for using pre-computed baseline diarization output JSONs

simpleoier

I left some comments in the code, which may cause some issues. But I'm not 100% sure myself.

simpleoier · 2023-05-30T18:57:21Z

egs2/chime7_task1/diar_asr1/run.sh

@@ -51,7 +52,7 @@ gss_max_batch_dur=90

 # ASR config
 use_pretrained=
-decode_only=1
+decode_only=


When it is empty, I'm afraid line 182 will raise an error.

--decode_only $decode_only --gss-max-batch-dur $gss_max_batch_dur \

I should add quotes, good catch

simpleoier · 2023-05-30T18:59:45Z

egs2/chime7_task1/diar_asr1/run.sh

@@ -166,5 +181,5 @@ if [ ${stage} -le 4 ] && [ $stop_stage -ge 4 ]; then
        --use-pretrained $use_pretrained \
        --decode_only $decode_only --gss-max-batch-dur $gss_max_batch_dur \
        --gss-iterations 5 \
-        --diar-score 1
+        --diar-score 1 \


Base on my previous experience, this extra \ will raise an error. You need an extra blank line.

simpleoier · 2023-05-30T19:02:24Z

egs2/chime7_task1/diar_asr1/run.sh

+  log "Using organizer-provided JSON manifests from the baseline diarization system."
+  #git clone
+  #mv
+  stage=4


Just want to double check if we can skip stage 3 if we have this. How can we download baseline diarization results?

Yeah you are right I changed it.
Now I also added the downloading part

simpleoier · 2023-05-30T19:02:59Z

egs2/chime7_task1/diar_asr1/README.md

@@ -147,14 +147,17 @@ You can also play with diarization hyperparameters such as:

 as said merge-closer can have quite an impact on the final WER.

+**NOTE**
+We found the diarization baseline to be highly sensitive to the


Not a complete note?

simpleoier · 2023-05-30T19:05:32Z

egs2/chime7_task1/asr1/run.sh

@@ -67,7 +67,7 @@ nbpe=500
 asr_max_epochs=8
 # put popcornell/chime7_task1_asr1_baseline if you want to test with pretrained model
 use_pretrained=
-decode_only=0
+decode_only=


I'm concerned that when it is empty, line 206 data_opts+=" --decode-only $decode_only" will raise some error, e.g. Error: No positional arguments are required.

now it is an empty string.

…task1_diar

… into chime7task1_diar

…1_diar

popcornell · 2023-06-02T09:26:55Z

I think this PR is ready.
@simpleoier I have addressed your comments

popcornell · 2023-06-02T10:15:08Z

FYI: there is still the numpy problem:

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/Users/runner/work/espnet/espnet/tools/venv/envs/espnet/lib/python3.10/site-packages/numpy-1.24.3.dist-info/METADATA'

popcornell · 2023-06-06T19:32:38Z

@simpleoier Can we merge this ?

simpleoier

LGTM!
But there are many changes in non-relevant files. Probably due to black version. Also, @popcornell, there is one failure test: check_kaldi_symlinks, can you solve it?
@sw005320 Can we merge this?

popcornell · 2023-06-07T05:14:57Z

I can rechange them back.
What black version are you using btw ?

simpleoier · 2023-06-07T14:25:27Z

@popcornell my black is black, 23.3.0 (compiled: yes)
BTW, here are the requirement for espnet test tools.

        "pytest>=3.3.0",
        "pytest-timeouts>=1.2.1",
        "pytest-pythonpath>=0.7.3",
        "pytest-cov>=2.7.1",
        "hacking>=2.0.0",
        "mock>=2.0.0",
        "pycodestyle",
        "jsondiff<2.0.0,>=1.2.0",
        "flake8>=3.7.8",
        "flake8-docstrings>=1.3.1",
        "black",
        "isort",

popcornell · 2023-06-07T15:11:30Z

I guess then it is fine ?! Like the black version is not enforced explicitly.
Maybe we should fix it however and upgrade it explicitly by one PR once in a while (with explicit reformatting of everything).
I think these questions can be answered only by @sw005320

Emrys365 · 2023-06-07T22:31:16Z

The black formatting is now always enforced in https://github.com/espnet/espnet/blob/master/.pre-commit-config.yaml#L16-L20, which will be done every time you push a new commit.

sw005320 · 2023-06-08T05:12:40Z

Yes, this part is not an issue.
However, CI says that egs2/TEMPLATE/asr1/utils/data/extend_segment_times.py and tools/kaldi/egs/wsj/s5/utils/data/extend_segment_times.py are different.
https://github.com/espnet/espnet/actions/runs/5192602992/jobs/9361995304?pr=5183

If they are necessary changes, it would be better to move this file to the local (recipe-specific) directory.
If their changes are due to the format only, it would be better to change it.

The issue is that they look very different due to the different formats, and I could not figure out the above changes.

…1_diar

for more information, see https://pre-commit.ci

Chime7task1 diar eval

popcornell · 2023-06-12T21:53:12Z

Made a fresh PR because it is faster. Moved to #5228

popcornell added 2 commits May 24, 2023 22:57

Update run.sh, making sure forced alignment only used in inference

fd55895

Merge branch 'espnet:master' into chime7task1_diar

c8a6b52

mergify bot added the ESPnet2 label May 24, 2023

popcornell added 5 commits May 28, 2023 16:58

add session name to rttms

b223d2e

add to rttms the sess name also in scoring script

68d8cfe

check for even gpus number

a5061b3

enable evaluation set inference

ba6ed67

Merge remote-tracking branch 'samco/chime7task1_diar' into chime7task…

a9e852e

…1_diar # Conflicts: # egs2/chime7_task1/asr1/run.sh

mergify bot added README CI Travis, Circle CI, etc Installation labels May 28, 2023

pre-commit-ci bot and others added 7 commits May 28, 2023 16:18

[pre-commit.ci] auto fixes from pre-commit.com hooks

3ddb000

for more information, see https://pre-commit.ci

decode_only now is a string

71214db

minor changes to allow for eval inference

dd22f00

Merge remote-tracking branch 'samco/chime7task1_diar' into chime7task…

4887d54

…1_diar

[pre-commit.ci] auto fixes from pre-commit.com hooks

9e6b7c0

for more information, see https://pre-commit.ci

minor changes to allow for eval inference

99b29ad

Merge branch 'master' into chime7task1_diar

fb42d9d

sw005320 added this to the v.202307 milestone May 30, 2023

sw005320 requested a review from simpleoier May 30, 2023 14:17

popcornell changed the title ~~Prevent using forced alignment annotation to train the ASR model~~ CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs May 30, 2023

popcornell marked this pull request as draft May 30, 2023 14:27

simpleoier reviewed May 30, 2023

View reviewed changes

popcornell added 4 commits June 1, 2023 10:43

remove line at end

e82b3b1

Merge branch 'master' of https://github.com/espnet/espnet into chime7…

2fcf843

…task1_diar

Merge branch 'chime7task1_diar' of https://github.com/popcornell/espnet…

c3d90ee

… into chime7task1_diar

added support for pre-computed diarization

3c0215d

popcornell marked this pull request as ready for review June 1, 2023 09:30

popcornell added 8 commits June 1, 2023 13:11

shellcheck fix

acb9ca1

applied black

34d4e27

fix shellcheck

9526855

fix copy using pre-computed diarization

487416d

Merge branch 'master' into chime7task1_diar

0022619

avoid scoring when running inference on evaluatgion

34f23b4

Merge remote-tracking branch 'samco/chime7task1_diar' into chime7task…

c3e2055

…1_diar

Merge branch 'master' into chime7task1_diar

6d89f1a

popcornell added 3 commits June 3, 2023 16:20

Merge branch 'master' into chime7task1_diar

59a4563

Merge branch 'master' into chime7task1_diar

5581b8b

Merge branch 'master' into chime7task1_diar

fc344e8

simpleoier approved these changes Jun 7, 2023

View reviewed changes

popcornell and others added 6 commits June 12, 2023 23:29

revert black

cfe01d3

Merge remote-tracking branch 'samco/chime7task1_diar' into chime7task…

dc87c58

…1_diar

revert black

4438c58

[pre-commit.ci] auto fixes from pre-commit.com hooks

afeecc1

for more information, see https://pre-commit.ci

adding changes

aaef96a

Merge pull request #4 from popcornell/chime7task1_diar_eval

2bb0484

Chime7task1 diar eval

popcornell closed this Jun 12, 2023

popcornell mentioned this pull request Jun 12, 2023

Closed CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs (new PR) #5228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs #5183

CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs #5183

popcornell commented May 24, 2023

codecov bot commented May 24, 2023 •

edited

Emrys365 commented May 24, 2023

popcornell commented May 30, 2023

simpleoier left a comment

simpleoier May 30, 2023

popcornell May 30, 2023

simpleoier May 30, 2023

simpleoier May 30, 2023

popcornell Jun 1, 2023

simpleoier May 30, 2023

popcornell Jun 1, 2023

simpleoier May 30, 2023

popcornell Jun 1, 2023

popcornell commented Jun 2, 2023

popcornell commented Jun 2, 2023

popcornell commented Jun 6, 2023

simpleoier left a comment •

edited

popcornell commented Jun 7, 2023

simpleoier commented Jun 7, 2023

popcornell commented Jun 7, 2023

Emrys365 commented Jun 7, 2023 •

edited

sw005320 commented Jun 8, 2023

popcornell commented Jun 12, 2023

CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs #5183

CHiME-7 DASR adding evaluation inference + adding support to use diarization baseline "pre-computed" JSONs #5183

Conversation

popcornell commented May 24, 2023

codecov bot commented May 24, 2023 • edited

Codecov Report

Emrys365 commented May 24, 2023

popcornell commented May 30, 2023

simpleoier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

popcornell commented Jun 2, 2023

popcornell commented Jun 2, 2023

popcornell commented Jun 6, 2023

simpleoier left a comment • edited

Choose a reason for hiding this comment

popcornell commented Jun 7, 2023

simpleoier commented Jun 7, 2023

popcornell commented Jun 7, 2023

Emrys365 commented Jun 7, 2023 • edited

sw005320 commented Jun 8, 2023

popcornell commented Jun 12, 2023

codecov bot commented May 24, 2023 •

edited

simpleoier left a comment •

edited

Emrys365 commented Jun 7, 2023 •

edited