ASR evaluator #5728

fayejf · 2023-01-03T21:23:31Z

What does this PR do ?

A tool to thoroughly evaluate an ASR model.

Collection: ASR

Changelog

Simple step to evaluate a model in all three modes currently supported by NeMo: offline, chunked, and offline_by_chunked.
On-the-fly data augmentation (silence, noise, etc.,) for ASR robustness evaluation.
Investigate the model's performance by detailed insertion, deletion, and substitution error rates for each and all samples.
Evaluate models' reliability on different target groups such as gender, and audio length if metadata is presented.

Usage

python asr_evaluator.py \
engine.pretrained_name="stt_en_conformer_transducer_large" \
engine.inference_mode.mode="offline" \
engine.test_ds.augmentor.noise.manifest_path=<manifest file for noise data>

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Note because the PR is getting bigger, on-the-fly augmentation for buffered inference mode would be included in another PR soon.

Signed-off-by: fayejf <fayejf07@gmail.com>

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

for more information, see https://pre-commit.ci

nemo/collections/asr/parts/utils/eval_utils.py

tools/speech_evaluator/asr_evaluator.py

nemo/collections/asr/parts/utils/transcribe_utils.py

examples/asr/transcribe_speech.py

Signed-off-by: fayejf <fayejf07@gmail.com>

tools/speech_evaluator/asr_evaluator.py

nemo/collections/asr/parts/utils/eval_utils.py

Signed-off-by: fayejf <fayejf07@gmail.com>

nemo/collections/asr/parts/utils/eval_utils.py

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52

Good work, just a few small comments. We should make sure that scripts inside nemo.collection do not depend on scripts in examples/ and functions in NeMo be language-agnostic.

stevehuang52 · 2023-01-08T15:43:48Z

nemo/collections/asr/models/ctc_models.py

@@ -182,6 +186,9 @@ def transcribe(
                    'channel_selector': channel_selector,
                }

+                if augmentor_config:
+                    config['augmentor_config'] = augmentor_config


'augmentor_config' can be simplified to just 'augmentor', since it's part of the config variable we can drop the _config suffix.

make sense. updated.

stevehuang52 · 2023-01-08T15:45:41Z

nemo/collections/asr/parts/utils/eval_utils.py

+    if (cfg.model_path and cfg.pretrained_name) or (not cfg.model_path and not cfg.pretrained_name):
+        raise ValueError("Please specify either cfg.model_path or cfg.pretrained_name!")
+
+    if cfg.inference_mode.mode == "offline":


cfg.inference_mode.mode seems a bit redundant, can we change to just cfg.inference.mode?

stevehuang52 · 2023-01-09T14:34:05Z

nemo/collections/asr/parts/utils/eval_utils.py

+        f"model_stride={cfg.inference_mode.model_stride} ",
+        shell=True,
+        check=True,
+    )


It's unsafe for things inside NeMo collections to depend on NeMo example scripts, since examples may not be installed or can be lost/modified easily. If this function has to use something from the example folder, it's better to move the whole function to an example scripts other than putting it in NeMo

Good points! Moved eval_utils.py to utils.py in tools/asr_evaluator

stevehuang52 · 2023-01-09T14:37:26Z

nemo/collections/asr/parts/utils/eval_utils.py

+        f"eval_config_yaml={temp_eval_config_yaml_file} ",
+        shell=True,
+        check=True,
+    )


Same as before, It's unsafe for things inside NeMo collections to depend on NeMo example scripts

stevehuang52 · 2023-01-09T14:54:30Z

nemo/collections/asr/parts/utils/eval_utils.py

+            cfg.output_filename = model_name + "-" + dataset_name + "-" + mode_name + ".json"
+
+    temp_eval_config_yaml_file = "temp_eval_config.yaml"
+    with open(temp_eval_config_yaml_file, "w") as f:


Consider (but not mandatory) using tempfile.NamedTemporaryFile, which automatically manages temp files:

import tempfile with tempfile.NamedTemporaryFile(mode='w', encoding='utf-8') as f: OmegaConf.save(cfg, f) f.seek(0) # reset file pointer subprocess.run( xxxxxxx eval_config_yaml=f.name )

good idea! updated

stevehuang52 · 2023-01-09T15:42:04Z

nemo/collections/asr/parts/utils/eval_utils.py

+    return cfg, total_res
+
+
+def target_metadata_wer(manifest: str, target: str, meta_cfg: DictConfig, eval_metric: str = "wer",) -> dict:


Maybe cal_target_metadata_wer?

stevehuang52 · 2023-01-09T15:45:14Z

nemo/collections/asr/parts/utils/eval_utils.py

+
+                words = sample["words"]
+                wer_each_class[target_class]["words"] += words
+                wer_each_class[target_class]["errors"] += words * sample[eval_metric]


Better to check whether the input eval_metric is supported, and raise exceptions if it's not valid

stevehuang52 · 2023-01-09T15:46:35Z

nemo/collections/asr/parts/utils/eval_utils.py

+        logging.info(f"metadata '{target}' does not present in manifest. Skipping! ")
+        return None
+
+    values = ['samples', 'words', 'errors', 'inss', 'dels', 'subs']


Please add some doctrings on each value to explain what each one means.

updated in function docstring

stevehuang52 · 2023-01-09T16:04:56Z

tools/asr_evaluator/asr_evaluator.py

+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+
+    # Set and save random seed and git hash for reproducibility
+    random.seed(cfg.env.random_seed)


What about numpy and pytorch random seeds?

updated. Need to fix rng in perturb in another PR.

stevehuang52 · 2023-01-09T16:25:27Z

nemo/collections/asr/parts/utils/eval_utils.py

+            )
+        cfg = run_chunked_inference(cfg)
+
+    elif cfg.inference_mode.mode == "offline_by_chunked":


What's the fundamental difference between "chunked" and "offline_by_chunked"? It seems that the only difference is only that "offline_by_chunked" will set default parameters while "chunked" will not. Could you please add more docstrings to explain the different use cases?

what you said is totally correct. Added more explanation in docstring.

Signed-off-by: fayejf <fayejf07@gmail.com>

tools/asr_evaluator/asr_evaluator.py

tools/asr_evaluator/utils.py

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52 · 2023-01-10T21:56:08Z

tools/asr_evaluator/README.md

+- **ENGINE**. To conduct ASR inference.
+- **ANALYST**. To evaluate model performance based on predictions. 
+
+In Analyst, you can evaluate on all metadata if it presents in manifest. For exmaple, you can evaluate the peformance of model based on duration of each sample, such as how's the model peforms on samples smaller than 5s and longer than 5s by [[0,5][5,100000]] and get wer/cer of each slot. Or how's the model performs on postive (happy, laugh) or neural (neural) or negative mood (sad) as below. And if you set save_wer_per_class=True, it will calculate wer for all (i.e. above 5 classes + cry) classes presented in the data. 


I think you mean "neutral"? The meanings for metadata needs more explanation. For example, we can say that, with the following config, we can calculate WERs for audios in different interval groups, where each group (in seconds) is defined by [[0,2],[2,5],[5,10],[10,20],[20,100000]]. Also, we calculate the WERs for three groups of emotions, where each group is defined by [['happy','laugh'],['neural'],['sad']] (words in the same group will be treated as the same).

stevehuang52 · 2023-01-10T21:56:25Z

tools/asr_evaluator/README.md

+
+        emotion: 
+            enable: True
+            slot: [['happy','laugh'],['neural'],['sad']] # we could have 'cry' in data but not in slot we focus on.


I think you mean "neutral"?

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52

LGTM.

* backbone Signed-off-by: fayejf <fayejf07@gmail.com> * engineer and analyzer Signed-off-by: fayejf <fayejf07@gmail.com> * offline_by_chunked Signed-off-by: fayejf <fayejf07@gmail.com> * test_ds wip Signed-off-by: fayejf <fayejf07@gmail.com> * temp remove inference Signed-off-by: fayejf <fayejf07@gmail.com> * mandarin yaml Signed-off-by: fayejf <fayejf07@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * augmentor and a few updates Signed-off-by: fayejf <fayejf07@gmail.com> * address alerts and revert unnecessary changes Signed-off-by: fayejf <fayejf07@gmail.com> * Add readme Signed-off-by: fayejf <fayejf07@gmail.com> * rename Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * add missing header Signed-off-by: fayejf <fayejf07@gmail.com> * rename augmentor_config to augmentor Signed-off-by: fayejf <fayejf07@gmail.com> * raname inference_mode to inference Signed-off-by: fayejf <fayejf07@gmail.com> * move utils.py Signed-off-by: fayejf <fayejf07@gmail.com> * update temp file Signed-off-by: fayejf <fayejf07@gmail.com> * make wer cer clear Signed-off-by: fayejf <fayejf07@gmail.com> * seed_everything Signed-off-by: fayejf <fayejf07@gmail.com> * fix missing rn augmentor_config in rnnt Signed-off-by: fayejf <fayejf07@gmail.com> * fix rnnt transcribe Signed-off-by: fayejf <fayejf07@gmail.com> * add more docstring and style fix Signed-off-by: fayejf <fayejf07@gmail.com> * address codeQL Signed-off-by: fayejf <fayejf07@gmail.com> * reflect comments Signed-off-by: fayejf <fayejf07@gmail.com> * update readme Signed-off-by: fayejf <fayejf07@gmail.com> * clearer Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* backbone Signed-off-by: fayejf <fayejf07@gmail.com> * engineer and analyzer Signed-off-by: fayejf <fayejf07@gmail.com> * offline_by_chunked Signed-off-by: fayejf <fayejf07@gmail.com> * test_ds wip Signed-off-by: fayejf <fayejf07@gmail.com> * temp remove inference Signed-off-by: fayejf <fayejf07@gmail.com> * mandarin yaml Signed-off-by: fayejf <fayejf07@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * augmentor and a few updates Signed-off-by: fayejf <fayejf07@gmail.com> * address alerts and revert unnecessary changes Signed-off-by: fayejf <fayejf07@gmail.com> * Add readme Signed-off-by: fayejf <fayejf07@gmail.com> * rename Signed-off-by: fayejf <fayejf07@gmail.com> * typo fix Signed-off-by: fayejf <fayejf07@gmail.com> * small fix Signed-off-by: fayejf <fayejf07@gmail.com> * add missing header Signed-off-by: fayejf <fayejf07@gmail.com> * rename augmentor_config to augmentor Signed-off-by: fayejf <fayejf07@gmail.com> * raname inference_mode to inference Signed-off-by: fayejf <fayejf07@gmail.com> * move utils.py Signed-off-by: fayejf <fayejf07@gmail.com> * update temp file Signed-off-by: fayejf <fayejf07@gmail.com> * make wer cer clear Signed-off-by: fayejf <fayejf07@gmail.com> * seed_everything Signed-off-by: fayejf <fayejf07@gmail.com> * fix missing rn augmentor_config in rnnt Signed-off-by: fayejf <fayejf07@gmail.com> * fix rnnt transcribe Signed-off-by: fayejf <fayejf07@gmail.com> * add more docstring and style fix Signed-off-by: fayejf <fayejf07@gmail.com> * address codeQL Signed-off-by: fayejf <fayejf07@gmail.com> * reflect comments Signed-off-by: fayejf <fayejf07@gmail.com> * update readme Signed-off-by: fayejf <fayejf07@gmail.com> * clearer Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <fayejf07@gmail.com> Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

fayejf added 6 commits November 28, 2022 17:27

backbone

306291c

Signed-off-by: fayejf <fayejf07@gmail.com>

engineer and analyzer

0d30528

Signed-off-by: fayejf <fayejf07@gmail.com>

offline_by_chunked

ff4db57

Signed-off-by: fayejf <fayejf07@gmail.com>

test_ds wip

bd3d4fd

Signed-off-by: fayejf <fayejf07@gmail.com>

temp remove inference

b8d0b6e

Signed-off-by: fayejf <fayejf07@gmail.com>

mandarin yaml

707caa2

Signed-off-by: fayejf <fayejf07@gmail.com>

github-actions bot added the ASR label Jan 3, 2023

fayejf and others added 2 commits January 3, 2023 13:26

Merge branch 'main' into asr_evaluator_engine

0fb9fa4

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

94fa630

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jan 3, 2023

View reviewed changes

fayejf and others added 2 commits January 5, 2023 11:37

Merge branch 'main' into asr_evaluator_engine

7ef4ac2

augmentor and a few updates

11e163c

Signed-off-by: fayejf <fayejf07@gmail.com>

github-advanced-security bot found potential problems Jan 6, 2023

View reviewed changes

tools/speech_evaluator/asr_evaluator.py Fixed Show fixed Hide fixed

nemo/collections/asr/parts/utils/eval_utils.py Fixed Show fixed Hide fixed

nemo/collections/asr/parts/utils/eval_utils.py Fixed Show fixed Hide fixed

fayejf and others added 3 commits January 5, 2023 16:41

Merge branch 'main' into asr_evaluator_engine

f40f6dc

Merge branch 'main' into asr_evaluator_engine

fdbfa20

address alerts and revert unnecessary changes

3e001ee

Signed-off-by: fayejf <fayejf07@gmail.com>

github-advanced-security bot found potential problems Jan 6, 2023

View reviewed changes

nemo/collections/asr/parts/utils/eval_utils.py Fixed Show fixed Hide fixed

fayejf added 4 commits January 5, 2023 22:44

Add readme

52122bf

Signed-off-by: fayejf <fayejf07@gmail.com>

rename

86092ca

Signed-off-by: fayejf <fayejf07@gmail.com>

typo fix

bf5cfe8

Signed-off-by: fayejf <fayejf07@gmail.com>

small fix

431877d

Signed-off-by: fayejf <fayejf07@gmail.com>

fayejf marked this pull request as ready for review January 6, 2023 07:02

fayejf requested a review from stevehuang52 January 6, 2023 07:02

fayejf and others added 2 commits January 6, 2023 14:21

Merge branch 'main' into asr_evaluator_engine

86fe931

add missing header

8af6afe

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52 requested changes Jan 9, 2023

View reviewed changes

fayejf and others added 4 commits January 9, 2023 11:40

Merge branch 'main' into asr_evaluator_engine

b50483b

rename augmentor_config to augmentor

9b1a52a

Signed-off-by: fayejf <fayejf07@gmail.com>

raname inference_mode to inference

810adcf

Signed-off-by: fayejf <fayejf07@gmail.com>

move utils.py

503fd04

Signed-off-by: fayejf <fayejf07@gmail.com>

fayejf and others added 7 commits January 9, 2023 12:18

update temp file

ae50af2

Signed-off-by: fayejf <fayejf07@gmail.com>

make wer cer clear

dcd842e

Signed-off-by: fayejf <fayejf07@gmail.com>

seed_everything

4f16213

Signed-off-by: fayejf <fayejf07@gmail.com>

fix missing rn augmentor_config in rnnt

d36309b

Signed-off-by: fayejf <fayejf07@gmail.com>

fix rnnt transcribe

d435217

Signed-off-by: fayejf <fayejf07@gmail.com>

add more docstring and style fix

1614acf

Signed-off-by: fayejf <fayejf07@gmail.com>

Merge branch 'main' into asr_evaluator_engine

814910a

github-advanced-security bot found potential problems Jan 10, 2023

View reviewed changes

tools/asr_evaluator/asr_evaluator.py Fixed Show fixed Hide fixed

tools/asr_evaluator/utils.py Fixed Show fixed Hide fixed

fayejf and others added 2 commits January 9, 2023 17:02

Merge branch 'main' into asr_evaluator_engine

6a0ad5a

address codeQL

50473e1

Signed-off-by: fayejf <fayejf07@gmail.com>

fayejf requested a review from stevehuang52 January 10, 2023 01:03

fayejf and others added 3 commits January 10, 2023 10:47

Merge branch 'main' into asr_evaluator_engine

00c7090

reflect comments

840c8de

Signed-off-by: fayejf <fayejf07@gmail.com>

update readme

32f48e8

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52 reviewed Jan 10, 2023

View reviewed changes

fayejf and others added 2 commits January 10, 2023 15:41

Merge branch 'main' into asr_evaluator_engine

047710b

clearer

70085ea

Signed-off-by: fayejf <fayejf07@gmail.com>

stevehuang52 approved these changes Jan 11, 2023

View reviewed changes

Merge branch 'main' into asr_evaluator_engine

7533651

fayejf merged commit c6b1ea5 into main Jan 11, 2023

fayejf deleted the asr_evaluator_engine branch January 11, 2023 16:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ASR evaluator #5728

ASR evaluator #5728

fayejf commented Jan 3, 2023 •

edited

stevehuang52 left a comment

stevehuang52 Jan 8, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 8, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023

stevehuang52 Jan 9, 2023

fayejf Jan 10, 2023 •

edited

stevehuang52 Jan 10, 2023

stevehuang52 Jan 10, 2023

stevehuang52 left a comment

		return cfg, total_res


		def target_metadata_wer(manifest: str, target: str, meta_cfg: DictConfig, eval_metric: str = "wer",) -> dict:

ASR evaluator #5728

ASR evaluator #5728

Conversation

fayejf commented Jan 3, 2023 • edited

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

stevehuang52 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fayejf Jan 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevehuang52 left a comment

Choose a reason for hiding this comment

fayejf commented Jan 3, 2023 •

edited

fayejf Jan 10, 2023 •

edited