Context-biasing by CTC-based Word Spotter (CTC-WS) #8223

andrusenkoau · 2024-01-23T13:51:57Z

What does this PR do ?

A new context-biasing method for CTC and Transducer (RNNT) models (only Hybrid Transducer-CTC in case of Transducer) by CTC-based Word Spotter (CTC-WS). The idea is to use CTC logprobs for fast decoding with context-biasing graph (tree). The context-biasing graph is built according to the context-biasing list of words/phrases and CTC transition topology. Obtained results (words with start/end timestamps) are merged with greedy CTC/Transducer word-level predictions.

CTC-WS allows context-biasing in greedy decoding mode which is much faster than approaches based on shallow fusion (beam search decoding with external lm or context-biasing graph).

A detailed description of the CTC-WS algorithm and the comparison results will be published later.

Collection: [ASR]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

python scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
            nemo_model_file=<path to the .nemo file of the model> \
            input_manifest=<path to the evaluation JSON manifest file \
            preds_output_folder=<optional folder to store the predictions> \
            decoder_type=<type of model decoder [ctc or rnnt]> \
            acoustic_batch_size=<batch size to calculate log probabilities> \
            apply_context_biasing=<True or False to apply context biasing> \
            context_file=<path to the context biasing file with key words/phrases> \
            beam_threshold=[<list of the beam thresholds, separated with commas>] \
            context_score=[<list of the context scores, separated with commas>] \
            ctc_ali_token_weight=[<list of the ctc alignment token weights, separated with commas>] \

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py

nemo/collections/asr/parts/context_biasing/context_biasing_utils.py

scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py

tests/collections/asr/test_asr_context_biasing.py

nemo/collections/asr/parts/context_biasing/context_biasing_utils.py

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py

andrusenkoau · 2024-01-23T15:17:49Z

jenkins

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

artbataev

Thanks for the work and tests!
Minor comment mostly related to the style

nemo/collections/asr/parts/context_biasing/context_biasing_utils.py

scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py

tests/collections/asr/test_asr_context_biasing.py

artbataev · 2024-02-02T15:01:41Z

tests/collections/asr/test_asr_context_biasing.py

+        assert len(ws_results) == 0
+
+        # with context biasing
+        ws_results = context_biasing.run_word_spotter(


Cool tests to cover primary functionality! Thanks!

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

nemo/collections/asr/parts/context_biasing/context_biasing_utils.py

nemo/collections/asr/parts/context_biasing/context_graph_ctc.py

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

for more information, see https://pre-commit.ci

andrusenkoau · 2024-02-12T07:20:04Z

jenkins

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

artbataev

LGTM. Thank you!

andrusenkoau · 2024-02-12T15:18:06Z

jenkins

andrusenkoau · 2024-02-12T16:11:29Z

jenkins

andrusenkoau · 2024-02-13T08:41:56Z

jenkins

* initial commit Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix blank_idx slow down Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add new non-blank pruning Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * descriptions fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * description fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add ctc only model Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * move scripts to nemo asr parts Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove scripts from scripts dir Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add first test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some tests Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix circular import Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix preds_output_folder Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * set loop_lables=True Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add .json to output manifest name Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix rnnt wer degradation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add round(score) for test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bow token Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add epsilon shift in alignment Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix transcribe modification Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix autocast Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Sasha Meister <ameister@nvidia.com>

* initial commit Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix blank_idx slow down Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add new non-blank pruning Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * descriptions fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * description fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add ctc only model Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * move scripts to nemo asr parts Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove scripts from scripts dir Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add first test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some tests Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix circular import Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix preds_output_folder Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * set loop_lables=True Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add .json to output manifest name Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix rnnt wer degradation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add round(score) for test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bow token Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add epsilon shift in alignment Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix transcribe modification Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix autocast Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com>

* initial commit Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix blank_idx slow down Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add new non-blank pruning Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * descriptions fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * description fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add ctc only model Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * move scripts to nemo asr parts Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * remove scripts from scripts dir Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add first test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add some tests Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * some fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix circular import Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix preds_output_folder Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * set loop_lables=True Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add .json to output manifest name Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix rnnt wer degradation Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * add round(score) for test Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bow token Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review fixes Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add epsilon shift in alignment Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fix Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * fix transcribe modification Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix autocast Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> --------- Signed-off-by: andrusenkoau <andrusenkoau@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

andrusenkoau added 19 commits January 16, 2024 01:55

initial commit

9da926f

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

some fixes

dff3257

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

fix blank_idx slow down

60b870b

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'main' of github.com:andrusenkoau/NeMo into ctcws

8dde9f6

add new non-blank pruning

105d242

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

some fixes

121322a

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

some fixes

59a3482

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

descriptions fix

e68354f

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

description fixes

b980d3f

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'main' of github.com:andrusenkoau/NeMo into ctcws

1e11fe7

add ctc only model

227710e

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

move scripts to nemo asr parts

d79a1b5

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

remove scripts from scripts dir

ec53a70

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

add first test

d639478

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

add some tests

46384c1

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

add test

bd2be97

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

some fixes

1068a63

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

fixes

033c68b

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'main' of github.com:andrusenkoau/NeMo into ctcws

92bce9e

github-actions bot added the ASR label Jan 23, 2024

andrusenkoau added the feature request/PR for a new feature label Jan 23, 2024

[pre-commit.ci] auto fixes from pre-commit.com hooks

9bbb1ad

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jan 23, 2024

View reviewed changes

andrusenkoau and others added 3 commits January 23, 2024 06:46

fixes

8e9de93

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

pull from aandrusenko

67fb46f

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

c6b9393

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jan 23, 2024

View reviewed changes

scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py Fixed Show fixed Hide fixed

andrusenkoau added 2 commits January 23, 2024 09:13

fix circular import

a17f4ae

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

fix conflict

ea37d83

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

artbataev reviewed Feb 2, 2024

View reviewed changes

andrusenkoau and others added 3 commits February 5, 2024 01:18

review fixes

e907580

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'main' of github.com:andrusenkoau/NeMo into ctcws

808f9e6

[pre-commit.ci] auto fixes from pre-commit.com hooks

81f5b39

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Feb 5, 2024

View reviewed changes

nemo/collections/asr/parts/context_biasing/context_biasing_utils.py Fixed Show fixed Hide fixed

nemo/collections/asr/parts/context_biasing/context_graph_ctc.py Fixed Show fixed Hide fixed

andrusenkoau and others added 13 commits February 5, 2024 04:52

review fixes

f97f179

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'ctcws' of github.com:andrusenkoau/NeMo into ctcws

ec70ffa

[pre-commit.ci] auto fixes from pre-commit.com hooks

52f9a1e

for more information, see https://pre-commit.ci

add epsilon shift in alignment

0557a08

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'ctcws' of github.com:andrusenkoau/NeMo into ctcws

b4ccf82

[pre-commit.ci] auto fixes from pre-commit.com hooks

a9421dc

for more information, see https://pre-commit.ci

minor fix

e6d517d

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

Merge branch 'ctcws' of github.com:andrusenkoau/NeMo into ctcws

50e59b2

Merge branch 'main' into ctcws

bd8306d

Merge branch 'main' of github.com:andrusenkoau/NeMo into ctcws

0c1a845

fix transcribe modification

f939ae9

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

ad283b9

for more information, see https://pre-commit.ci

Merge branch 'main' into ctcws

4542383

fix autocast

25e140e

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

artbataev approved these changes Feb 12, 2024

View reviewed changes

Merge branch 'main' into ctcws

bf722f4

andrusenkoau merged commit 0bfac69 into NVIDIA-NeMo:main Feb 13, 2024

sandorkonya mentioned this pull request Sep 21, 2025

ASR Context Biasing for EncDecHybridRNNTCTCModel (parakeet tdt 0.6b v3) #14772

Closed

Context-biasing by CTC-based Word Spotter (CTC-WS) #8223

Context-biasing by CTC-based Word Spotter (CTC-WS) #8223

Uh oh!

Conversation

andrusenkoau commented Jan 23, 2024

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrusenkoau commented Jan 23, 2024

Uh oh!

artbataev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

artbataev Feb 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrusenkoau commented Feb 12, 2024

Uh oh!

artbataev left a comment

Choose a reason for hiding this comment

Uh oh!

andrusenkoau commented Feb 12, 2024

Uh oh!

andrusenkoau commented Feb 12, 2024

Uh oh!

andrusenkoau commented Feb 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants