Natural language prompting for Canary #8833

pzelasko · 2024-04-05T15:32:50Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

…ontrol the effective audio vs text size (requires latest lhotse) Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: w4-jinhyeonkim <131935801+w4-jinhyeonkim@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* yttm deprecation init commit Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * removed tests Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * bug fix Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * path fix Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * fixing path Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * updated tests to spm Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * updated Jenkinsfile Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * new model with spm in tests Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * yttm removed Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> * updated aayn config Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> --------- Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Added copy import to rnnt_decoding.py Signed-off-by: Isaac McFadyen <isaac@imcf.me> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Isaac McFadyen <isaac@imcf.me> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fixed cp HtoD sync Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Co-authored-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Fix triton import guards Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Update attention.py Signed-off-by: Michal Futrega <mfutrega@nvidia.com> --------- Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* enable layernorm output gathered Signed-off-by: Chen Cui <chcui@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

…pport (#8564) * POC using bucketing in transcribe_speech.py Signed-off-by: Piotr Żelasko <petezor@gmail.com> * extend to multi task aed Signed-off-by: Piotr Żelasko <petezor@gmail.com> * fixes for aed multi task text/lang field selectors Signed-off-by: Piotr Żelasko <petezor@gmail.com> * remove assert Signed-off-by: Piotr Żelasko <petezor@gmail.com> * fix Signed-off-by: Piotr Żelasko <petezor@gmail.com> * expose option for bucket buffer size Signed-off-by: Piotr Żelasko <petezor@gmail.com> * fixes, ctc support Signed-off-by: Piotr Żelasko <petezor@gmail.com> * support pre-sorting manifests in transcribe_speech.py Signed-off-by: Piotr Żelasko <petezor@gmail.com> * cleanup Signed-off-by: Piotr Żelasko <petezor@gmail.com> * reorder transcriptions back to original manifest order Signed-off-by: Piotr Żelasko <petezor@gmail.com> * remove bucketing entirely Signed-off-by: Piotr Żelasko <petezor@gmail.com> * code review changes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * code review changes--amend Signed-off-by: Piotr Żelasko <petezor@gmail.com> * refactor text_field/lang_field passing Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix reordering bug; disable presorting for multi task for now Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add support for presort + multi task model Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Code reviews Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix jenkins tests, add user-friendly error msg for canary Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

…atures Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

nemo/collections/asr/parts/mixins/transcription.py

@@ -17,7 +17,7 @@
 import tempfile
 from abc import ABC, abstractmethod
 from collections.abc import Iterable
-from dataclasses import dataclass
+from dataclasses import dataclass, is_dataclass


github-actions · 2024-05-31T01:47:14Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2024-06-07T01:48:40Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

pzelasko and others added 28 commits March 7, 2024 14:43

wip

596330d

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Partially working config groups

f54dc1c

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Working test with abasic group in the input config

dd064ab

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Working test with nested groups in input config

073f582

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Working test with specifying a YAML path for input_cfg

ad6df3f

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

a very rough example of text dataloading via lhotse

a929f0a

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Cleaner integration of multimodal audio/text loading that allows to c…

a7ab811

…ontrol the effective audio vs text size (requires latest lhotse) Signed-off-by: Piotr Żelasko <petezor@gmail.com>

remove obsolete test

05efa1b

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Fix an import in export_utils.py (#8571)

d48ecd9

Signed-off-by: w4-jinhyeonkim <131935801+w4-jinhyeonkim@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Fix bug in RNNT Joint WER calculation for fused batch (#8587)

bff40aa

Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

change default and add key to config files (#8594)

032a17f

Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Add config key for dropout position in LoRA adapter (#8583)

c8ffcc6

Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

fix ia3 mlp infused adapter (#8597)

29c1225

Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com>

fix tests

8f0c9de

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Bump min required lhotse version

06ab317

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Add some documentation about this config format and the multimodal fe…

fc0f589

…atures Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Merge branch 'main' into extended-nemo-data-config

3e9b6fa

Add caution about multiple shards

5e96d4a

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Merge branch 'main' into extended-nemo-data-config

fbd3efa

Address Tom's code review

c921e8b

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Add copyright header

41d246e

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Fix (hopefully) issue with forced ascii encoding in CI

d2cb979

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

github-actions bot added ASR common labels Apr 5, 2024

pzelasko and others added 5 commits April 5, 2024 11:35

Support resolving input_cfg path into config contents

4543cde

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Register natural prompt fn for canary

078e649

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Fixes

a5d574c

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Fixes for WER

86dab55

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Remove prompts from prediction and scoring

5daffc0

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

pzelasko force-pushed the canary-natural-prompt branch from 52cb05c to 5daffc0 Compare April 5, 2024 15:46

github-actions bot removed the common label Apr 5, 2024

Support stripping PAD and EOS from MultiTask AED beam search hypotheses

d68406e

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Base automatically changed from extended-nemo-data-config to main April 16, 2024 13:18

pzelasko and others added 9 commits April 23, 2024 14:49

Merge branch 'main' into canary-natural-prompt

9379a42

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

fixes

63ba238

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

b61f8e2

support agg tokenizer for natural prompts + add unit tests

e46462d

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

41de3a2

Support specifying prompt_lang for CanaryTokenizer + natural prompts

7caa888

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

41e36c4

Merge remote-tracking branch 'origin/main' into canary-natural-prompt

dc35cf3

Apply isort and black reformatting

3b121e8

Signed-off-by: pzelasko <pzelasko@users.noreply.github.com>

github-advanced-security bot found potential problems May 16, 2024

View reviewed changes

github-actions bot added the stale label May 31, 2024

github-actions bot closed this Jun 7, 2024

pzelasko mentioned this pull request Jul 31, 2024

Fix Canary not stripping prompt from reference + more test coverage #9987

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Natural language prompting for Canary #8833

Natural language prompting for Canary #8833

pzelasko commented Apr 5, 2024

github-actions bot commented May 31, 2024

github-actions bot commented Jun 7, 2024

Natural language prompting for Canary #8833

Natural language prompting for Canary #8833

Conversation

pzelasko commented Apr 5, 2024

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

github-actions bot commented May 31, 2024

github-actions bot commented Jun 7, 2024