main to ssl synthesis (#9)

* Megatron BART BOS / EOS bug fix (#4495)

* 1. Debugging.

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* 1. BART dataset fixes missing <EOS> for deocder output.

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* 1. Debugging.

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* 1. Debugging.

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* 1. Removed extra padding from BARTDataset.

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* GPT Prompt Learning Improvements (#4496)

* Updated pipeline parallel code to speed up training

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Load global batch size not local mini batch size

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Python reformatting

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Megatron perceiver with tensor parallelism only (#4318)

* Temp

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add megatron dataset

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update config and fix global batch fetcher

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add dataset class

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update yaml

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix duplicate yaml key

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Translate method and preprocess script for raw text

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove pdb

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix arg name

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix other arg

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Change sampler back

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Move back to global batch fetcher to use distributed sampler

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add text memmap data

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update monitor

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fixes for PP

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove unused import

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Truncate examples in text memmap

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* NMT training batch interpolation key

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* tarred data fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Change dataset type check

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix sampler

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Pass dataset cfg to determine type

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Log global step on validation step as well

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix NMT model saving with artifacts

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Initialize DDP in decode if not initialized. Needed for inference only mode

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Megatron NMT inference script

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Inference config file

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* hardcode max delta temporarily

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* detokenizer if processor is not none

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Sampler config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Compat with configs without sampler arg

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Comment for validation dataset type

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix tokenizer building

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* CI test for megatron nmt

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix tokenizer in restore

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* O2 restore from fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove print

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Change tokenizer model name in config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Logging

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Set seed for distributed sampler

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Cluster debugging messages

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix max generation delta

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* No LM Init

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Use nlp save restore connector

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove useless infer args

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Typo

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* UTF8 safe print of translation result

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add save restore connector back with comment

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Refactor

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add missing args

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Empty to restart

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Check for test ds

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* set fusion to false

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Initial perceiver encoder

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Perceiver with PP=1

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove init cross attn

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* CI test and remove init cross attn arg

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove init cross attn layers from file

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Clean up

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* Set headscale false (#4364)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add wandb as dependency (#4365)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Raise trainer error (#4356)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Set headscale false (#4364) (#4366)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Finetuning changes for BART (#4003)

* Temp

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Checkpoint converter to nemo for bart

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (#4357)

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix logging warning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Refactor bias act fusion

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update NMT config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix electronic bug, new time ITN rule (#4355)

* fix electronic bug

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* add new itn time rule

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* revert domain changes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* remove repetition

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Update ci tests

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Correct support for dataclasses in default module dim (#4372)

* Correct support for dataclasses in default module dim

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix path for save of results

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* fix pad id bug (#4377)

Signed-off-by: Yi Dong <yidong@nvidia.com>

* Question answering bug fix (#4381)

* refactor dialogue state tracking for modelling/dataset interoperability

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style changes

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style raised by lgtm

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style formatting

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update template to include description of intent

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* changes based on requests in review

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add compatibility with assistant dataset

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove dialogue_state_tracking

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update huggingface utils for dialogue

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/__init__.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add docstrings for assistant data processsor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins for SGDGEN local checkpoint

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* use local vocab file for Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* patch for Jenkins CI using local file

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add slot filling prediction and metrics

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused code

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* refactor metrics code out of Dialogue GPT Model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate backward compatible support for IntentSlotClassificationModel (bert model)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* save prediction file for IntentSlotClassification

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue gpt model training for megatron gpt

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove batch generate for HF GPT2, which causes lower performance

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add few shot capability to dialogue gpt model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile and remove unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update code description and clarity

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address PR comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate compatibility with ZeroShotIntentModel

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* rename folder to dialogue due to increased scope and further refactor for clarity

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* added dialogue GPT for sequence generation task (e.g. answer extender)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add CI test for DialogueGPTGenerationModel

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate DialogueS2SGenerationModel for generation task (e.g. answer extender)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* modify huggingface utils to support HF t5/BART models

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix bleu metric style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* debug bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* debug bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update 2 based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update 3 based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate sgd generation based on user user utterance and system slot-values to generate system utterance

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add validation model saving capabilities

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* cleaned up code for SGD Based Answer extender

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Dialogue Generation CI

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix Jenkins CI issue"

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add support for design dataset

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unnecessary imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support megatron for dialogue_s2s_generation_model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* reduce loaded samples in MSMarcoDataProcessor to 64 when cfg.model.dataset.debug_mode=True

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update CI

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update checkpoint and predictions filename to include epoch number

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate HF BART MNLI into zero shot intent model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate Dialogue Nearest Neighbour Model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* refactor Dialogue SGD Data Processor to make interface for models cleaner

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Dialogue S2S Generation model for DialogueSGDDataProcessor interface

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support sgd and drive thru datasets by zero shot model and nearest neighbour model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add prediction saving code to nearest neighbour and zero shot intent models

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo in sgd data processor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate Dialogue Mellon QA Data Processor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update mellon qa

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue.py to remove outdated info

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue_config.yaml

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue_config.yaml

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add dialogue docs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address review comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix for cfg

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* make dependency on apex optional

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* change NLPDDPluggin calling logic to make it possible to run without apex

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add first draft of tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* reduce ms marco size by removing lines without wellFormedAnswers

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address pr comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update colab tutorial link in dialogue docs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* include unit test and some refactor to facilitate unit test

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address pr issues

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove typos in dialogue tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support larger files for question answering

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unnecessary artifacts to reduce memory use

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* put 0 tensor to device

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update link within dialogue tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* restore previously delete files

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error handling when loss = nan

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update nan handling

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update spanning loss func

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update spanning loss

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix type error raised in qa_dataset.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add error checking message

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert back to float32

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert back to float32

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update exp logging

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update loading of large file from pickle to json

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update loading of large file from pickle to json

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* limit number of negative samples

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert post processing

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert post processing

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused methods and style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add more documentation

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* changes base on PR review

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* set wandb logger falseby default

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

* style fix

* correct typo

* style fix

* style fix

Co-authored-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Fix ASR Typos in tutorials (#4384)

* Fix typos

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Quick wav2vec fix. In-place operation adding convolutional positions to encoder was overwriting leaf history. Wasn't caught on previous torch versions. (#4383)

Signed-off-by: tbartley94 <tbartley@nvidia.com>

Co-authored-by: tbartley94 <tbartley@nvidia.com>
(cherry picked from commit 0322b158f26a0b690edca7a84714e33752283923)

Co-authored-by: Travis Bartley <Travismbartley@gmail.com>

* Add Docs for NeMo Adapters (#4369)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update NeMo docs (#4397)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Co-authored-by: Eric Harper <complex451@gmail.com>

* Punctuation and capitalization tests race condition (#4399)

* Add draft of race condition fixes

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* Minor improvements

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* More race condition fixes

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* Improve error message

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* Improve error message

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* Improve error message

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

* bias act fusion changes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix geglu without fusion

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Reset files to main

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Remove hidden blocks

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: ericharper <complex451@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Travis Bartley <Travismbartley@gmail.com>
Co-authored-by: PeganovAnton <peganoff2@mail.ru>

* NMESC speaker counting algorithm update (#4500)

* initial commit

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Default maj_vote = False, max_rp=0.25

Signed-off-by: Taejin Park <tango4j@gmail.com>

* doc strings and style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Docstring minor edit

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Default False in the functions

Signed-off-by: Taejin Park <tango4j@gmail.com>

* fixed repeated variable

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Default as maj_vote=False

Signed-off-by: Taejin Park <tango4j@gmail.com>

* removed redundant part in wrtie_rttm func

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Removed unused function

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Updated and tested silence and very short samples

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Style fix and removing unnecessary parts

Signed-off-by: Taejin Park <tango4j@gmail.com>

* unused variables are removed

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Fixed commented torch.jit.script

Signed-off-by: Taejin Park <tango4j@gmail.com>

* majority voting update

Signed-off-by: Taejin Park <tango4j@gmail.com>

* cancelling the update on speaker_utils and clus_diarizer

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* bug fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Added fp32 converting for torch.mm

Signed-off-by: Taejin Park <tango4j@gmail.com>

Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>

* Fix dataset parameter typo on tacotron2 example yaml (#4471)

Signed-off-by: saarus72 <saarus72@gmail.com>

Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Noam lr sched: do not force min_lr after max_steps (#4472)

Signed-off-by: Adrian Lancucki <alancucki@users.noreply.github.com>

Co-authored-by: Adrian Lancucki <alancucki@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Refactor for punctuation model (#4367)

* Dataloader, collector, loss and metric for multiscale diarization decoder  (#4187)

* First commit

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Checked funtionality and imports

Signed-off-by: Taejin Park <tango4j@gmail.com>

* fixed import issues

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Removed the changed made by mistake

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Fixed LGTM errors 001

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Fixed LGTM and style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Changed docstrings

Signed-off-by: Taejin Park <tango4j@gmail.com>

* LGTM again

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Removed unnecessary torch setting lines

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Style fix and isort

Signed-off-by: Taejin Park <tango4j@gmail.com>

* jbalam-nv comments reflected

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Reflected comments and created _diar_label.py

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Typo fix and style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Fixed target_spks[0] index error

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* LGTM unused import IterDataset

Signed-off-by: Taejin Park <tango4j@gmail.com>

* revert collection doc year

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Code format error in collections.py

Signed-off-by: Taejin Park <tango4j@gmail.com>

* fix collections space format error

Signed-off-by: Taejin Park <tango4j@gmail.com>

* merged main correctly

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Reflected all comments and tested

Signed-off-by: Taejin Park <tango4j@gmail.com>

* style fix and LGTM

Signed-off-by: Taejin Park <tango4j@gmail.com>

* rttm_filepath to rttm_file and removed self included funcs, tested

Signed-off-by: Taejin Park <tango4j@gmail.com>

Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* removed references to data_dir

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* added missing parameters to data preparation script

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* removed unnecessary file extension check

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Add ASR CTC Decoding module (#4342)

* Initial commit

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Full support for decoding strategy

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Temp

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix labels of y_sequence

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Set support for sentencepiece subword merging

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix char and word based token merge alignment

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Revert incorrect change

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update docstring

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Improve compatibility with greedy tokens and log probs

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update scripts to use decoding strategy

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add tests and docs

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add tests and docs

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix speaker decoder timestamps

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix speaker decoder timestamps

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix decoding of ctc models

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Address reviewer comments

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Address reviewer comments

Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Option to disable mp in VAD via num_workers=1 (#4317)

* Option to disable mp in VAD via num_workers=1

In certain environments python multiprocessing can deadlock. This adds a convenient version to disable by setting num_workers to 1.

Signed-off-by: Georg Kucsko <gkucsko@gmail.com>

* add none handling

Signed-off-by: Georg Kucsko <gkucsko@gmail.com>

* additional none handling

Signed-off-by: Georg Kucsko <gkucsko@gmail.com>

Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* remove redundant bias expand (#4382)

* remove redundant bias expand

Signed-off-by: Xiaowei Ren <xren@nvidia.com>

* delete redundant code

Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* fixed style

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Add option for specifying wandb save_dir from config (#4379)

* give option to user to specify wandb save dir via config

Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>

* create save_dir directory for wandb logger if not exists

Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>

* update save_dir get method with a default value

Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Quick wav2vec fix. In-place operation adding convolutional positions to encoder was overwriting leaf history. Wasn't caught on previous torch versions. (#4383)

Signed-off-by: tbartley94 <tbartley@nvidia.com>

Co-authored-by: tbartley94 <tbartley@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* [Bugfix][TTS] wrong order of returned tuple for general_collate_fn. (#4388)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Merge r1.10.0 main (#4398)

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* Set headscale false (#4364)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add wandb as dependency (#4365)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Raise trainer error (#4356)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Set headscale false (#4364) (#4366)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Finetuning changes for BART (#4003)

* Temp

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Checkpoint converter to nemo for bart

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (#4357)

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix logging warning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>

* Fix electronic bug, new time ITN rule (#4355)

* fix electronic bug

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* add new itn time rule

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* revert domain changes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* remove repetition

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Correct support for dataclasses in default module dim (#4372)

* Correct support for dataclasses in default module dim

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Fix path for save of results

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* fix pad id bug (#4377)

Signed-off-by: Yi Dong <yidong@nvidia.com>

* Question answering bug fix (#4381)

* refactor dialogue state tracking for modelling/dataset interoperability

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style changes

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style raised by lgtm

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style formatting

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update template to include description of intent

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* changes based on requests in review

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add compatibility with assistant dataset

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove dialogue_state_tracking

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update huggingface utils for dialogue

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/__init__.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add docstrings for assistant data processsor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins for SGDGEN local checkpoint

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* use local vocab file for Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* patch for Jenkins CI using local file

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add slot filling prediction and metrics

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused code

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* refactor metrics code out of Dialogue GPT Model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate backward compatible support for IntentSlotClassificationModel (bert model)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* save prediction file for IntentSlotClassification

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue gpt model training for megatron gpt

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove batch generate for HF GPT2, which causes lower performance

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add few shot capability to dialogue gpt model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile and remove unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update code description and clarity

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address PR comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate compatibility with ZeroShotIntentModel

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* rename folder to dialogue due to increased scope and further refactor for clarity

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* added dialogue GPT for sequence generation task (e.g. answer extender)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add CI test for DialogueGPTGenerationModel

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate DialogueS2SGenerationModel for generation task (e.g. answer extender)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* modify huggingface utils to support HF t5/BART models

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix bleu metric style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* debug bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* debug bleu metric

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update 2 based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update 3 based on PR #3893

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate sgd generation based on user user utterance and system slot-values to generate system utterance

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add validation model saving capabilities

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* cleaned up code for SGD Based Answer extender

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Dialogue Generation CI

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix Jenkins CI issue"

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add support for design dataset

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unnecessary imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support megatron for dialogue_s2s_generation_model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* reduce loaded samples in MSMarcoDataProcessor to 64 when cfg.model.dataset.debug_mode=True

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update CI

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update checkpoint and predictions filename to include epoch number

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate HF BART MNLI into zero shot intent model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate Dialogue Nearest Neighbour Model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* refactor Dialogue SGD Data Processor to make interface for models cleaner

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Dialogue S2S Generation model for DialogueSGDDataProcessor interface

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support sgd and drive thru datasets by zero shot model and nearest neighbour model

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add prediction saving code to nearest neighbour and zero shot intent models

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo in sgd data processor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* integrate Dialogue Mellon QA Data Processor

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update mellon qa

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue.py to remove outdated info

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue_config.yaml

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update dialogue_config.yaml

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add dialogue docs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address review comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix for cfg

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* make dependency on apex optional

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* change NLPDDPluggin calling logic to make it possible to run without apex

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add first draft of tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* reduce ms marco size by removing lines without wellFormedAnswers

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address pr comments

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update colab tutorial link in dialogue docs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* include unit test and some refactor to facilitate unit test

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* address pr issues

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove typos in dialogue tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* support larger files for question answering

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unnecessary artifacts to reduce memory use

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* put 0 tensor to device

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update link within dialogue tutorial

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* restore previously delete files

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error handling when loss = nan

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update nan handling

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update spanning loss func

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update spanning loss

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix type error raised in qa_dataset.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add error checking message

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert back to float32

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert back to float32

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update exp logging

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update error msgs

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update loading of large file from pickle to json

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update loading of large file from pickle to json

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* limit number of negative samples

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert post processing

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* revert post processing

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused methods and style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add more documentation

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove unused imports

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* changes base on PR review

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* set wandb logger falseby default

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

* style fix

* correct typo

* style fix

* style fix

Co-authored-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Fix ASR Typos in tutorials (#4384)

* Fix typos

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Quick wav2vec fix. In-place operation adding convolutional positions to encoder was overwriting leaf history. Wasn't caught on previous torch versions. (#4383)

Signed-off-by: tbartley94 <tbartley@nvidia.com>

Co-authored-by: tbartley94 <tbartley@nvidia.com>
(cherry picked from commit 0322b158f26a0b690edca7a84714e33752283923)

Co-authored-by: Travis Bartley <Travismbartley@gmail.com>

* Add Docs for NeMo Adapters (#4369)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update NeMo docs (#4397)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Co-authored-by: Eric Harper <complex451@gmail.com>

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* remove Copy of

Signed-off-by: ericharper <complex451@gmail.com>

Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Travis Bartley <Travismbartley@gmail.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* [bugfix][TTS] pitch, voiced_mask, prob_voiced have the same values. (#4392)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Fixing import error in some cases (#4401)

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Fixing bugs in calling method ctc_decoder_predictions_tensor. (#4414)

* updated ctc decoding calls.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed the ones for timestamp_utils.py

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed the ones for timestamp_utils.py

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed the ones for timestamp_utils.py

Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* Update with new conformer checkpoints. (#4417)

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* [TTS] add static method decorator. (#4443)

* [TTS] add static method decorator.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* remove protect prefix

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* fixed style error

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

Co-authored-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Georg Kucsko <gkucsko@users.noreply.github.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: Xiaowei Ren <103958965+xrennvidia@users.noreply.github.com>
Co-authored-by: Shantanu Acharya <shantanua@nvidia.com>
Co-authored-by: Travis Bartley <Travismbartley@gmail.com>
Co-authored-by: tbartley94 <tbartley@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* bug fix - sample rate was being ignored in vocoder dataset when not loading mel

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com>

* Add ITN pt (#4516)

* Add ITN pt

Signed-off-by: Guilherme Steinmann <guist@linse.ufsc.br>

* Fix style

Signed-off-by: Guilherme Steinmann <guist@linse.ufsc.br>

* Fix style

Signed-off-by: Guilherme Steinmann <guist@linse.ufsc.br>

* Update copyright year to 2022 on ITN pt rules and tests

Signed-off-by: Guilherme Steinmann <guist@linse.ufsc.br>

* Fixed WER initialization in ASR_with_Nemo notebook (#4523)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

Co-authored-by: Ante Jukić <ajukic@nvidia.com>

* Update cmudict (#4510)

phoneme IY1 -> IH1 in NVIDIA
Added phonemes for CUSTOMIZABLE

Update cmudict file revision and its reference.

Signed-off-by: Jason Roche <jroche@nvidia.com>

Co-authored-by: Jason Roche <jroche@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [Add] Support for Different LRs with Param Groups (#4508)

* add support for param groups

Signed-off-by: stevehuang52 <heh@nvidia.com>

* make config more general

Signed-off-by: stevehuang52 <heh@nvidia.com>

Co-authored-by: Eric Harper <complex451@gmail.com>

* Weighted bucketing (#4474)

* Add silence handling for speaker diarization pipeline (#4512)

* initial commit

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* fixed silence wav file issue causing clustering to evaluate on null embeddings

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* fixed zero duration issue

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* updated with comments

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* minor doc change

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* update log

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* Fix runtime check (#4501)

* Runtime check refinements

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Added fp32 casting for ASR nets export

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* style

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Used torch.float32 for clarity

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Fixing parameters passing

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Update finetune label models (#4504)

* initial_script

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* move old script

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* remove finetune func from label models

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* style clean

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* updated config

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* update tutorial

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* lgtm fixes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* updated based on comments

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* update doc

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* [ASR][Breaking Change] Update signature of Hypothesis alignments (#4511)

* Preserve logprobs when preserving alignments

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update tests for rnnt gredy and beam search

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update all dependents of alignments

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update docs

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Weighted bucketing (#4530)

* Additional sentencepiece args - Byte fallback, split digits, split_on_whitespace (#4525)

* Fix geglu without fusion

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add extra args

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Reset transformer

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix spm arg

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix help string

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add support for ASR Adapter Auxiliary Losses (#4480)

* Add support for access mixin registry of custom losses

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* add support for asr custom losses

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update for l2 loss

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add unittests

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add unittests

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add unittests

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update registration of tensors to reset after finishing step

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Remove comment

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Remove comment

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Update SSL models

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Add support for validation step properly registering tensors

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Move reset of registry outside

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* update (#4520)

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix duplex inference with grammars (#4517)

* fix duplex inference with grammars

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* add ci test for duplex, fix electronic last sym bug

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* test fix

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* update jenkins grammars

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* add pt to the docs

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* disable test

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins refactor

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* jenkins

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* test

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* test

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* test

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* test

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>

* Add Bucketing support to TarredAudioToClassificationLabelDataset (#4465)

* Add Bucketing support to TarredAudioToClassificationLabelDataset

Signed-off-by: Ewald Enzinger <ewald.enzinger@entn.at>

* Add MTEncDec Finetune support (#4540)

* add FT support

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* rm preproc

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* review changes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* add CI

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* newline fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* CI fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* clean up

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* post training cleanup

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* test

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* revert

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* CI test

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* revert CI changes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* original CI

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Add nsys profiling (#4539)

* add nsys profiling

Signed-off-by: ericharper <complex451@gmail.com>

* only access omegaconf in setup

Signed-off-by: ericharper <complex451@gmail.com>

* use robust get_rank function

Signed-off-by: ericharper <complex451@gmail.com>

* simplify

Signed-off-by: ericharper <complex451@gmail.com>

* Update megatron prompt learning interface to dialogue  (#4545)

* refactor dialogue state tracking for modelling/dataset interoperability

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style changes

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style raised by lgtm

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style formatting

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update template to include description of intent

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* changes based on requests in review

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add compatibility with assistant dataset

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkins

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* remove dialogue_state_tracking

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update huggingface utils for dialogue

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix style

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/__init__.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* update Jenkinsfile for SGDGEN

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* fix typo

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

* add docstrings for assist…

README.rst

@@ -45,7 +45,7 @@ Key Features
 
 * Speech processing
     * `Automatic Speech Recognition (ASR) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/intro.html>`_
-        * Supported models: Jasper, QuartzNet, CitriNet, Conformer-CTC, Conformer-Transducer, ContextNet, LSTM-Transducer (RNNT), LSTM-CTC, ...
+        * Supported models: Jasper, QuartzNet, CitriNet, Conformer-CTC, Conformer-Transducer, Squeezeformer-CTC, Squeezeformer-Transducer, ContextNet, LSTM-Transducer (RNNT), LSTM-CTC, ...
         * Supports CTC and Transducer/RNNT losses/decoders
         * Beam Search decoding
         * `Language Modelling for ASR <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html>`_: N-gram LM in fusion with Beam Search decoding, Neural Rescoring with Transformer

docs/source/asr/asr_all.bib

@@ -1045,4 +1045,15 @@ @misc{ssl_inter
   publisher = {arXiv},
   year = {2021},
   copyright = {arXiv.org perpetual, non-exclusive license}
-}
+}
+
+@misc{kim2022squeezeformer,
+  doi = {10.48550/ARXIV.2206.00888},
+  url = {https://arxiv.org/abs/2206.00888},
+  author = {Kim, Sehoon and Gholami, Amir and Shaw, Albert and Lee, Nicholas and Mangalam, Karttikeya and Malik, Jitendra and Mahoney, Michael W. and Keutzer, Kurt},
+  keywords = {Audio and Speech Processing (eess.AS), Computation and Language (cs.CL), Sound (cs.SD), FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Computer and information sciences, FOS: Computer and information sciences},
+  title = {Squeezeformer: An Efficient Transformer for Automatic Speech Recognition},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {arXiv.org perpetual, non-exclusive license}
+}

docs/source/asr/configs.rst

@@ -502,6 +502,17 @@ specify the tokenizer if you want to use sub-word encoding instead of character-
 The encoder section includes the details about the Conformer-CTC encoder architecture. You may find more information in the 
 config files and also :doc:`nemo.collections.asr.modules.ConformerEncoder<./api.html#nemo.collections.asr.modules.ConformerEncoder>`.
 
+Squeezeformer-CTC
+~~~~~~~~~~~~~~~~~
+
+The config files for Squeezeformer-CTC model contain character-based encoding and sub-word encoding at
+``<NeMo_git_root>/examples/asr/conf/squeezeformer/squeezeformer_ctc_char.yaml`` and ``<NeMo_git_root>/examples/asr/conf/squeezeformer/squeezeformer_ctc_bpe.yaml``
+respectively. Components of the configs of `Squeezeformer-CTC <./models.html#Squeezeformer-CTC>`__ are similar to Conformer config - `QuartzNet <./configs.html#Conformer-CTC>`__.
+
+The encoder section includes the details about the Squeezeformer-CTC encoder architecture. You may find more information in the
+config files and also :doc:`nemo.collections.asr.modules.SqueezeformerEncoder<./api.html#nemo.collections.asr.modules.SqueezeformerEncoder>`.
+
+
 ContextNet
 ~~~~~~~~~~
 

docs/source/asr/data/benchmark_rw.csv

@@ -0,0 +1,3 @@
+Model,Model Base Class,Model Card
+stt_rw_conformer_ctc_large,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_rw_conformer_ctc_large"
+stt_rw_conformer_transducer_large,EncDecRNNTBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_rw_conformer_transducer_large"

docs/source/asr/data/scores/rw/conformer_rw.csv

@@ -0,0 +1,3 @@
+Model Name,Language,MCV Test-Set v9.0 (rw)
+stt_rw_conformer_ctc_large,rw,18.22
+stt_rw_conformer_transducer_large,rw,16.19

docs/source/asr/datasets.rst

@@ -1,7 +1,7 @@
 Datasets
 ========
 
-NeMo has scripts to convert several common ASR datasets into the format expected by the ``nemo_asr`` collection. You can get started 
+NeMo has scripts to convert several common ASR datasets into the format expected by the ``nemo_asr`` collection. You can get started
 with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
 
 If the user has their own data and want to preprocess it to use with NeMo ASR models, refer to the `Preparing Custom ASR Data`_ section.
@@ -13,8 +13,8 @@ If the user already has a dataset that you want to convert to a tarred format, r
 LibriSpeech
 -----------
 
-Run the following scripts to download the LibriSpeech data and convert it into the format expected by `nemo_asr`. At least 250GB free 
-space is required. 
+Run the following scripts to download the LibriSpeech data and convert it into the format expected by `nemo_asr`. At least 250GB free
+space is required.
 
 .. code-block:: bash
 
@@ -37,18 +37,18 @@ Fisher English Training Speech
 
 Run these scripts to convert the Fisher English Training Speech data into a format expected by the ``nemo_asr`` collection.
 
-In brief, the following scripts convert the ``.sph`` files to ``.wav``, slices those files into smaller audio samples, matches the 
-smaller slices with their corresponding transcripts, and splits the resulting audio segments into train, validation, and test sets 
+In brief, the following scripts convert the ``.sph`` files to ``.wav``, slices those files into smaller audio samples, matches the
+smaller slices with their corresponding transcripts, and splits the resulting audio segments into train, validation, and test sets
 (with one manifest each).
 
 .. note::
   - 106 GB of space is required to run the ``.wav`` conversion
   - additional 105 GB is required for the slicing and matching
-  - ``sph2pipe`` is required in order to run the ``.wav`` conversion 
+  - ``sph2pipe`` is required in order to run the ``.wav`` conversion
 
 **Instructions**
 
-The following scripts assume that you already have the Fisher dataset from the Linguistic Data Consortium, with a directory structure 
+The following scripts assume that you already have the Fisher dataset from the Linguistic Data Consortium, with a directory structure
 that looks similar to the following:
 
 .. code-block:: bash
@@ -67,7 +67,7 @@ that looks similar to the following:
       ├── fe_03_p2_sph3
       └── ...
 
-The transcripts that will be used are located in the ``fe_03_p<1,2>_transcripts/data/trans`` directory. The audio files (``.sph``) 
+The transcripts that will be used are located in the ``fe_03_p<1,2>_transcripts/data/trans`` directory. The audio files (``.sph``)
 are located in the remaining directories in an ``audio`` subdirectory.
 
 #. Convert the audio files from ``.sph`` to ``.wav`` by running:
@@ -78,7 +78,7 @@ are located in the remaining directories in an ``audio`` subdirectory.
      python fisher_audio_to_wav.py \
        --data_root=<fisher_root> --dest_root=<conversion_target_dir>
 
-   This will place the unsliced ``.wav`` files in ``<conversion_target_dir>/LDC200[4,5]S13-Part[1,2]/audio-wav/``. It will take several 
+   This will place the unsliced ``.wav`` files in ``<conversion_target_dir>/LDC200[4,5]S13-Part[1,2]/audio-wav/``. It will take several
    minutes to run.
 
 #. Process the transcripts and slice the audio data.
@@ -90,7 +90,7 @@ are located in the remaining directories in an ``audio`` subdirectory.
        --dest_root=<processing_target_dir> \
        --remove_noises
 
-   This script splits the full dataset into train, validation, test sets, and places the audio slices in the corresponding folders 
+   This script splits the full dataset into train, validation, test sets, and places the audio slices in the corresponding folders
    in the destination directory. One manifest is written out per set, which includes each slice's transcript, duration, and path.
 
    This will likely take around 20 minutes to run. Once finished, delete the 10 minute long ``.wav`` files.
@@ -100,8 +100,8 @@ are located in the remaining directories in an ``audio`` subdirectory.
 
 Run the following script to convert the HUB5 data into a format expected by the ``nemo_asr`` collection.
 
-Similarly, to the Fisher dataset processing scripts, this script converts the ``.sph`` files to ``.wav``, slices the audio files and 
-transcripts into utterances, and combines them into segments of some minimum length (default is 10 seconds). The resulting segments 
+Similarly, to the Fisher dataset processing scripts, this script converts the ``.sph`` files to ``.wav``, slices the audio files and
+transcripts into utterances, and combines them into segments of some minimum length (default is 10 seconds). The resulting segments
 are all written out to an audio directory and the corresponding transcripts are written to a manifest JSON file.
 
 .. note::
@@ -123,7 +123,7 @@ You can optionally include ``--min_slice_duration=<num_seconds>`` if you would l
 AN4 Dataset
 -----------
 
-This is a small dataset recorded and distributed by Carnegie Mellon University. It consists of recordings of people spelling out 
+This is a small dataset recorded and distributed by Carnegie Mellon University. It consists of recordings of people spelling out
 addresses, names, etc. Information about this dataset can be found on the `official CMU site <http://www.speech.cs.cmu.edu/databases/an4/>`_.
 
 #. `Download and extract the dataset <http://www.speech.cs.cmu.edu/databases/an4/an4_sphere.tar.gz>`_ (which is labeled "NIST's Sphere audio (.sph) format (64M)".
@@ -153,14 +153,14 @@ After the script finishes, the ``data`` folder should contain a ``data_aishell``
 Aishell-2
 ---------
 
-To process the AIShell-2 dataset, in the command below, set the data folder of AIShell-2 using ``--audio_folder`` and where to push 
-these files using ``--dest_folder``. In order to generate files in the supported format of ``nemo_asr``, run: 
+To process the AIShell-2 dataset, in the command below, set the data folder of AIShell-2 using ``--audio_folder`` and where to push
+these files using ``--dest_folder``. In order to generate files in the supported format of ``nemo_asr``, run:
 
 .. code-block:: bash
 
     python process_aishell2_data.py --audio_folder=<data directory> --dest_folder=<destination directory>
 
-After the script finishes, the ``train.json``, ``dev.json``, ``test.json``, and ``vocab.txt`` files can be found in the ``dest_folder`` directory. 
+After the script finishes, the ``train.json``, ``dev.json``, ``test.json``, and ``vocab.txt`` files can be found in the ``dest_folder`` directory.
 
 Preparing Custom ASR Data
 -------------------------
@@ -171,7 +171,7 @@ The audio files can be of any format supported by `Pydub <https://github.com/jia
 WAV files as they are the default and have been most thoroughly tested.
 
 There should be one manifest file per dataset that will be passed in, therefore, if the user wants separate training and validation
-datasets, they should also have separate manifests. Otherwise, thay will be loading validation data with their training data and vice 
+datasets, they should also have separate manifests. Otherwise, thay will be loading validation data with their training data and vice
 versa.
 
 Each line of the manifest should be in the following format:
@@ -210,16 +210,22 @@ of filepaths, e.g. ``['/data/shard1.tar', '/data/shard2.tar']``, or in a single
 ``'/data/shard_{1..64}.tar'`` or ``'/data/shard__OP_1..64_CL_'`` (recommended, see note below).
 
 .. note::
-  For brace expansion, there may be cases where ``{x..y}`` syntax cannot be used due to shell interference. This occurs most commonly 
-  inside SLURM scripts. Therefore, we provide a few equivalent replacements. Supported opening braces (equivalent to ``{``) are ``(``, 
-  ``[``, ``<`` and the special tag ``_OP_``. Supported closing braces (equivalent to ``}``) are ``)``, ``]``, ``>`` and the special 
+  For brace expansion, there may be cases where ``{x..y}`` syntax cannot be used due to shell interference. This occurs most commonly
+  inside SLURM scripts. Therefore, we provide a few equivalent replacements. Supported opening braces (equivalent to ``{``) are ``(``,
+  ``[``, ``<`` and the special tag ``_OP_``. Supported closing braces (equivalent to ``}``) are ``)``, ``]``, ``>`` and the special
   tag ``_CL_``. For SLURM based tasks, we suggest the use of the special tags for ease of use.
 
-As with non-tarred datasets, the manifest file should be passed in ``manifest_filepath``. The dataloader assumes that the length 
+As with non-tarred datasets, the manifest file should be passed in ``manifest_filepath``. The dataloader assumes that the length
 of the manifest after filtering is the correct size of the dataset for reporting training progress.
 
-The ``tarred_shard_strategy`` field of the config file can be set if you have multiple shards and are running an experiment with 
+The ``tarred_shard_strategy`` field of the config file can be set if you have multiple shards and are running an experiment with
 multiple workers. It defaults to ``scatter``, which preallocates a set of shards per worker which do not change during runtime.
+Note that this strategy, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample
+the entire dataset. As an alternative the ``replicate`` strategy, will preallocate the entire set of shards to every worker and not
+change it during runtime. The benefit of this strategy is that it allows each worker to sample data points from the entire dataset
+independently of others. Note, though, that more than one worker may sample the same shard, and even sample the same data points!
+As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Note that
+for these reasons it is not advisable to use tarred datasets as validation and test datasets.
 
 For more information about the individual tarred datasets and the parameters available, including shuffling options,
 see the corresponding class APIs in the `Datasets <./api.html#Datasets>`__ section.
@@ -228,7 +234,7 @@ see the corresponding class APIs in the `Datasets <./api.html#Datasets>`__ secti
   If using multiple workers, the number of shards should be divisible by the world size to ensure an even
   split among workers. If it is not divisible, logging will give a warning but training will proceed, but likely hang at the last epoch.
   In addition, if using distributed processing, each shard must have the same number of entries after filtering is
-  applied such that each worker ends up with the same number of files. We currently do not check for this in any dataloader, but the user's 
+  applied such that each worker ends up with the same number of files. We currently do not check for this in any dataloader, but the user's
   program may hang if the shards are uneven.
 
 Conversion to Tarred Datasets
@@ -262,9 +268,9 @@ The files in the target directory should look similar to the following:
   ├── metadata.yaml
   └── tarred_audio_manifest.json
 
-Note that file structures are flattened such that all audio files are at the top level in each tarball. This ensures that 
-filenames are unique in the tarred dataset and the filepaths do not contain "-sub" and forward slashes in each ``audio_filepath`` are 
-simply converted to underscores. For example, a manifest entry for ``/data/directory1/file.wav`` would be ``_data_directory1_file.wav`` 
+Note that file structures are flattened such that all audio files are at the top level in each tarball. This ensures that
+filenames are unique in the tarred dataset and the filepaths do not contain "-sub" and forward slashes in each ``audio_filepath`` are
+simply converted to underscores. For example, a manifest entry for ``/data/directory1/file.wav`` would be ``_data_directory1_file.wav``
 in the tarred dataset manifest, and ``/data/directory2/file.wav`` would be converted to ``_data_directory2_file.wav``.
 
 Bucketing Datasets
@@ -325,9 +331,9 @@ Currently bucketing feature is just supported for tarred datasets.
 Upsampling Datasets
 ------------------
 
-Buckets may also be 'weighted' to allow multiple runs through a target dataset during each training epoch. This can be beneficial in cases when a dataset is composed of several component sets of unequal sizes and one desires to mitigate bias towards the larger sets through oversampling.   
+Buckets may also be 'weighted' to allow multiple runs through a target dataset during each training epoch. This can be beneficial in cases when a dataset is composed of several component sets of unequal sizes and one desires to mitigate bias towards the larger sets through oversampling.
 
-Weighting is managed with the `bucketing_weights` parameter. After passing your composite tarred datasets in the format described above for bucketing, pass a list of integers (one per bucket) to indicate how many times a manifest should be read during training. 
+Weighting is managed with the `bucketing_weights` parameter. After passing your composite tarred datasets in the format described above for bucketing, pass a list of integers (one per bucket) to indicate how many times a manifest should be read during training.
 
 For example, by passing `[2,1,1,3]` to the code below:
 
@@ -363,7 +369,7 @@ If using adaptive bucketing, note that the same batch size will be assigned to e
 	model.train_ds.bucketing_weights=[2,1,1,3]
 	model.train_ds.bucketing_batch_size=[4,4,4,2]
 
-All instances of data from `bucket4` will still be trained with a batch size of 2 while all others would have a batch size of 4. As with standard bucketing, this requires `batch_size`` to be set to 1. 
+All instances of data from `bucket4` will still be trained with a batch size of 2 while all others would have a batch size of 4. As with standard bucketing, this requires `batch_size`` to be set to 1.
 If `bucketing_batch_size` is not specified, all datasets will be passed with the same fixed batch size as specified by the `batch_size` parameter.
 
-It is recommended to set bucketing strategies to `fully_randomized` during multi-GPU training to prevent possible dataset bias during training. 
+It is recommended to set bucketing strategies to `fully_randomized` during multi-GPU training to prevent possible dataset bias during training.