Skip to content

Commit

Permalink
merging main to asr_normalize (#6676)
Browse files Browse the repository at this point in the history
* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by



* updated ASR docs with the fastconformer hybrid checkpoints



* added the fastconformer RNNT and CTC models



---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* Add scores for FastConformer models (#6557) (#6558)

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Fix fp16 (#6543) (#6544)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559)

Signed-off-by: fayejf <fayejf07@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>

* Fix notebook bad json (#6561)

Signed-off-by: smajumdar <titu1994@gmail.com>

* Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552)

* persistent workers



* fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Make KenLM with PC for AggregateTokenizer and merge it (#6081)

* do_lowercase, rm_punctuation

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* support beam_strategy = beam

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* black

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix config and^Cunctuation capitalization

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm math

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* update kenlm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add opengrm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* mv install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* punctuation_to_preserve

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Only tikenizer opion

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DEFAULT_TOKEN_OFFSET

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* aggregate_tokenizer

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install kenlm with more than 5gram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* ngram_bin_path kenlm_bin_path

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix greedy PC bug

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* move global params

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix description and perplexity

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix description

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* NEMO_PATH

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo:23.01

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* License

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* description

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* isinstance

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* refactor kenlm stdin

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add cmd arg

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* use new iter_files

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* EncDecHybridRNNTCTCModel

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* punctuation

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* train_kenlm args

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add docstrings

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ngram_merge docs

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* ngram_prune

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to ngram_merge

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add comments

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo_model_file

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install_opengrm_ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* install opengrm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to install_opengrm.sh

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm extra import

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* train_paths

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix ngram_bin_path

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DECODERS_PATH

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* farcompile

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm text processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* AggregateTokenizer.DummyTokenizer

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* comments

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TextProcessingConfig

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* doc

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* types

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo_model_file

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm assert

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* import kenlm_utils

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* return None

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Copyright

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* 2022

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* 2023

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

---------

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
Signed-off-by: Nikolay Karpov <karpnv@gmail.com>
Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* temp rtd fix (#6568) (#6569)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* [TTS] Add script for mapping speaker names to indices (#6509)

Signed-off-by: Ryan <rlangman@nvidia.com>

* whitespace (#6574)

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Update manifest.py for speedup (#6565) (#6573)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths



* Update manifest.py



---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* More streaming conformer export fixes (#6567) (#6578)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386)

* user selection should not break model max limit



* eval max seq length



---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Framework for PEFT via mixins  (#6391)

* init commit ptuning via mixin

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* gpt ptuning places virtual tokens on the left only

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* encoder input modified when pre_process is true

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* optimizer group and state dict updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adapter ptuning working for pp>1

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter ptuining config defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* training works

Signed-off-by: arendu <adithya.r@gmail.com>

* loading and saving adapter only params during training

Signed-off-by: arendu <adithya.r@gmail.com>

* added checks and comments

Signed-off-by: arendu <adithya.r@gmail.com>

* clean up

Signed-off-by: arendu <adithya.r@gmail.com>

* checks for grad is None before calling all_reduce

Signed-off-by: arendu <adithya.r@gmail.com>

* load adapter .nemo file working

Signed-off-by: arendu <adithya.r@gmail.com>

* resume training for adapters

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* peft tuning

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <adithya.r@gmail.com>

* file not needed

Signed-off-by: arendu <adithya.r@gmail.com>

* undo prompt learning dataset changes

Signed-off-by: arendu <adithya.r@gmail.com>

* undo updates to gpt prompt learning model

Signed-off-by: arendu <adithya.r@gmail.com>

* naming updates

Signed-off-by: arendu <adithya.r@gmail.com>

* decoding

Signed-off-by: arendu <adithya.r@gmail.com>

* predict_step in gpt_sft_model

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed inference from tuning config

Signed-off-by: arendu <adithya.r@gmail.com>

* no test in peft training

Signed-off-by: arendu <adithya.r@gmail.com>

* answer only loss and correct defaults for val_loss

Signed-off-by: arendu <adithya.r@gmail.com>

* hybrid adapters and ptuning

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval working..

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* prepending tokens for ptuning

Signed-off-by: arendu <adithya.r@gmail.com>

* cleaned up eval config

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* default prompt template

Signed-off-by: arendu <adithya.r@gmail.com>

* Lora added

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support synamic length with GPT SFT

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* make branch functional

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* defaults to max_pad_length=False in GPT SFT dataset

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter parallel_adapters to support Lora

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added early stopping by default

Signed-off-by: arendu <adithya.r@gmail.com>

* eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* better defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* docs

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* cache and reuse inputs (#6422) (#6452)

Co-authored-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Pass `.scale` instead of scaler object to core (#6551)

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Update megatron_gpt_model.py

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* scale changes for main

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Documentation for ASR-TTS models (#6594) (#6595)

* Add docs about hybrid ASR-TTS models



* Add docs about text-only datasets



* Add docs about ASR-TTS checkpoints



* Add docs about ASR-TTS configs and training



* Clean up



* ASR-TTS docs: add to api, fix imports



* Clean up



* Wrap optional import



* Revert general ASR import



---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>

* [TTS] Fix aligner nan loss in fp32 (#6435)

* Fix nan loss in fp32

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update SDP docs (#6485) (#6596)

* add info about SDP e.g. processor classes in docs



* add link to SDP docs in README



* address code review comments and add SDP overview diagram



* Fix spelling typo



---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>

* Bug/typo fixes (#6599)

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Manual garbage collection with an interval (#6469) (#6482)

* Manual garbage collection with an interval



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use trainer.global_step for tracking the interval of GC



---------

Signed-off-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Make tensor split contiguous (#6580) (#6593)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* [ASR] Fix for old models in change_attention_model (#6608)

* fixes

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* done already

Signed-off-by: sam1373 <samuelkriman@gmail.com>

---------

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* Update manifest.py to use os.path for get_full_path (#6598)

* Update manifest.py to use os.path for get_full_path

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py to get rid of pathlib

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Cherry pick commits in #6601 to main (#6611)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

* revert partial

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>

* Create dummy iters to satisy len checks (#6600) (#6603)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* add GPT eval mode fix for interleaved to main (#6610)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Not doing CastToFloat by default (#6524) (#6563)

* Not doing CastToFloat by default



* Added docustring



* Dummy commit



---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Turn autocast off when precision is fp32 (#6576)

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* correct auto-merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* correct auto-merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* add to GPT SFT

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update core commit hash in readme (#6622) (#6623)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* add hat image to docs (#6619) (#6621)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* Allow indices exchange via distributed (#6618) (#6624)

Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>
Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com>

* Offline and streaming inference support for hybrid model (#6570)

* streaming buffered for hybrid + ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* change default model_stride in eval.yaml

Signed-off-by: fayejf <fayejf07@gmail.com>

* add fc model_stride

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* check whether model and decoding match

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* streaming buffered for hybrid + rnnt

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix yaml

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect comment wip

Signed-off-by: fayejf <fayejf07@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* refactor and verified

Signed-off-by: fayejf <fayejf07@gmail.com>

* add get_full_path to buffered

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* add RNNTDecodingConfig

Signed-off-by: fayejf <fayejf07@gmail.com>

* model name & instruction of changing decoding

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Patch decoding for PC models (#6630) (#6631)

* Patch decoding logic for PC models



* Patch decoding logic for PC models



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Fix wer.py where 'errors' variable was not set (#6633) (#6634)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Restore GPT support for interleaved pipeline parallelism (#6528) (#6613)

* Restore logic for data-parallel communication with pipeline parallelism in GPT



* Support dynamic attention masks in GPT



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos



* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit



* Distinguish between list of data iterators and data iterator that is a list



* Create dummy iters to satisy len checks



* Kludge while waiting for Megatron-LM update



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting



---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* bugfix (#6636)

Signed-off-by: fayejf <fayejf07@gmail.com>

* Disable interctc tests (#6638)

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add megatron_core to requirements (#6639) (#6640)

* add megatron_core to requirements



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove from jenkins (#6642)

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove dup

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* sft model can use this script for eval (#6637)

* sft model can use this script for eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* please fix me

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Fix TTS audio preprocessing bugs (#6628)

Signed-off-by: Ryan <rlangman@nvidia.com>

* Move black parameters to pyproject.toml (#6647)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620)

* ASR-TTS: support hybrid RNNT-CTC models
* Do not warn on optional import
* Explain adding options to config
* Fix import guard docs
* Add docs for ConcatDataset
* Add explanation for sampling parameters
* Initial docs for the enhancer model
* Fix use_start_end_token parameter usage

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* fix conversion and eval (#6648)

* fix conversion and eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Confidence ensembles implementation (#6614)

* Working version to train conf model + save ensemble class

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Working version

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Remove copy of transcribe_speech.py

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Move models parameter to config

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add explicit parameters to transcribe

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Small cleanups

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add temperature and integration tests

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add more tests

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add pc removal config

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Cleanup

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Fix typo

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Address review comments

Signed-off-by: Igor Gitman <igitman@nvidia.com>

---------

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Patch memory used for NeMo Megatron models (#6615)

* Patch memory used for NeMo Megatron models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Cleanup the dtype of embeddings

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Try patch for Megatron O2

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <titu1994@gmail.com>

* Correct indent

Signed-off-by: smajumdar <titu1994@gmail.com>

* Correct utils import

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* handle artifacts when path is dir (#6658)

Signed-off-by: arendu <adithya.r@gmail.com>

* remove upgrading setuptools in reinstall.sh (#6659)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>

* merge lora weights into base model (#6597)

* merge lora weights into base model

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo fix

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor update

Signed-off-by: arendu <adithya.r@gmail.com>

* update copyright

Signed-off-by: arendu <adithya.r@gmail.com>

* eval needs to know the PEFT class

Signed-off-by: arendu <adithya.r@gmail.com>

* add target class in training script so that we can use it in eval

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update to work for tp1

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set restore model path

Signed-off-by: arendu <adithya.r@gmail.com>

* peft can be none

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated merge script so that eval works easily

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval with peft or sft model

Signed-off-by: arendu <adithya.r@gmail.com>

* keep sentences in jsonl format

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* convert sft using correct classpath

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated to force sft yaml to have the correct target

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated docs

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix conversion and eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* upgrade to 23.04 (#6660)

Signed-off-by: ericharper <complex451@gmail.com>

* Merge r1.18.0 bugfixes and doc updates to main (#6655)

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup

Signed-off-by: ericharper <complex451@gmail.com>

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Confidence ensembles: fix issues and add tuning functionality (#6657)

* Implement compute confidence to properly handle blanks

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Implement proper confidence for transducers

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Implement tuning logic

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add tests for confidence tuning

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Remove unused imports

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add types/docs

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add comment about the main conf compute loop

Signed-off-by: Igor Gitman <igitman@nvidia.com>

---------

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* [TTS] Implement new TextToSpeech dataset (#6575)

* [TTS] Implement new TextToSpeech dataset

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Add unit tests

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix defaulting of use_log_energy

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix TTS export test

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* Dialogue dataset  (#6654)

* chatbot interface

Signed-off-by: Yi Dong <yidong@nvidia.com>

* latest gradio

Signed-off-by: Yi Dong <yidong@nvidia.com>

* default greedy

Signed-off-by: Yi Dong <yidong@nvidia.com>

* better chatbot

Signed-off-by: Yi Dong <yidong@nvidia.com>

* handle preamble

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added chatbot training capablity

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added chatbot ui

Signed-off-by: Yi Dong <yidong@nvidia.com>

* remove debug code

Signed-off-by: Yi Dong <yidong@nvidia.com>

* default human

Signed-off-by: Yi Dong <yidong@nvidia.com>

* use special token for roles

Signed-off-by: Yi Dong <yidong@nvidia.com>

* special tokens

Signed-off-by: Yi Dong <yidong@nvidia.com>

* fix name

Signed-off-by: Yi Dong <yidong@nvidia.com>

* new chat dataset

Signed-off-by: Yi Dong <yidong@nvidia.com>

* fix the system token

Signed-off-by: Yi Dong <yidong@nvidia.com>

* upgrade gradio

Signed-off-by: Yi Dong <yidong@nvidia.com>

* save the chat history

Signed-off-by: Yi Dong <yidong@nvidia.com>

* update ui

Signed-off-by: root <you@example.com>

* update chat interface

Signed-off-by: Yi Dong <yidong@nvidia.com>

* handles canonical form

Signed-off-by: Yi Dong <yidong@nvidia.com>

* new sft chatbot

Signed-off-by: Yi Dong <yidong@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change format

Signed-off-by: Yi Dong <yidong@nvidia.com>

* check extra_id in the tokenizer

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added vocab property check

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added missing file

Signed-off-by: Yi Dong <yidong@nvidia.com>

---------

Signed-off-by: Yi Dong <yidong@nvidia.com>
Signed-off-by: root <you@example.com>
Co-authored-by: root <you@example.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Add support for RNNT/hybrid models to partial transcribe (#6609)

* Add support for RNNT/hybrid models to partial transcribe

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcribe_utils.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcribe_speech.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcribe_utils.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* eval_beamsearch_ngram.py with hybrid ctc (#6656)

* separate_punctuation = false

* ctc decoding strategy = model.decoding

* transcribe(files, logprobs=True) returns logprobs



---------

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix bucketing bug issue for picking new bucket (#6663)

Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: Nithin Rao Koluguri <nithinraok>

* minor fix for missing chat attr (#6671)

Signed-off-by: arendu <adithya.r@gmail.com>

* [TTS] Add callback for saving audio during FastPitch training (#6665)

* [TTS] Add callback for saving audio during FastPitch training

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Allow NGC model name for vocoder

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
Signed-off-by: Nikolay Karpov <karpnv@gmail.com>
Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Sangkug Lym <slym@nvidia.com>
Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>
Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Yi Dong <yidong@nvidia.com>
Signed-off-by: root <you@example.com>
Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Nikolay Karpov <karpnv@gmail.com>
Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ryan Langman <rlangman@nvidia.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Co-authored-by: Igor Gitman <igitman@nvidia.com>
Co-authored-by: Samuel Kriman <samuelkriman@gmail.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Adi Renduchintala <adithya.r@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com>
Co-authored-by: root <you@example.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
  • Loading branch information
1 parent ced79bc commit e57c051
Show file tree
Hide file tree
Showing 197 changed files with 7,956 additions and 1,039 deletions.
1 change: 0 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,4 @@ repos:
hooks:
- id: black
name: Format code
args: [--skip-string-normalization, --line-length=119]
additional_dependencies: ['click==8.0.2']
9 changes: 2 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:23.03-py3
ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:23.04-py3

# build an image that includes only the nemo dependencies, ensures that dependencies
# are included first for optimal caching, and useful for building a development
Expand Down Expand Up @@ -43,11 +43,6 @@ RUN apt-get update && \
rm -rf /var/lib/apt/lists/*

WORKDIR /workspace/
# Install Megatron-core
RUN git clone https://github.com/NVIDIA/Megatron-LM.git && \
cd Megatron-LM && \
git checkout 3db2063b1ff992a971ba18f7101eecc9c4e90f03 && \
pip install -e .

WORKDIR /tmp/
# TODO: Remove once this Apex commit (2/24/23) is included in PyTorch
Expand Down Expand Up @@ -94,7 +89,7 @@ COPY . .

# start building the final container
FROM nemo-deps as nemo
ARG NEMO_VERSION=1.18.0
ARG NEMO_VERSION=1.19.0

# Check that NEMO_VERSION is set. Build will fail without this. Expose NEMO and base container
# version information as runtime environment variable for introspection purposes
Expand Down
58 changes: 24 additions & 34 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
pipeline {
agent {
docker {
image 'pytorch_23.03:apex_57057e2fcf1c084c0fcc818f55c0ff6ea1b24ae2'
args '--device=/dev/nvidia0 --gpus all --user 0:128 -v /home/TestData:/home/TestData -v $HOME/.cache:/root/.cache --shm-size=8g'
image 'nvcr.io/nvidia/pytorch:23.04-py3'
args '--device=/dev/nvidia0 --gpus all --user 0:128 -v /home/TestData:/home/TestData -v $HOME/.cache:/root/.cache --shm-size=8g --env TRANSFORMERS_OFFLINE=1'
}
}
options {
Expand Down Expand Up @@ -57,16 +57,6 @@ pipeline {
}
}

// TODO: remove when pip package is available
stage('Megatron Core installation') {
steps {
sh 'git clone https://github.com/NVIDIA/Megatron-LM.git && \
cd Megatron-LM && \
git checkout 3db2063b1ff992a971ba18f7101eecc9c4e90f03 && \
pip install -e .'
}
}

stage('PyTorch Lightning version') {
steps {
sh 'python -c "import pytorch_lightning; print(pytorch_lightning.__version__)"'
Expand Down Expand Up @@ -1014,7 +1004,7 @@ pipeline {
// TODO: pleasefixme @redoctopus
// stage('ByT5G2P training, evaluation and inference') {
// steps {
// sh 'TRANSFORMERS_OFFLINE=0 && cd examples/tts/g2p && \
// sh 'TRANSFORMERS_OFFLINE=1 && cd examples/tts/g2p && \
// TIME=`date +"%Y-%m-%d-%T"` && OUTPUT_DIR_T5=output_byt5_${TIME} && \
// python g2p_train_and_evaluate.py \
// train_manifest=/home/TestData/g2p/g2p.json \
Expand Down Expand Up @@ -1158,7 +1148,7 @@ pipeline {
parallel {
stage('Dialogue: Intent and slot classification using GPT') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
model.dataset.data_dir=/home/TestData/nlp/sgd_small \
model.language_model.lm_checkpoint=/home/TestData/nlp/gpt2/pytorch_model.bin\
Expand All @@ -1185,7 +1175,7 @@ pipeline {
}
stage('Intent and slot classification using SGDQA') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
model.dataset.data_dir=/home/TestData/nlp/sgd_small \
model.dataset.dialogues_example_dir=sgd_gen_bert_outputs \
Expand All @@ -1208,7 +1198,7 @@ pipeline {
}
stage('Intent and slot classification using IntentSlotClassificationModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
model.dataset.data_dir=/home/TestData/nlp/processed_assistant \
model.dataset.dialogues_example_dir=sgd_gen_bert_intent_classification_outputs \
Expand All @@ -1230,7 +1220,7 @@ pipeline {
}
stage('Intent classification using ZeroShotIntentModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/drive_thru_revised \
Expand All @@ -1255,7 +1245,7 @@ pipeline {
}
stage('Design Intent classification using ZeroShotIntentModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/design_dataset \
Expand All @@ -1281,7 +1271,7 @@ pipeline {
}
stage('Design Intent classification using ZeroShotIntentModel BART Classifier') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/design_dataset \
Expand All @@ -1300,7 +1290,7 @@ pipeline {
}
stage('Design Intent classification using DialogueNearestNeighbourModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/design_dataset \
Expand Down Expand Up @@ -1329,7 +1319,7 @@ pipeline {
parallel {
stage('Dialogue: Answer Extender using DialogueS2SGenerationModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/ms-marco-qa \
Expand All @@ -1354,7 +1344,7 @@ pipeline {
}
stage('Dialogue: SGD Based Answer Extender using DialogueS2SGenerationModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/sgd_small \
Expand Down Expand Up @@ -1395,7 +1385,7 @@ pipeline {
// parallel {
// stage('Dialogue: Answer Extender using DialogueGPTGenerationModel') {
// steps {
// sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
// sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
// python dialogue.py \
// do_training=False \
// model.dataset.data_dir=/home/TestData/nlp/ms-marco-qa \
Expand Down Expand Up @@ -1425,7 +1415,7 @@ pipeline {
parallel {
stage('Dialogue: Answer Extender using DialogueGPTGenerationModel') {
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/dialogue && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/dialogue && \
python dialogue.py \
do_training=False \
model.dataset.data_dir=/home/TestData/nlp/ms-marco-qa \
Expand Down Expand Up @@ -1549,7 +1539,7 @@ pipeline {
stage('BERT SQUAD 1.1') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v1.1/train-v1.1.json \
model.dataset.use_cache=false \
Expand All @@ -1574,7 +1564,7 @@ pipeline {
stage('BERT SQUAD 2.0') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v2.0/train-v2.0.json \
model.dataset.use_cache=false \
Expand Down Expand Up @@ -1608,7 +1598,7 @@ pipeline {
stage('BART SQUAD 1.1') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v1.1/train-v1.1.json \
model.dataset.use_cache=false \
Expand All @@ -1634,7 +1624,7 @@ pipeline {
stage('BART SQUAD 2.0') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v2.0/train-v2.0.json \
model.dataset.use_cache=false \
Expand Down Expand Up @@ -1669,7 +1659,7 @@ pipeline {
stage('GPT2 SQUAD 1.1') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v1.1/train-v1.1.json \
model.dataset.use_cache=false \
Expand All @@ -1695,7 +1685,7 @@ pipeline {
stage('GPT2 SQUAD 2.0') {
// Cannot do fast_dev_run because squad needs whole dev dataset
steps {
sh 'TRANSFORMERS_OFFLINE=0 && cd examples/nlp/question_answering && \
sh 'TRANSFORMERS_OFFLINE=1 && cd examples/nlp/question_answering && \
python question_answering.py \
model.train_ds.file=/home/TestData/nlp/squad_mini/v2.0/train-v2.0.json \
model.dataset.use_cache=false \
Expand Down Expand Up @@ -3809,8 +3799,8 @@ assert_frame_equal(training_curve, gt_curve, rtol=1e-3, atol=1e-3)"'''
stage('L2: Megatron T5 with KERPLE Pretraining and Resume Training TP=2') {
when {
anyOf {
branch 'r1.18.0'
changeRequest target: 'r1.18.0'
branch 'main'
changeRequest target: 'main'
}
}
failFast true
Expand Down Expand Up @@ -4016,7 +4006,7 @@ assert_frame_equal(training_curve, gt_curve, rtol=1e-3, atol=1e-3)"'''
sh "rm -rf examples/nlp/language_modeling/t5_index_mappings"
}
}

stage('L2: Megatron T5 Prompt Learning TP1 PP1') {
when {
anyOf {
Expand Down Expand Up @@ -4101,7 +4091,7 @@ assert_frame_equal(training_curve, gt_curve, rtol=1e-3, atol=1e-3)"'''
}
}
}

// TODO: add when https://github.com/NVIDIA/apex/pull/1596 is merged
// stage('L2: Megatron T5 Prompt Learning TP1 PP2') {
// when {
Expand Down
18 changes: 5 additions & 13 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ Key Features
* `Text Processing (text normalization and inverse text normalization) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/text_normalization/intro.html>`_
* `CTC-Segmentation tool <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tools/ctc_segmentation.html>`_
* `Speech Data Explorer <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tools/speech_data_explorer.html>`_: a dash-based tool for interactive exploration of ASR/TTS datasets
* `Speech Data Processor <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tools/speech_data_processor.html>`_


Built for speed, NeMo can utilize NVIDIA's Tensor Cores and scale out training to multiple GPUs and multiple nodes.
Expand Down Expand Up @@ -235,8 +236,8 @@ Note that RNNT requires numba to be installed from conda.
NeMo Megatron
~~~~~~~~~~~~~
NeMo Megatron training requires NVIDIA Apex and Megatron-core to be installed.
Install them manually if not using the NVIDIA PyTorch container.
NeMo Megatron training requires NVIDIA Apex to be installed.
Install it manually if not using the NVIDIA PyTorch container.

To install Apex, run

Expand All @@ -247,15 +248,6 @@ To install Apex, run
git checkout 57057e2fcf1c084c0fcc818f55c0ff6ea1b24ae2
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
To install Megatron-core, run

.. code-block:: bash
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout 3db2063b1ff992a971ba18f7101eecc9c4e90f03
pip install -e .
It is highly recommended to use the NVIDIA PyTorch or NeMo container if having issues installing Apex or any other dependencies.

While installing Apex, it may raise an error if the CUDA version on your system does not match the CUDA version torch was compiled with.
Expand Down Expand Up @@ -309,13 +301,13 @@ To build a nemo container with Dockerfile from a branch, please run
DOCKER_BUILDKIT=1 docker build -f Dockerfile -t nemo:latest .
If you chose to work with main branch, we recommend using NVIDIA's PyTorch container version 23.03-py3 and then installing from GitHub.
If you chose to work with main branch, we recommend using NVIDIA's PyTorch container version 23.04-py3 and then installing from GitHub.

.. code-block:: bash
docker run --gpus all -it --rm -v <nemo_github_folder>:/NeMo --shm-size=8g \
-p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit \
stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:23.03-py3
stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:23.04-py3
Examples
--------
Expand Down
18 changes: 18 additions & 0 deletions docs/source/asr/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ Model Classes
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact


.. autoclass:: nemo.collections.asr.models.hybrid_asr_tts_models.ASRWithTTSModel
:show-inheritance:
:members: from_asr_config, from_pretrained_models, save_asr_model_to, setup_training_data


Modules
-------

Expand Down Expand Up @@ -131,6 +136,19 @@ Character Encoding Datasets
:show-inheritance:
:members:


Text-to-Text Datasets for Hybrid ASR-TTS models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextDataset
:show-inheritance:
:members:

.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextIterableDataset
:show-inheritance:
:members:


Subword Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down

0 comments on commit e57c051

Please sign in to comment.