r1.10.0 MegaMolBART Compatibility #4603

michalivne · 2022-07-25T22:37:44Z

What does this PR do ?

Fixes RegExTokenizer missing attributes.
Fixes broken Megatron encoder-decoder LM model ability to receive directly encoder hiddens.

    MegatronLMEncoderDecoderModel.forward(
        self,
        encoder_input_ids,
        decoder_input_ids,
        encoder_attn_mask,
        decoder_attn_mask,
        token_type_ids=None,
        lm_labels=None,
        # NEW API
        enc_output=None,
        enc_output_attn_mask=None,
        # NEW API
        output_enc_hidden_only=False,
        enc_input=None,
    ):

Added MegatronLMEncoderDecoderModel.encode method.
Improved decoding speed.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

…t-regex-tokenizer-fixes

Signed-off-by: Micha Livne <mlivne@nvidia.com>

…ivne/NeMo into megamolbart-regex-tokenizer-fixes

Signed-off-by: Micha Livne <mlivne@nvidia.com>

…t-regex-tokenizer-fixes

Signed-off-by: Micha Livne <mlivne@nvidia.com>

lgtm-com · 2022-07-25T23:46:38Z

This pull request introduces 2 alerts when merging 8c50ac8 into c324499 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

lgtm-com · 2022-07-26T00:06:03Z

This pull request introduces 2 alerts when merging e630bf7 into c324499 - view on LGTM.com

new alerts:

2 for Unused import

lgtm-com · 2022-07-26T05:52:38Z

This pull request introduces 2 alerts when merging 932cae6 into faf8ad8 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

… r1.10.0-megamolbart

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

lgtm-com · 2022-07-26T22:34:27Z

This pull request introduces 2 alerts when merging d1db727 into aa0a98c - view on LGTM.com

new alerts:

2 for Unused local variable

lgtm-com · 2022-07-27T15:26:57Z

This pull request introduces 2 alerts when merging ca45e6d into cbf3f66 - view on LGTM.com

new alerts:

2 for Unused local variable

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

lgtm-com · 2022-07-28T21:35:00Z

This pull request introduces 2 alerts when merging 61acb6e into 72d78d8 - view on LGTM.com

new alerts:

2 for Unused local variable

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

lgtm-com · 2022-07-28T23:45:42Z

This pull request introduces 2 alerts when merging eee5810 into 72d78d8 - view on LGTM.com

new alerts:

2 for Unused local variable

* bug fix - sample rate was being ignored in vocoder dataset when not loading mel Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> * handled n segments for a different sampling rate than original sampling rate Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> * Added case for n_segments 0, warning for n_segments greater than file length Signed-off-by: Paarth Neekhara <paarth.n@gmail.com> * Fix metric setup for finetuning without a test set (NVIDIA#4585) * Fix metric setup for finetuning without a test set Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix log key Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove pdb Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Minor Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix skip train ds building while finetuning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com> * r1.10.0 MegaMolBART Compatibility (NVIDIA#4603) * 1. Added vocab_size property to RegExTokenizer. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed passing hiddens directly. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in encoder outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added comments. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added automatic mapping of kwargs to args in forward. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added encode function. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. PP and TP works (but not together) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Separated get_forward_output_only_func_encode and get_forward_output_only_func_decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * update branch Signed-off-by: ericharper <complex451@gmail.com> * Set headscale false (NVIDIA#4364) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add wandb as dependency (NVIDIA#4365) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Raise trainer error (NVIDIA#4356) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Set headscale false (NVIDIA#4364) (NVIDIA#4366) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Finetuning changes for BART (NVIDIA#4003) * Temp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Checkpoint converter to nemo for bart Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (NVIDIA#4357) * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix logging warning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * 1. Added return logits to validation. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed unkown token during sampling. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegExTokenizer loading. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed ckpt file with samples int(0). Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed allowing enc_tokens to be None. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added ability to ignore tokens by id during decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer .nemo loading issue. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegEx test. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * r1.10.0 untie embeddings weights (NVIDIA#4519) * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added independent decoder embeddings, and independent decoder token_head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in yaml config. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed initialization. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added tests for untied embeddings and decoder token head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Updated share_word_embeddings to share_token_embeddings. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed error in __del__ when TextMemMapDataset fails to build. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed comments. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1.Made method private. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed config names. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed alerts and style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed PP, TP, PP+TP still fails. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: ericharper <complex451@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Update megatron t5 interface to dialogue (NVIDIA#4626) * G2P Aligner (NVIDIA#4604) * Aligner inference notebook in progress. Preprocessing, forward, attn viz Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Hard attn, duration extraction, distance matrix Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Started: phoneme disambiguation using Aligner distance matrix Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Decouple encode_from_g2p() from phoneme tokenizer encode() for disambiguation inference Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner G2P disambiguation using mean L2 embedding distance Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Rename aligner inference notebook Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Header text for Aligner notebook, formatting Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner notebook formatting, header, license updates Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner G2P disambiguation script draft Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner G2P disambiguation script finished Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Remove normalization step to fix words with apostrophes (G2P) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix normalization args for G2P disambiguation Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Allow str to be passed in for supp data, add 'text_normalized' as manifest option Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner G2P script fixes: normalization, tokenization, add brackets around tokens, etc. Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Only disambiguate words in the given heteronyms list Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Filtering option for disambiguation script Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Add confidence thresholding, add PASTY to cmudict entries Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * TTS Aligner tutorial updates to generic path text Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Add confidence to aligner_g2p.py run example Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Move avg word distance function to Aligner encoder, add docstring, fix license Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Aligner Inference notebook updates (link to sample, resources added) Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com> * Fix HF check for model card info (NVIDIA#4628) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Tiny VAD refactoring for postprocessing (NVIDIA#4625) * binarization start index Signed-off-by: fayejf <fayejf07@gmail.com> * fix frame len Signed-off-by: fayejf <fayejf07@gmail.com> * style fix Signed-off-by: fayejf <fayejf07@gmail.com> * rame UNIT_FRAME_LEN Signed-off-by: fayejf <fayejf07@gmail.com> * update overlap script and fix lgtm Signed-off-by: fayejf <fayejf07@gmail.com> * style fi Signed-off-by: fayejf <fayejf07@gmail.com> * Fix ITN pt (NVIDIA#4623) Signed-off-by: Guilherme Steinmann <guist@linse.ufsc.br> * [TN] bug fix "hundred" in Audio-based, added method so split text in sentences (NVIDIA#4610) * fix duplex inference with grammars Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix hundred TN audio bug, add split text Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix header year Signed-off-by: ekmb <ebakhturina@nvidia.com> * style fix Signed-off-by: ekmb <ebakhturina@nvidia.com> * exclude I from roman-ordinal form Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix graph_with_and Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix tests Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix split regex Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix warning Signed-off-by: ekmb <ebakhturina@nvidia.com> * [Text Processing] G2P for OOV and heteronyms (NVIDIA#4624) * add models Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix header and t5 inference Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix lgtm Signed-off-by: ekmb <ebakhturina@nvidia.com> * review fixes Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix if/else and removed unused imports Signed-off-by: ekmb <ebakhturina@nvidia.com> * replace ModelPT with G2PModel Signed-off-by: ekmb <ebakhturina@nvidia.com> * black Signed-off-by: ekmb <ebakhturina@nvidia.com> * add missing headers Signed-off-by: ekmb <ebakhturina@nvidia.com> * jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * fix TRANSFORMERS_OFFLINE flag Signed-off-by: ekmb <ebakhturina@nvidia.com> * jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * jenkins Signed-off-by: ekmb <ebakhturina@nvidia.com> * Update README.rst * Fp16 support for Conformer (NVIDIA#4571) * adding auto-select best precision for mhsa * cleanup * moving mhsa32 check into mhsa * switching to torch.cuda.is_bf16_supported() * now using torch.is_autocast_enabled() * added to non rel mhsa * only forcing 32bit subsampling if using bf16 * removing unused imports * moving contexts to utils Signed-off-by: Dima Rekesh <drekesh@nvidia.com> * formatting Signed-off-by: Dima Rekesh <drekesh@nvidia.com> * naming Co-authored-by: Dima Rekesh <drekesh@nvidia.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Maximum sample-based training for Megatron NMT and Text Memmap based Seq2seq Pre-training (NVIDIA#4396) * Update blendable dataset, and refactor seq2seq data Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Blendable dataset with binarized mmap working Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Pass seed from cfg to dataset Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix multilingual setup Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add on epoch start reconfiguration Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update tokenizer creation for multilingual Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Tmp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update NMT script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove unused import Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update training script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Log consumed samples Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Logging on val epoch end Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove redundant print Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Ckpt averaging for non model parallel megatron models Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Empty Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update error message Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Restore fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove ipdb Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fixes Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Testing a simple solution Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed. Seems to work. Need to validate. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in CSV and text memmap toMEgatron encoder-decoder Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in CSV. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. 2. Fixed bugs. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed bugs. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Updated yaml. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed warnings. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed a bug. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added a test for text_memmap Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * Fix retro Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * add docstrings Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Minor Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Uncomment CI tests and fix existing gpt ci tests Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Tmp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove max step hacking and move on_train_batch_end to base model Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Empty Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@cs.toronto.edu> Co-authored-by: Eric Harper <complex451@gmail.com> * NeMo Megatron Doc updates1 (NVIDIA#4633) * Work on NeMo Megatron OSS documentation Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> * NeMo Megatron doc updates Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: ericharper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: Jocelyn <jocelynh@nvidia.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: Guilherme Steinmann <guist@linse.ufsc.br> Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com> Co-authored-by: Dima Rekesh <bmwshop@gmail.com> Co-authored-by: Dima Rekesh <drekesh@nvidia.com> Co-authored-by: Micha Livne <mlivne@cs.toronto.edu>

* 1. Added vocab_size property to RegExTokenizer. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed passing hiddens directly. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in encoder outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added comments. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added automatic mapping of kwargs to args in forward. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added encode function. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. PP and TP works (but not together) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Separated get_forward_output_only_func_encode and get_forward_output_only_func_decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * update branch Signed-off-by: ericharper <complex451@gmail.com> * Set headscale false (NVIDIA#4364) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add wandb as dependency (NVIDIA#4365) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Raise trainer error (NVIDIA#4356) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Set headscale false (NVIDIA#4364) (NVIDIA#4366) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Finetuning changes for BART (NVIDIA#4003) * Temp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Checkpoint converter to nemo for bart Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (NVIDIA#4357) * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix logging warning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * 1. Added return logits to validation. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed unkown token during sampling. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegExTokenizer loading. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed ckpt file with samples int(0). Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed allowing enc_tokens to be None. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added ability to ignore tokens by id during decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer .nemo loading issue. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegEx test. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * r1.10.0 untie embeddings weights (NVIDIA#4519) * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added independent decoder embeddings, and independent decoder token_head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in yaml config. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed initialization. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added tests for untied embeddings and decoder token head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Updated share_word_embeddings to share_token_embeddings. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed error in __del__ when TextMemMapDataset fails to build. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed comments. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1.Made method private. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed config names. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed alerts and style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed PP, TP, PP+TP still fails. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: ericharper <complex451@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* 1. Added vocab_size property to RegExTokenizer. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed passing hiddens directly. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in encoder outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added comments. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added automatic mapping of kwargs to args in forward. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added encode function. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. PP and TP works (but not together) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Separated get_forward_output_only_func_encode and get_forward_output_only_func_decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * update branch Signed-off-by: ericharper <complex451@gmail.com> * Set headscale false (NVIDIA#4364) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add wandb as dependency (NVIDIA#4365) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Raise trainer error (NVIDIA#4356) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Set headscale false (NVIDIA#4364) (NVIDIA#4366) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Finetuning changes for BART (NVIDIA#4003) * Temp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Checkpoint converter to nemo for bart Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (NVIDIA#4357) * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix logging warning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * 1. Added return logits to validation. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed unkown token during sampling. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegExTokenizer loading. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed ckpt file with samples int(0). Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed allowing enc_tokens to be None. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added ability to ignore tokens by id during decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer .nemo loading issue. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegEx test. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * r1.10.0 untie embeddings weights (NVIDIA#4519) * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added independent decoder embeddings, and independent decoder token_head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in yaml config. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed initialization. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added tests for untied embeddings and decoder token head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Updated share_word_embeddings to share_token_embeddings. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed error in __del__ when TextMemMapDataset fails to build. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed comments. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1.Made method private. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed config names. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed alerts and style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed PP, TP, PP+TP still fails. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: ericharper <complex451@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

* 1. Added vocab_size property to RegExTokenizer. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed passing hiddens directly. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in encoder outputs. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added comments. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added automatic mapping of kwargs to args in forward. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Added encode function. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com> * 1. PP and TP works (but not together) Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Separated get_forward_output_only_func_encode and get_forward_output_only_func_decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * update branch Signed-off-by: ericharper <complex451@gmail.com> * Set headscale false (NVIDIA#4364) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add wandb as dependency (NVIDIA#4365) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Raise trainer error (NVIDIA#4356) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Set headscale false (NVIDIA#4364) (NVIDIA#4366) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: smajumdar <smajumdar@nvidia.com> * Finetuning changes for BART (NVIDIA#4003) * Temp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Checkpoint converter to nemo for bart Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * Make position embedding expansion specific to a batch to avoid checkpoint size mismatches (NVIDIA#4357) * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix logging warning Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> * 1. Added return logits to validation. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed unkown token during sampling. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegExTokenizer loading. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed ckpt file with samples int(0). Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed allowing enc_tokens to be None. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added ability to ignore tokens by id during decode. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed regex tokenizer .nemo loading issue. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed RegEx test. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * r1.10.0 untie embeddings weights (NVIDIA#4519) * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added independent decoder embeddings, and independent decoder token_head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added support in yaml config. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed initialization. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Added tests for untied embeddings and decoder token head. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Updated share_word_embeddings to share_token_embeddings. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed error in __del__ when TextMemMapDataset fails to build. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed comments. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1.Made method private. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed config names. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed alerts and style. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Fixed PP, TP, PP+TP still fails. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> * 1. Debugging. Signed-off-by: Micha Livne <mlivne@cs.toronto.edu> Co-authored-by: Micha Livne <mlivne@nvidia.com> Co-authored-by: ericharper <complex451@gmail.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

michalivne and others added 30 commits May 18, 2022 02:36

1. Added vocab_size property to RegExTokenizer.

e272b8a

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Fixed passing hiddens directly.

2bcf1b7

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

bc60252

Signed-off-by: Micha Livne <mlivne@nvidia.com>

Merge branch 'main' into megamolbart-regex-tokenizer-fixes

52f5c3f

1. Debugging.

8cdc3ca

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

43be37b

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Fixed style.

67f5748

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into megamolbar…

1a86cd1

…t-regex-tokenizer-fixes

1. Added support in encoder outputs.

8d084ce

Signed-off-by: Micha Livne <mlivne@nvidia.com>

Merge branch 'megamolbart-regex-tokenizer-fixes' of github.com:michal…

2bc81b8

…ivne/NeMo into megamolbart-regex-tokenizer-fixes

1. Added comments.

66b35ce

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

4bc489b

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Added automatic mapping of kwargs to args in forward.

3ba4242

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Added encode function.

fa7f600

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

83caff4

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

beee4a2

Signed-off-by: Micha Livne <mlivne@nvidia.com>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into megamolbar…

08c4737

…t-regex-tokenizer-fixes

1. Debugging.

85c5785

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

1e1685a

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

a480cf4

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

e3a61c7

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

3d00a99

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

3c4edab

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

7a2082d

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

c7cf350

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

9866568

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

1e3bcce

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

e363e4d

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

cf3308c

Signed-off-by: Micha Livne <mlivne@nvidia.com>

1. Debugging.

c17e3e0

Signed-off-by: Micha Livne <mlivne@nvidia.com>

Merge branch 'main' into r1.10.0-megamolbart

8c50ac8

michalivne mentioned this pull request Jul 25, 2022

r1.10.0 MegaMolBART Compatibility #4462

Closed

8 tasks

1. Fixed config names.

e630bf7

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Merge branch 'main' into r1.10.0-megamolbart

932cae6

michalivne and others added 5 commits July 26, 2022 10:55

1. Fixed alerts and style.

245aaef

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Merge branch 'main' into r1.10.0-megamolbart

55274e8

Merge branch 'r1.10.0-megamolbart' of github.com:michalivne/NeMo into…

837e51e

… r1.10.0-megamolbart

1. Debugging.

a0b7946

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Merge branch 'main' into r1.10.0-megamolbart

d1db727

Merge branch 'main' into r1.10.0-megamolbart

ca45e6d

okuchaiev requested a review from Davood-M July 28, 2022 20:06

1. Fixed PP, TP, PP+TP still fails.

61acb6e

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

michalivne and others added 2 commits July 29, 2022 02:25

Merge branch 'main' into r1.10.0-megamolbart

7434bc7

1. Debugging.

eee5810

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

MaximumEntropy self-requested a review July 29, 2022 00:34

MaximumEntropy approved these changes Jul 29, 2022

View reviewed changes

MaximumEntropy merged commit 59d635c into NVIDIA:main Jul 29, 2022

michalivne deleted the r1.10.0-megamolbart branch August 1, 2022 22:35

michalivne restored the r1.10.0-megamolbart branch August 4, 2022 18:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

r1.10.0 MegaMolBART Compatibility #4603

r1.10.0 MegaMolBART Compatibility #4603

michalivne commented Jul 25, 2022

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 27, 2022

lgtm-com bot commented Jul 28, 2022

lgtm-com bot commented Jul 28, 2022

r1.10.0 MegaMolBART Compatibility #4603

r1.10.0 MegaMolBART Compatibility #4603

Conversation

michalivne commented Jul 25, 2022

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Jul 27, 2022

lgtm-com bot commented Jul 28, 2022

lgtm-com bot commented Jul 28, 2022