Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend multimodal/speech_llm with lhotse, t5 and bestow supports #9169

Merged
merged 470 commits into from
Jun 7, 2024
Merged

Extend multimodal/speech_llm with lhotse, t5 and bestow supports #9169

merged 470 commits into from
Jun 7, 2024
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Dec 8, 2023

  1. Fixes

    @pzelasko
    pzelasko committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    0c7b399 View commit details
    Browse the repository at this point in the history

  2. Docs fix

    @pzelasko
    pzelasko committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    3b282aa View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. Add support for custom NeMo fields in Lhotse-NeMo adapters (attach to… 

    … cut.custom)
    @pzelasko
    pzelasko committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    5034d77 View commit details
    Browse the repository at this point in the history

  2. Add support for custom NeMo fields in Lhotse-NeMo adapters (attach to… 

    … cut.custom)
    @pzelasko
    pzelasko committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    31b1973 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    0880d44 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'feature/lhotse-integration' of https://github.com/pzela… 

    …sko/nemo into feature/lhotse-integration
    @pzelasko
    pzelasko committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    30ce202 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    5f11fdb View commit details
    Browse the repository at this point in the history

  6. support distributed_fused_adam 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    02f0f0a View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. support distributed_fused_adam 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    378af7c View commit details
    Browse the repository at this point in the history

  2. Add support for sharded NeMo manifest files

    @pzelasko
    pzelasko committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    35412fb View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    1f2acde View commit details
    Browse the repository at this point in the history

  4. support megatron_amp_O2 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    5b58e69 View commit details
    Browse the repository at this point in the history

  5. Support heterogeneous sampling rates in non tarred NeMo manifests

    @pzelasko
    pzelasko committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    37cabcc View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. migrate to PTL2.0 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    1270609 View commit details
    Browse the repository at this point in the history

  2. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    6df13f1 View commit details
    Browse the repository at this point in the history

  3. update manifest util 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    fa0493a View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2023

  1. Support multiple tokenizer/parser types, aggregate tokenizers, and cu… 

    …stom language fields
    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    22e3bff View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    60cdea6 View commit details
    Browse the repository at this point in the history

  3. fix

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    96020e6 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'feature/lhotse-integration' of https://github.com/pzela… 

    …sko/NeMo into feature/lhotse-integration
    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    949fbbc View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    5630ad4 View commit details
    Browse the repository at this point in the history

  6. fix

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    fc13c42 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'feature/lhotse-integration' of https://github.com/pzela… 

    …sko/NeMo into feature/lhotse-integration
    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    cee170f View commit details
    Browse the repository at this point in the history

  8. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    6eb16fa View commit details
    Browse the repository at this point in the history

  9. agg and normal tokenizers actually work

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    fa73e72 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'feature/lhotse-integration' of https://github.com/pzela… 

    …sko/NeMo into feature/lhotse-integration
    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    0fe901c View commit details
    Browse the repository at this point in the history

  11. Support weights for NeMo tarred manifests

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    c014c85 View commit details
    Browse the repository at this point in the history

  12. Temporarily hardcoded pnc stripping/lowercasing

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    034f55f View commit details
    Browse the repository at this point in the history

  13. fix

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    1526dcb View commit details
    Browse the repository at this point in the history

  14. Merge branch 'feature/lhotse-integration' into canary

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    94fcb1f View commit details
    Browse the repository at this point in the history

  15. make pnc hack configurable from the config and disabled by default

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    0e589a5 View commit details
    Browse the repository at this point in the history

  16. fix the hack

    @pzelasko
    pzelasko committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    31e5bf7 View commit details
    Browse the repository at this point in the history

  17. migrate to ptl2.1 to support multiple dataloaders 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    2eb320a View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2023

  1. support encoder overwrite 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    8437bdd View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. update misc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    3fc0db6 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. fix eval and clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    4f947ce View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2023

  1. Merge branch 'heh/modular_speechlm_tmp' into modular_speechllm_clean_… 

    …cross_ptn2.1

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    4b70343 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2023

  1. support add_sep for perception model 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    63131d0 View commit details
    Browse the repository at this point in the history

  2. fix Lightning-AI/pytorch-lightning#18803 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    318f784 View commit details
    Browse the repository at this point in the history

Commits on Dec 25, 2023

  1. add_bos 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    59c0d4d View commit details
    Browse the repository at this point in the history

Commits on Dec 27, 2023

  1. Transformer decoder with conditioning for canary (NVIDIA#8091) 

    * initial commit for multi-task conf-enc transf-dec for canary

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* removing decoder states caching during training

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>
Co-authored-by: Krishna Puvvada <kpuvvada@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    @krishnacpuvvada @pre-commit-ci
    3 people committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    72cbc94 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2023

  1. Option to limit the number of open streams (NVIDIA#8095)

    @pzelasko
    pzelasko committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    dc45efc View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2023

  1. audio signal support in multi 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Dec 30, 2023
    Configuration menu
    Copy the full SHA
    401efed View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. update asr evaluator 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    12487c2 View commit details
    Browse the repository at this point in the history

  2. fix from 

    NVIDIA@fcc0f9f
and
NVIDIA@f97c901

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    0535fdd View commit details
    Browse the repository at this point in the history

  3. transcribe fn for Canary models (NVIDIA#8110) 

    * improve readability

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* adding context in transcribe function for ConfTransfModels

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* supporting relative paths in transcribe function for canary

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* removing cuts.sort_by_duration in __getitem__ to maintain manifest order during inference

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Krishna Puvvada <kpuvvada@nvidia.com>
Co-authored-by: Krishna Puvvada <kpuvvada@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    @krishnacpuvvada @pre-commit-ci
    3 people committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    03255be View commit details
    Browse the repository at this point in the history

  4. update for evaluation 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    6148198 View commit details
    Browse the repository at this point in the history

  5. update for eval 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    6a237c9 View commit details
    Browse the repository at this point in the history

  6. update for evaluation 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    15d162b View commit details
    Browse the repository at this point in the history

  7. fix bleu 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    faeaac3 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. fix typo 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    9fc3ae5 View commit details
    Browse the repository at this point in the history

  2. Add missing audio_filepath validation for Canary (NVIDIA#8119) 

    * Add missing audio_filepath validation for Canary

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    @pzelasko @pre-commit-ci
    pzelasko and pre-commit-ci[bot] committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    1e7cfd6 View commit details
    Browse the repository at this point in the history

  3. add default concat_sampling_probabilities 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    8b04025 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Merge branch 'modular_speechllm_clean_cross_ptn2.1' into canary_speec… 

    …hllm

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    abaa3b0 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. support lhotse dataset in speechllm 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    0ddedd4 View commit details
    Browse the repository at this point in the history

  2. bypass get_iterator_k_split 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    467fb24 View commit details
    Browse the repository at this point in the history

  3. tmp fix 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    1b169fa View commit details
    Browse the repository at this point in the history

  4. try to use fixed batch with megatron 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    5dc8660 View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2024

  1. add batch logging 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    c0f5f0c View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. support unfrozen llm 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    6d6be8a View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. Create README.md 

    Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52
    stevehuang52 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    209f752 View commit details
    Browse the repository at this point in the history

  2. Update README.md 

    Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52
    stevehuang52 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    01dd0d6 View commit details
    Browse the repository at this point in the history

  3. Update README.md 

    Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52
    stevehuang52 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    528d1bf View commit details
    Browse the repository at this point in the history

  4. update 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    d94f9dd View commit details
    Browse the repository at this point in the history

  5. rename 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    dbad4ac View commit details
    Browse the repository at this point in the history

  6. add llama prompt template 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    02e91d3 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. update and refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    73736ad View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. support sample alpha 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    0d5c6da View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. support lhotse validation set and canary pretrained ckpt with pseudo … 

    …label

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    18b27d1 View commit details
    Browse the repository at this point in the history

  2. make sure backward compatibility 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    c12044a View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. remove pad 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    76be5ce View commit details
    Browse the repository at this point in the history

  2. make sure asr_model is frozen 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    efa862a View commit details
    Browse the repository at this point in the history

  3. support greedy decoding 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    3ca65df View commit details
    Browse the repository at this point in the history

  4. valid on lhotse 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    4e17ced View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. fix multi dataloader in val case for lhotse SALM; add default data 

    names; keep asr model tokenizer by default to enable adding canary
dataset

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    a7aeddc View commit details
    Browse the repository at this point in the history

  2. remove the bruteforce _keep_special_tokens implementation 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4ccc271 View commit details
    Browse the repository at this point in the history

  3. decoding_ratio and convert_canary_prompt_to_text support 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    6817833 View commit details
    Browse the repository at this point in the history

  4. canary_tokens_augment_ratio 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f437770 View commit details
    Browse the repository at this point in the history

  5. debug 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    bf51ad4 View commit details
    Browse the repository at this point in the history

  6. bug fix 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    1aa6fe8 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. fix lhotse based eval of llama canary model 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    a38488d View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. support some overwrite for eval 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    430c5bf View commit details
    Browse the repository at this point in the history

  2. support zero shot prompt in training 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    a7dcafe View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2024

  1. support cross attention based SALM 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    62a0cf0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'canary_speechllm1' of github.com:zhehuaichen/NeMo into … 

    …canary_speechllm1
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    f17e8fa View commit details
    Browse the repository at this point in the history

  3. support cross attention based SALM 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    42d74bf View commit details
    Browse the repository at this point in the history

  4. fix for batch train/valid of cross 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    50571f2 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'canary_speechllm1_cross' of github.com:zhehuaichen/NeMo… 

    … into canary_speechllm1_cross
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    7160027 View commit details
    Browse the repository at this point in the history

  6. support learnable gate and plotting 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    2fe48aa View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. support using pseudo label in prompt rather than cross att 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    39545b7 View commit details
    Browse the repository at this point in the history

  2. bug fix for perception cfg and context tokens shift 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    2cec2f5 View commit details
    Browse the repository at this point in the history

  3. DentityConnectorsAdd 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    fa57fb2 View commit details
    Browse the repository at this point in the history

  4. fix ckpt saving 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    666aa44 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Support RnnGatedCrossAttention 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    fa8e00e View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2024

  1. add include_ffw and fix _optimizer_param_groups for all unfrozen run 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    175b66e View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. support grad acc when using bucket 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    dcb5084 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. support TransformerCrossAttention 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    6c0a798 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2024

  1. support ProjectTransformerCrossAttention 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    e9935dc View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. support ++model.use_am_tokenizer ++model.override_vocab_size ++model.… 

    …override.hidden_size

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    45932a5 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. support question set on val without canary 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    d7cc642 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. support load_audio_encoder and wip in optim_param_groups 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    eeaad1f View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. minor fix for audio pretrain model init 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    2504a0a View commit details
    Browse the repository at this point in the history

  2. simplify canary_tokens_augment 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    a9478ef View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. use question in the manifest if it exists 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    b754b61 View commit details
    Browse the repository at this point in the history

  2. support dataset weighting for non tar 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    c94a632 View commit details
    Browse the repository at this point in the history

  3. Update SpeechLLM code (NVIDIA#8475) 

    * add pleasefixme marker for potential failed nightly tests. (#7678)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Add new text segmentation library for better TTS quality (#7645)

* Add new text segmentation library for better TTS quality
* Update zh_cn_pinyin.py

added detailed instruction on how to install pkuseg.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update requirements_tts.txt

remove pkuseg as the default dependency of NeMo TTS, and instead, direct users to manually install pkuseg if they really need.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>


---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Create PrecisionPlugin for megatron_ckpt_to_nemo.py trainer (#7767) (#7774)

* Create PrecisionPlugin for megatron_ckpt_to_nemo.py trainer



* Add ddp_find_unused_parameters_true for punctuation_capitalization_train_evaluate.py



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add '32-true' for precision values



---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix(clustering_diarizer.py): fix typo (#7772)

Signed-off-by: Jean-Louis Queguiner <jean-louis.queguiner@gadz.org>

* fix(diarization-README): typo (#7771)

Signed-off-by: Jean-Louis Queguiner <jean-louis.queguiner@gadz.org>

* Fix bug wrt change decoding strategy for bpe models (#7762) (#7764)

* Fix bug wrt change decoding strategy for bpe models



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove incorrect extra argument for load_from_checkpoint_dir() (#7500)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add nemo to mcore GPT conversion script  (#7730)

* add conversion script

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove references to 'ckpt'

Signed-off-by: Chen Cui <chcui@nvidia.com>

* add one more sanity check to make sure there is no unexpected keys in state dict

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* make cpu loading work

Signed-off-by: Chen Cui <chcui@nvidia.com>

* make script work for llama2 models

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address code check

Signed-off-by: Chen Cui <chcui@nvidia.com>

* remove trainer precision (was for old sanity check)

Signed-off-by: Chen Cui <chcui@nvidia.com>

* fix script for llama2 model

Signed-off-by: Chen Cui <chcui@nvidia.com>

* remove commented code

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix bug in ConditionalInput: cat along the feature dim, not the batch dim (#7785)

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* Add some docs and update scripts for ASR (#7790)

* Add some docs and update scripts

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* set context for text memmap to fork (#7784)

* set context for text memmap to fork

Signed-off-by: arendu <adithyare@nvidia.com>

* typo

Signed-off-by: arendu <adithyare@nvidia.com>

---------

Signed-off-by: arendu <adithyare@nvidia.com>

* add training with multiple audios

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Support flash decoding (#7744)

* Add flash-decoding

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>

* Change accelerator to 'auto' in nlp_checkpoint_port.py (#7761)

* Change accelerator to 'auto' in nlp_checkpoint_port.py (#7747)

* Change accelerator to auto

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Pass omegaconf object to trainer in nlp_checkpoint_port.py

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Pass omegaconf object to trainer in export.py

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* docs: fix typos (#7758)

Signed-off-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Snake act (#7736)

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Update gpt_dataset.py (#6963)

Signed-off-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Signed-off-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Add selection criteria for reference audios in the `GlobalStyleToken` submodule (#7788)

* add selection criteria for reference audios

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* Update configuration files

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* add informative comment in config files

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* sample random index for reference audio selection

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: anferico <f.cariaggi4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update text server to support compute logprobs (#7733)

* update text server to support compute logprobs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo

---------

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add multi-layer feat extract and fix random question insertion

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Configure MCore logger (#7781)

Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>

* Revert "PEFT eval fix (#7626) (#7638)" (#7693)

This reverts commit f03dd660bd26d88fd569e76c6f74b83a7c203ff9.

* remove TN from ctc_segm tut (#7807)

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [TTS] Support audio offsets in TTS data loaders (#7156)

* [TTS] Support audio offsets in TTS data loaders

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Change docstring mentions of .pt to .npy

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* Update Apex install command in Dockerfile (#7794) (#7804)

* move core install to /workspace (#7706)



* update apex install in dockerfile



* use fetch head



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix typo

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Nemo to HF converter for LLaMA model (#7770)

* Create config_llama_truncate.yaml

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Add files via upload

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update config_llama_truncate.yaml

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* clean up trainer

* remove dependency on yaml config. load config from nemo file instead.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* enable ckpt saving into other precision formats

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* support 70b + cleanup qkv slice logic

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug

* move hf model folder code from comment to function and add instruction to run

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>

* Save best NeMo model only when necessary (#7836)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* add guard if its a distributed checkpoint (#7845)

Signed-off-by: Gerald Shen <geshen@nvidia.com>

* Fix tn duplex (#7808)

* fix duplex tn infer

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix typo

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix TN docs

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update transformers cache on Jenkins (#7854)

* update transformers cache

Signed-off-by: eharper <eharper@nvidia.com>

* update

Signed-off-by: eharper <eharper@nvidia.com>

* add cd

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>

* Update README.rst for container update (#7844)

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

* Add support for finetuning with huggingface datasets (#7834)

* add finetune with huggingface dataset

Signed-off-by: stevehuang52 <heh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update yaml

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update and refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add extrac hf text and update

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update and refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* move dataset dependency to common

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add docstring

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Add to Dics

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add ci test

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add max steps in jenkins

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* reduce max steps

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* jenkins test

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add bs=2

Signed-off-by: Nithin Rao Koluguri <nithinraok>

---------

Signed-off-by: stevehuang52 <heh@nvidia.com>
Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>

* Multimodal merge (#7728)

* ControlNet TRT export

* Final MR before release

* SD2 update

* Fixed export issue

* Fix for instruct p2p and reformat

* Fix SD export issue

* Add nemo clip export for DB

* Fix ins pix2pix

* fix sd2 config

* [Mingyuan Ma] BF16 and SD conversion script

* [Imagen] NHWC Feature

* Fix .nemo loading issue for NeMo CLIP in SD

* NeMo r1.20.0 Multimodal Merge

* fix the inductor issue in inference

* Fix inductor loading .nemo issue

* Add Neva Model Support

* Imagen Optimizations

* Neva inference code

* NeMo TOT 1.21 to Internal/main

* Update neva_inference.yaml

* REBASING  for latest code changes

* Update internal/main to main tot

* Parallel DDIM implementation

* 1. Fixing indentation bug. (#7352)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* NeMo MCore llama2 support + MCore PEFT adapters (#7299)

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* add TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* add todo

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* remove import

Signed-off-by: ericharper <complex451@gmail.com>

* small clean up

Signed-off-by: ericharper <complex451@gmail.com>

* update hidden size in peft base model, add mcore commit to jenkins

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* add config obj to flash attention tests

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove sequence parallel arg

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to test

Signed-off-by: ericharper <complex451@gmail.com>

* get hidden_size from config

Signed-off-by: ericharper <complex451@gmail.com>

* add try except

Signed-off-by: ericharper <complex451@gmail.com>

* use default

Signed-off-by: ericharper <complex451@gmail.com>

* update config with hidden size

Signed-off-by: ericharper <complex451@gmail.com>

* remove arg

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* comment out jenkins test

Signed-off-by: ericharper <complex451@gmail.com>

* revert import

Signed-off-by: ericharper <complex451@gmail.com>

* build transformer config

Signed-off-by: ericharper <complex451@gmail.com>

* add model to provider func

Signed-off-by: ericharper <complex451@gmail.com>

* update forward and float16 wrapper

Signed-off-by: ericharper <complex451@gmail.com>

* instantiate model parallel config after init model parallel

Signed-off-by: ericharper <complex451@gmail.com>

* set virtual rank

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add GQA config to megatron gpt model (#7096)

* Add GQA config in gpt config file

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Verify mcore is enabled when using GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* mcore llama2 ckpt conversion & small fix

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Add inference & sft config by Hongbin

Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add inference param. update TP/PP script to support mcore gpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* modify ckpt conversion script (adding model cast)

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ckpt conversion use relative path for config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* add todo

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove import

Signed-off-by: ericharper <complex451@gmail.com>

* small clean up

Signed-off-by: ericharper <complex451@gmail.com>

* update hidden size in peft base model, add mcore commit to jenkins

Signed-off-by: ericharper <complex451@gmail.com>

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add config obj to flash attention tests

Signed-off-by: ericharper <complex451@gmail.com>

* remove args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove sequence parallel arg

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to test

Signed-off-by: ericharper <complex451@gmail.com>

* get hidden_size from config

Signed-off-by: ericharper <complex451@gmail.com>

* add try except

Signed-off-by: ericharper <complex451@gmail.com>

* use default

Signed-off-by: ericharper <complex451@gmail.com>

* update config with hidden size

Signed-off-by: ericharper <complex451@gmail.com>

* remove arg

Signed-off-by: ericharper <complex451@gmail.com>

* comment out jenkins test

Signed-off-by: ericharper <complex451@gmail.com>

* revert import

Signed-off-by: ericharper <complex451@gmail.com>

* remove optimizer_idx

Signed-off-by: eharper <eharper@nvidia.com>

* prefetch num microbatches

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* fix for p-tuning sequence parallel

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support SFT/distOpt mcore (#7207)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* change layer names for SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug in SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* build transformer config

Signed-off-by: ericharper <complex451@gmail.com>

* add model to provider func

Signed-off-by: ericharper <complex451@gmail.com>

* update forward and float16 wrapper

Signed-off-by: ericharper <complex451@gmail.com>

* instantiate model parallel config after init model parallel

Signed-off-by: ericharper <complex451@gmail.com>

* set virtual rank

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add GQA config to megatron gpt model (#7096)

* Add GQA config in gpt config file

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Verify mcore is enabled when using GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* remove import

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rollback model cast for p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update for dist adam

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use get_gpt_module_list

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ptl2.0 patch for llama config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add plugins to trainer in scripts

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix activation checkpointing mcore

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix variable names

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* overwrite normalization type for mcore/te

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Update megatron_llama_sft.yaml

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* add PEFT adapter support for mcore gpt path (#7276)

* implementation for mcore adapter/mxins

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* small fix for lora and ptuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support layerwise peft

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support multiple target layers

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support lora GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support amp O2

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert & more O2 fix

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lora inject to attention

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support lora weight tying

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add copyright header

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback ptuning name change. full string match mcore target

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove comment

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* clean up config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Sync llama branch (#7297)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* change layer names for SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug in SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug: cpu initialization is not really enabled

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* add use_cpu_initialization to TransformerConfig

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug: wrong config path when using relative cjpt path

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* revert mcore config change

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

* clean up ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback git merge errors

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update mcore, add check for mcore+te

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* formatting

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* make sft test dataset optional. fix indentation in config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* one more fix for optional test set

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support merging lora weights in mcore

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update mcore for cpu init

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ckpt conversion for code llama

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add seq_len_interpolation_factor support for long-context llama ckpts (#7312)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add seq_len_interpolation_factor

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

* fix old ptuning model, update mcore to support seq_len_interpolation_factor

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support fused layernorm linear, fix ptuning O2

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* drop loss mask for mcore for now

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* disable dist ckpt in peft

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix loading non dist ckpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add ckpt conversion to CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* mcore_mixin docstring

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor change in mcore peft error message

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix amp o2 in lora weight tying

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* correct mcore fp8 config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add TE installation

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support mcore adapter tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* comment out new CI test. rollback docker image

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ignore FA tests, try new CI on 23.08

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* mark new CI as L2, put to beginning to test

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor fix for prompt learning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback to 23.06. comment out CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor fix ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor rollback gpt model change

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: eharper <eharper@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: eharper <eharper@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: Kelvin Liu <lhb8125@users.noreply.github.com>

* Hiddens modules documentation (#7303)

* 1. Changed hiddens transformations module from `transformations` to `hiddens`.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Finished doc.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Support for flash attention 2.0 (#7063)

* Add flash attn 2

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add FA2 feature

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove debugging

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Signed-off-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* lora merge fix for O2 names (#7325)

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* adjust key names based on O2

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* minor

Signed-off-by: arendu <adithyare@nvidia.com>

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* multiple fields can form a context (#7147)

* list of context fields and flexible prompt template

Signed-off-by: arendu <adithya.r@gmail.com>

* list of fields for context

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Add multiple truncation fields and middle truncation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Compatible to old ckpt

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix tokenize detokenize issue

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove detokenization, add truncation augmentation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve comments

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove unused import

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert eos

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Add tokenizer space_sensitive attribute

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix erorr and use re

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Change assert logic

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow adi suggestion

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove merge function

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add example and comment

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove context_key and add comment

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove random truncation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix template none

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Signed-off-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>

* Load buffers in checkpoint (#7357)

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* Add migration guide for lightning 2.0 upgrade (#7360)

* Add lightning 2.0 migration guide in NeMo docs

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add remaining guide for lightning 2.0 upgrade

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Remove line spill over and continue in next line

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add missing dataloader_iter in the guide

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Fix minor typo

Signed-off-by: Abhishree <abhishreetm@gmail.com>

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* adding bias_dropout_add_fusion option for BERT (#7332)

Signed-off-by: Alexander Jipa <azzhipa@amazon.com>
Co-authored-by: Alexander Jipa <azzhipa@amazon.com>

* [TTS] Change audio codec token type to TokenIndex (#7356)

Signed-off-by: Ryan <rlangman@nvidia.com>

* enable selective unfreeze (#7326)

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* avoid PTL method conflicts

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix typos (#7361)

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

---------

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* pin numba=0.57.1 to fix reinstall.sh error (#7366)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update new conversion script for converting safetensors.

* Upgrade pytorch container to 23.08 (#7353)

* upgrade pytorch container

Signed-off-by: eharper <eharper@nvidia.com>

* use mcore

Signed-off-by: eharper <eharper@nvidia.com>

* revert test change

Signed-off-by: eharper <eharper@nvidia.com>

* pleasefixme

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check for ampere

Signed-off-by: eharper <eharper@nvidia.com>

* comment test temporarily

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* enable fp32 optimizer for output_layer in mcore (#7355)

Signed-off-by: lhb8125 <lhb8125@gmail.com>

* revert comment (#7368)

Signed-off-by: eharper <eharper@nvidia.com>

* Update to core 23.08 branch ToT (#7371)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* upper bounding ptl (#7370)

Signed-off-by: eharper <eharper@nvidia.com>

* fix pipeline parallel inference (#7367)

* fix pp inference

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix for peft tied weights (#7372)

Signed-off-by: arendu <adithyare@nvidia.com>

* fixed trainer.strategy=auto from None. (#7369)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* add O2 option in gpt eval (#7358)

* add O2 option in eval

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add doc for O2 config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add to llama inference config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Move model precision copy (#7336)

* move cfg precision set to megatron base model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* remove copy from other models

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* modify attribute not arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* fix gpt model test for ptl 2.0

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* rename function and add docstring

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* replace precision to dtype conditionals with func call

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* unnecessary function and cfg reset

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* set default value

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* fix precision lookup in a few more places

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* rename mapping function

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* ununsed import

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* save torch datatype to model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* set weights precision wrt amp o2

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Revert "set weights precision wrt amp o2"

This reverts commit 313a4bfe5eb69d771a6d2433898c0685836aef5c.

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* revert half precision at inference attempt

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move autocast dtype to base model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move params dtype to base model, enable fp16 O2 inf

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* unused imports

Signed-off-by: Maanu Grover <maanug@nvidia.com>

---------

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Fix PEFT checkpoint loading (#7388)

* Fix PEFT checkpoint loading

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use distributed optimizer support for multiple dtypes (#7359)

* Update distopt wrapper with multiple dtype support

Remove manual handling of separate FP32 optimizer.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Use distopt support for contiguous buffers with multiple dtypes

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix typo

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Separate distopt buckets for first GPT layer and non-overlapped params

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Add distopt logic for int dtypes

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Apex commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Remove unused variables

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Apex commit in README and Jenkensfile

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug Dockerfile and Jenkinsfile

Signed-off-by: Tim Moon <tmoon@nvidia.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* minor fix for llama ckpt conversion script (#7387)

* minor fix for llama ckpt conversion script

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* Update Jenkinsfile

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* remove fast_swiglu configuration

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix wrong calling of librosa.get_duration() in notebook (#7376)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* [PATCH] PEFT import mcore (#7393)

* [PATCH] PEFT import mcore

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Added a callback for logging initial data (#7384)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Update Core Commit (#7402)

* Update Core Commit

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* update commit

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Use cfg attribute in bert (#7394)

* use cfg attribute instead of arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* use torch_dtype in place of cfg.precision

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move precision copy before super constructor

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* use trainer arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

---------

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Add support for bias conversion in Swiglu models (#7386)

* Add support for bias conversion in Swiglu models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for auto extracting tokenizer model

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add support for auto extracting tokenizer model

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix issue with missing tokenizer

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update save_to and restore_from for dist checkpointing (#7343)

* add dist ckpt to save to, in progress

Signed-off-by: eharper <eharper@nvidia.com>

* move dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update restore from, need to figure out how to initialize distributed

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* launch distrib if needed when restoring dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* when using mcore we can change tp pp on the fly

Signed-off-by: eharper <eharper@nvidia.com>

* add load_from_checkpoint support for dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update llama convert script to save dist .nemo

Signed-off-by: eharper <eharper@nvidia.com>

* fix load dist ckpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* setup TE TP groups if needed

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* setup te tp groups if needed

Signed-off-by: eharper <eharper@nvidia.com>

* remove import

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>
Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>

* fix forward for with mcore=false (#7403)

Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Co-authored-by: Jimmy Zhang <jiemingz@nvidia.com>

* Fix logging to remove 's/it' from progress bar in Megatron models and add train_step_timing (#7374)

* Add CustomProgressBar class to exp_manager and trainer callbacks

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix the progress bar to reflect total microbatch cnt

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Modify CustomProgressBar class

1) Modify CustomProgressBar class to update progress bar per global_step instead of per microbatch
2) Add the callback to other megatron training/finetuning files that are not using MegatronTrainerBuilder

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add CustomProgressBar callback to tuning files

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Set Activation Checkpointing Defaults (#7404)

* Set Activation Checkpointing Defaults

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check for None

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* make loss mask default to false (#7407)

Signed-off-by: eharper <eharper@nvidia.com>

* Add dummy userbuffer config files (#7408)

Signed-off-by: Sangkug Lym <slym@nvidia.com>

* add missing ubconf files (#7412)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* New tutorial on Speech Data Explorer (#7405)

* Added Google Colab based tutorial on Speech Data Explorer 

Signed-off-by: George Zelenfroynd <gzelenfroind@nvidia.com>

* Update ptl training ckpt conversion script to work with dist ckpt (#7416)

* update ptl convert script

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* don't break legacy

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Allow disabling sanity checking when num_sanity_val_steps=0 (#7413)

* Allow disabling sanity checking when num_sanity_val_steps=0

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Update num_sanity_val_steps to be a multiple of num_microbatches

Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add comprehensive error messages (#7261)

Signed-off-by: Anton Peganov <apeganov@nvidia.com>

* check NEMO_PATH (#7418)

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* layer selection for ia3 (#7417)

* layer selection for ia3

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix missing pip package 'einops' (#7397)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Fix failure of pyaudio in Google Colab (#7396)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update README.md: output_path --> output_manifest_filepath (#7442)

Signed-off-by: Samuele Cornell <cornellsamuele@gmail.com>

* Updating FlashAttention API to match FlashAttentionV2

* Multiple fixes for mm

* Fix CI inductor issue and update to torch compile

* Remove suppress error

* Fix when conversion config uses fp16 and it complains about precision plugin

* Fixing FAv2 API usage

* Initial release of content filtering model

* Added synthetic dataloader for precached and online mode

* Mingyuanm/dreambooth opt

* Add llama2 support in neva training

* Fix sampler length

* Fix all precision issues in nemo multimodal

* Add rope dynamic linear scaling (#7437)

* Add dynamic linear scaling

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>

* Fix None dataloader issue in PTL2.0 (#7455)

* Fix None dataloader issue in PTL2.0

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updating values of self._validation_dl and self._test_dl as well

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updating values of self._validation_dl and self._test_dl as well

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [ASR] Confidence measure -> method renames (#7434)

* measure -> method

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add steps for document of getting dataset 'SF Bilingual Speech' (#7378)

* Add steps for document of getting dataset 'SF Bilingual Speech'

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update datasets.rst

added a link from a tutorial demonstrating detailed data prep steps.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* RNN-T confidence and alignment bugfix (#7381)

* new frame_confidence and alignments lists are now always created after the while loop

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* tests added

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

---------

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* Fix resume from checkpoint in exp_manager (#7424) (#7426)

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix checking of cuda/cpu device for inputs of Decoder (#7444)

* Fix checking of cuda/cpu device for inputs of Decoder

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update tacotron2.py

Signed-off-by: Jason <jasoli@nvidia.com>

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Signed-off-by: Jason <jasoli@nvidia.com>
Co-authored-by: Jason <jasoli@nvidia.com>

* Fix failure of ljspeech's get_data.py (#7430)

* Fix failure of ljspeech's get_data.py

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Fix audio codec type checks (#7373)

* [TTS] Fix audio codec type checks

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix audio codec tests

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Add dataset to path of logged artifacts (#7462)

* [TTS] Add dataset to path of logged artifacts

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Revert axis name back to Audio Frames

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* Fix sft dataset truncation (#7464)

* Add fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) (#7330)

* striding_conv1d_k5 and dw_striding_conv1d_k5 subsampling

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* transpose conv1d inputs

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, s…
    @stevehuang52 @XuesongYang
    @RobinDong @pre-commit-ci @github-actions @athitten @jqueguiner @titu1994 @ericharper @cuichenx @anferico @arendu @hsiehjackson @yzhang123 @shuoer86 @nithinraok @yaox12 @MaximumEntropy @Zhilin123 @mikolajblaz @ekmb @rlangman @aklife97 @uppalutkarsh @anteju @gshennvm @fayejf @yaoyu-33 @Victor49152 @milesial @suiyoubi @meatybobby @maanug-nv @michalivne @blahBlahhhJ @lhb8125 @okuchaiev @azzhipa @omahs @timmoon10 @JimmyZhang12 @jiemingz @erhoo82 @Jorjeous @PeganovAnton @karpnv @popcornell @hXl3s @KunalDhawan @GNroy @blisc @burchim @Kipok @janekl @The0nix @stas00 @redoctopus @GiacomoLeoneMaria @odelalleau @marcromeyn @hkelly33 @BestJuly @erastorgueva-nv @dimapihtar @menon92 @ahmadki @ssh-meister @yidong72 @jbaczek @jzi040941 @Davood-M @tango4j @artbataev @sanandaraj5597 @vysarge @xrennvidia @zhehuaichen @shanmugamr1992 @trias702 @Faith-Nchifor @martin0258 @orena1 @sjmikler @xuanzic @HuiyingLi @bene-ges @shan18 @pzelasko @Agoniii @stephenmcconnachie @tbartley94 @terrykong @michal2409 @itzsimpl @pablo-garay @Zhuoyao1012 @mwawrzos @chiachihchen @JRD971000 @harishankar-gopalan @layalir @hainan-xv @akoumpa @mgrafu @krishnacpuvvada @jbieniusiewi @andrusenkoau
    129 people committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    94bd346 View commit details
    Browse the repository at this point in the history

  4. Update README.md 

    Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52
    stevehuang52 committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    8afd277 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. update speechllm (NVIDIA#8486) 

    * fix(clustering_diarizer.py): fix typo (#7772)

Signed-off-by: Jean-Louis Queguiner <jean-louis.queguiner@gadz.org>

* fix(diarization-README): typo (#7771)

Signed-off-by: Jean-Louis Queguiner <jean-louis.queguiner@gadz.org>

* Fix bug wrt change decoding strategy for bpe models (#7762) (#7764)

* Fix bug wrt change decoding strategy for bpe models



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove incorrect extra argument for load_from_checkpoint_dir() (#7500)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add nemo to mcore GPT conversion script  (#7730)

* add conversion script

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove references to 'ckpt'

Signed-off-by: Chen Cui <chcui@nvidia.com>

* add one more sanity check to make sure there is no unexpected keys in state dict

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* make cpu loading work

Signed-off-by: Chen Cui <chcui@nvidia.com>

* make script work for llama2 models

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address code check

Signed-off-by: Chen Cui <chcui@nvidia.com>

* remove trainer precision (was for old sanity check)

Signed-off-by: Chen Cui <chcui@nvidia.com>

* fix script for llama2 model

Signed-off-by: Chen Cui <chcui@nvidia.com>

* remove commented code

Signed-off-by: Chen Cui <chcui@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix bug in ConditionalInput: cat along the feature dim, not the batch dim (#7785)

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* Add some docs and update scripts for ASR (#7790)

* Add some docs and update scripts

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* set context for text memmap to fork (#7784)

* set context for text memmap to fork

Signed-off-by: arendu <adithyare@nvidia.com>

* typo

Signed-off-by: arendu <adithyare@nvidia.com>

---------

Signed-off-by: arendu <adithyare@nvidia.com>

* add training with multiple audios

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Support flash decoding (#7744)

* Add flash-decoding

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>

* Change accelerator to 'auto' in nlp_checkpoint_port.py (#7761)

* Change accelerator to 'auto' in nlp_checkpoint_port.py (#7747)

* Change accelerator to auto

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Pass omegaconf object to trainer in nlp_checkpoint_port.py

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Pass omegaconf object to trainer in export.py

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* docs: fix typos (#7758)

Signed-off-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Snake act (#7736)

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Update gpt_dataset.py (#6963)

Signed-off-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Signed-off-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: shuoer86 <129674997+shuoer86@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xin Yao <yaox12@outlook.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Add selection criteria for reference audios in the `GlobalStyleToken` submodule (#7788)

* add selection criteria for reference audios

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* Update configuration files

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* add informative comment in config files

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* sample random index for reference audio selection

Signed-off-by: anferico <f.cariaggi4@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: anferico <f.cariaggi4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update text server to support compute logprobs (#7733)

* update text server to support compute logprobs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix typo

---------

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add multi-layer feat extract and fix random question insertion

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Configure MCore logger (#7781)

Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>

* Revert "PEFT eval fix (#7626) (#7638)" (#7693)

This reverts commit f03dd660bd26d88fd569e76c6f74b83a7c203ff9.

* remove TN from ctc_segm tut (#7807)

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [TTS] Support audio offsets in TTS data loaders (#7156)

* [TTS] Support audio offsets in TTS data loaders

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Change docstring mentions of .pt to .npy

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* Update Apex install command in Dockerfile (#7794) (#7804)

* move core install to /workspace (#7706)



* update apex install in dockerfile



* use fetch head



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix typo

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Nemo to HF converter for LLaMA model (#7770)

* Create config_llama_truncate.yaml

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Add files via upload

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update config_llama_truncate.yaml

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update convert_nemo_llama_to_hf.py

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>

* clean up trainer

* remove dependency on yaml config. load config from nemo file instead.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* enable ckpt saving into other precision formats

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* support 70b + cleanup qkv slice logic

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug

* move hf model folder code from comment to function and add instruction to run

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Utkarsh <49331882+uppalutkarsh@users.noreply.github.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>

* Save best NeMo model only when necessary (#7836)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* add guard if its a distributed checkpoint (#7845)

Signed-off-by: Gerald Shen <geshen@nvidia.com>

* Fix tn duplex (#7808)

* fix duplex tn infer

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix typo

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix TN docs

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update transformers cache on Jenkins (#7854)

* update transformers cache

Signed-off-by: eharper <eharper@nvidia.com>

* update

Signed-off-by: eharper <eharper@nvidia.com>

* add cd

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>

* Update README.rst for container update (#7844)

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

* Add support for finetuning with huggingface datasets (#7834)

* add finetune with huggingface dataset

Signed-off-by: stevehuang52 <heh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update yaml

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update and refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add extrac hf text and update

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update and refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* move dataset dependency to common

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add docstring

Signed-off-by: stevehuang52 <heh@nvidia.com>

* Add to Dics

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add ci test

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add max steps in jenkins

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* reduce max steps

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* jenkins test

Signed-off-by: Nithin Rao Koluguri <nithinraok>

* add bs=2

Signed-off-by: Nithin Rao Koluguri <nithinraok>

---------

Signed-off-by: stevehuang52 <heh@nvidia.com>
Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>

* Multimodal merge (#7728)

* ControlNet TRT export

* Final MR before release

* SD2 update

* Fixed export issue

* Fix for instruct p2p and reformat

* Fix SD export issue

* Add nemo clip export for DB

* Fix ins pix2pix

* fix sd2 config

* [Mingyuan Ma] BF16 and SD conversion script

* [Imagen] NHWC Feature

* Fix .nemo loading issue for NeMo CLIP in SD

* NeMo r1.20.0 Multimodal Merge

* fix the inductor issue in inference

* Fix inductor loading .nemo issue

* Add Neva Model Support

* Imagen Optimizations

* Neva inference code

* NeMo TOT 1.21 to Internal/main

* Update neva_inference.yaml

* REBASING  for latest code changes

* Update internal/main to main tot

* Parallel DDIM implementation

* 1. Fixing indentation bug. (#7352)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* NeMo MCore llama2 support + MCore PEFT adapters (#7299)

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* add TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* add todo

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* remove import

Signed-off-by: ericharper <complex451@gmail.com>

* small clean up

Signed-off-by: ericharper <complex451@gmail.com>

* update hidden size in peft base model, add mcore commit to jenkins

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* add config obj to flash attention tests

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove sequence parallel arg

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to test

Signed-off-by: ericharper <complex451@gmail.com>

* get hidden_size from config

Signed-off-by: ericharper <complex451@gmail.com>

* add try except

Signed-off-by: ericharper <complex451@gmail.com>

* use default

Signed-off-by: ericharper <complex451@gmail.com>

* update config with hidden size

Signed-off-by: ericharper <complex451@gmail.com>

* remove arg

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* comment out jenkins test

Signed-off-by: ericharper <complex451@gmail.com>

* revert import

Signed-off-by: ericharper <complex451@gmail.com>

* build transformer config

Signed-off-by: ericharper <complex451@gmail.com>

* add model to provider func

Signed-off-by: ericharper <complex451@gmail.com>

* update forward and float16 wrapper

Signed-off-by: ericharper <complex451@gmail.com>

* instantiate model parallel config after init model parallel

Signed-off-by: ericharper <complex451@gmail.com>

* set virtual rank

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add GQA config to megatron gpt model (#7096)

* Add GQA config in gpt config file

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Verify mcore is enabled when using GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* mcore llama2 ckpt conversion & small fix

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Add inference & sft config by Hongbin

Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add inference param. update TP/PP script to support mcore gpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* modify ckpt conversion script (adding model cast)

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ckpt conversion use relative path for config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* set vp size to none if it is 1

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* add todo

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove import

Signed-off-by: ericharper <complex451@gmail.com>

* small clean up

Signed-off-by: ericharper <complex451@gmail.com>

* update hidden size in peft base model, add mcore commit to jenkins

Signed-off-by: ericharper <complex451@gmail.com>

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add config obj to flash attention tests

Signed-off-by: ericharper <complex451@gmail.com>

* remove args

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove sequence parallel arg

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to test

Signed-off-by: ericharper <complex451@gmail.com>

* get hidden_size from config

Signed-off-by: ericharper <complex451@gmail.com>

* add try except

Signed-off-by: ericharper <complex451@gmail.com>

* use default

Signed-off-by: ericharper <complex451@gmail.com>

* update config with hidden size

Signed-off-by: ericharper <complex451@gmail.com>

* remove arg

Signed-off-by: ericharper <complex451@gmail.com>

* comment out jenkins test

Signed-off-by: ericharper <complex451@gmail.com>

* revert import

Signed-off-by: ericharper <complex451@gmail.com>

* remove optimizer_idx

Signed-off-by: eharper <eharper@nvidia.com>

* prefetch num microbatches

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* start adding gpt from megatron core path

Signed-off-by: ericharper <complex451@gmail.com>

* set model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* use model parallel config object

Signed-off-by: ericharper <complex451@gmail.com>

* update args

Signed-off-by: ericharper <complex451@gmail.com>

* fix for p-tuning sequence parallel

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support SFT/distOpt mcore (#7207)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* change layer names for SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug in SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* start updating to TransformerConfig

Signed-off-by: ericharper <complex451@gmail.com>

* revert to model parallel config

Signed-off-by: ericharper <complex451@gmail.com>

* add hidden_size to model_parallel_config

Signed-off-by: ericharper <complex451@gmail.com>

* remove imports

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update module args

Signed-off-by: ericharper <complex451@gmail.com>

* add config to self

Signed-off-by: ericharper <complex451@gmail.com>

* build transformer config

Signed-off-by: ericharper <complex451@gmail.com>

* add model to provider func

Signed-off-by: ericharper <complex451@gmail.com>

* update forward and float16 wrapper

Signed-off-by: ericharper <complex451@gmail.com>

* instantiate model parallel config after init model parallel

Signed-off-by: ericharper <complex451@gmail.com>

* set virtual rank

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add GQA config to megatron gpt model (#7096)

* Add GQA config in gpt config file

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Verify mcore is enabled when using GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert

Signed-off-by: ericharper <complex451@gmail.com>

* remove import

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rollback model cast for p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update for dist adam

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use get_gpt_module_list

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ptl2.0 patch for llama config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add plugins to trainer in scripts

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix activation checkpointing mcore

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix variable names

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* overwrite normalization type for mcore/te

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Update megatron_llama_sft.yaml

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* add PEFT adapter support for mcore gpt path (#7276)

* implementation for mcore adapter/mxins

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* small fix for lora and ptuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support layerwise peft

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support multiple target layers

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support lora GQA

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support amp O2

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* revert & more O2 fix

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lora inject to attention

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support lora weight tying

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add copyright header

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback ptuning name change. full string match mcore target

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove comment

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* clean up config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* Sync llama branch (#7297)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* change layer names for SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug in SFT

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug: cpu initialization is not really enabled

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* add use_cpu_initialization to TransformerConfig

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* fix bug: wrong config path when using relative cjpt path

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

* revert mcore config change

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

* clean up ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback git merge errors

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update mcore, add check for mcore+te

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* formatting

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* make sft test dataset optional. fix indentation in config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* one more fix for optional test set

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support merging lora weights in mcore

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update mcore for cpu init

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ckpt conversion for code llama

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add seq_len_interpolation_factor support for long-context llama ckpts (#7312)

* add inference param. update TP/PP script to support mcore gpt

* p-tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add seq_len_interpolation_factor

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>

* fix old ptuning model, update mcore to support seq_len_interpolation_factor

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support fused layernorm linear, fix ptuning O2

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* drop loss mask for mcore for now

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* disable dist ckpt in peft

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix loading non dist ckpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add ckpt conversion to CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* update CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* mcore_mixin docstring

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor change in mcore peft error message

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* fix amp o2 in lora weight tying

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* correct mcore fp8 config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add TE installation

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* support mcore adapter tuning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* comment out new CI test. rollback docker image

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* ignore FA tests, try new CI on 23.08

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* mark new CI as L2, put to beginning to test

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor fix for prompt learning

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* rollback to 23.06. comment out CI

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* minor fix ckpt conversion script

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor rollback gpt model change

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: jasonwan <jasonwan@nvidia.com>
Signed-off-by: eharper <eharper@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: eharper <eharper@nvidia.com>
Co-authored-by: Hongbin Liu <hongbinl@nvidia.com>
Co-authored-by: Kelvin Liu <lhb8125@users.noreply.github.com>

* Hiddens modules documentation (#7303)

* 1. Changed hiddens transformations module from `transformations` to `hiddens`.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Finished doc.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Support for flash attention 2.0 (#7063)

* Add flash attn 2

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add FA2 feature

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove debugging

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Signed-off-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* lora merge fix for O2 names (#7325)

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* adjust key names based on O2

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* minor

Signed-off-by: arendu <adithyare@nvidia.com>

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* multiple fields can form a context (#7147)

* list of context fields and flexible prompt template

Signed-off-by: arendu <adithya.r@gmail.com>

* list of fields for context

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Add multiple truncation fields and middle truncation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Compatible to old ckpt

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix tokenize detokenize issue

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove detokenization, add truncation augmentation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve comments

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove unused import

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert eos

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Add tokenizer space_sensitive attribute

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Fix erorr and use re

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Change assert logic

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow adi suggestion

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove merge function

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add example and comment

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove context_key and add comment

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* Remove random truncation

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix template none

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Signed-off-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>

* Load buffers in checkpoint (#7357)

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* Add migration guide for lightning 2.0 upgrade (#7360)

* Add lightning 2.0 migration guide in NeMo docs

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add remaining guide for lightning 2.0 upgrade

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Remove line spill over and continue in next line

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add missing dataloader_iter in the guide

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Fix minor typo

Signed-off-by: Abhishree <abhishreetm@gmail.com>

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* adding bias_dropout_add_fusion option for BERT (#7332)

Signed-off-by: Alexander Jipa <azzhipa@amazon.com>
Co-authored-by: Alexander Jipa <azzhipa@amazon.com>

* [TTS] Change audio codec token type to TokenIndex (#7356)

Signed-off-by: Ryan <rlangman@nvidia.com>

* enable selective unfreeze (#7326)

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* avoid PTL method conflicts

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix typos (#7361)

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typos

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* fix typo

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

---------

Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>

* pin numba=0.57.1 to fix reinstall.sh error (#7366)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update new conversion script for converting safetensors.

* Upgrade pytorch container to 23.08 (#7353)

* upgrade pytorch container

Signed-off-by: eharper <eharper@nvidia.com>

* use mcore

Signed-off-by: eharper <eharper@nvidia.com>

* revert test change

Signed-off-by: eharper <eharper@nvidia.com>

* pleasefixme

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check for ampere

Signed-off-by: eharper <eharper@nvidia.com>

* comment test temporarily

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* enable fp32 optimizer for output_layer in mcore (#7355)

Signed-off-by: lhb8125 <lhb8125@gmail.com>

* revert comment (#7368)

Signed-off-by: eharper <eharper@nvidia.com>

* Update to core 23.08 branch ToT (#7371)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* upper bounding ptl (#7370)

Signed-off-by: eharper <eharper@nvidia.com>

* fix pipeline parallel inference (#7367)

* fix pp inference

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix for peft tied weights (#7372)

Signed-off-by: arendu <adithyare@nvidia.com>

* fixed trainer.strategy=auto from None. (#7369)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* add O2 option in gpt eval (#7358)

* add O2 option in eval

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add doc for O2 config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* add to llama inference config

Signed-off-by: jasonwan <jasonwan@nvidia.com>

---------

Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Move model precision copy (#7336)

* move cfg precision set to megatron base model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* remove copy from other models

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* modify attribute not arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* fix gpt model test for ptl 2.0

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* rename function and add docstring

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* replace precision to dtype conditionals with func call

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* unnecessary function and cfg reset

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* set default value

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* fix precision lookup in a few more places

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* rename mapping function

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* ununsed import

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* save torch datatype to model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* set weights precision wrt amp o2

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Revert "set weights precision wrt amp o2"

This reverts commit 313a4bfe5eb69d771a6d2433898c0685836aef5c.

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* revert half precision at inference attempt

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move autocast dtype to base model

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move params dtype to base model, enable fp16 O2 inf

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* unused imports

Signed-off-by: Maanu Grover <maanug@nvidia.com>

---------

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Fix PEFT checkpoint loading (#7388)

* Fix PEFT checkpoint loading

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use distributed optimizer support for multiple dtypes (#7359)

* Update distopt wrapper with multiple dtype support

Remove manual handling of separate FP32 optimizer.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Use distopt support for contiguous buffers with multiple dtypes

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix typo

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Separate distopt buckets for first GPT layer and non-overlapped params

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Add distopt logic for int dtypes

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Apex commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Remove unused variables

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Apex commit in README and Jenkensfile

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug Dockerfile and Jenkinsfile

Signed-off-by: Tim Moon <tmoon@nvidia.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* minor fix for llama ckpt conversion script (#7387)

* minor fix for llama ckpt conversion script

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* Update Jenkinsfile

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* remove fast_swiglu configuration

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix wrong calling of librosa.get_duration() in notebook (#7376)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* [PATCH] PEFT import mcore (#7393)

* [PATCH] PEFT import mcore

Signed-off-by: Jason Wang <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jason Wang <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Added a callback for logging initial data (#7384)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Update Core Commit (#7402)

* Update Core Commit

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* update commit

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Use cfg attribute in bert (#7394)

* use cfg attribute instead of arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* use torch_dtype in place of cfg.precision

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* move precision copy before super constructor

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* use trainer arg

Signed-off-by: Maanu Grover <maanug@nvidia.com>

---------

Signed-off-by: Maanu Grover <maanug@nvidia.com>

* Add support for bias conversion in Swiglu models (#7386)

* Add support for bias conversion in Swiglu models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for auto extracting tokenizer model

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add support for auto extracting tokenizer model

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix issue with missing tokenizer

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update save_to and restore_from for dist checkpointing (#7343)

* add dist ckpt to save to, in progress

Signed-off-by: eharper <eharper@nvidia.com>

* move dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update restore from, need to figure out how to initialize distributed

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* launch distrib if needed when restoring dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* when using mcore we can change tp pp on the fly

Signed-off-by: eharper <eharper@nvidia.com>

* add load_from_checkpoint support for dist ckpt

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update llama convert script to save dist .nemo

Signed-off-by: eharper <eharper@nvidia.com>

* fix load dist ckpt

Signed-off-by: jasonwan <jasonwan@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* setup TE TP groups if needed

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* setup te tp groups if needed

Signed-off-by: eharper <eharper@nvidia.com>

* remove import

Signed-off-by: eharper <eharper@nvidia.com>

---------

Signed-off-by: eharper <eharper@nvidia.com>
Signed-off-by: jasonwan <jasonwan@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: jasonwan <jasonwan@nvidia.com>

* fix forward for with mcore=false (#7403)

Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Co-authored-by: Jimmy Zhang <jiemingz@nvidia.com>

* Fix logging to remove 's/it' from progress bar in Megatron models and add train_step_timing (#7374)

* Add CustomProgressBar class to exp_manager and trainer callbacks

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix the progress bar to reflect total microbatch cnt

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Modify CustomProgressBar class

1) Modify CustomProgressBar class to update progress bar per global_step instead of per microbatch
2) Add the callback to other megatron training/finetuning files that are not using MegatronTrainerBuilder

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Add CustomProgressBar callback to tuning files

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Set Activation Checkpointing Defaults (#7404)

* Set Activation Checkpointing Defaults

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check for None

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* make loss mask default to false (#7407)

Signed-off-by: eharper <eharper@nvidia.com>

* Add dummy userbuffer config files (#7408)

Signed-off-by: Sangkug Lym <slym@nvidia.com>

* add missing ubconf files (#7412)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* New tutorial on Speech Data Explorer (#7405)

* Added Google Colab based tutorial on Speech Data Explorer 

Signed-off-by: George Zelenfroynd <gzelenfroind@nvidia.com>

* Update ptl training ckpt conversion script to work with dist ckpt (#7416)

* update ptl convert script

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* don't break legacy

Signed-off-by: eharper <eharper@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: eharper <eharper@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Allow disabling sanity checking when num_sanity_val_steps=0 (#7413)

* Allow disabling sanity checking when num_sanity_val_steps=0

Signed-off-by: Abhishree <abhishreetm@gmail.com>

* Update num_sanity_val_steps to be a multiple of num_microbatches

Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add comprehensive error messages (#7261)

Signed-off-by: Anton Peganov <apeganov@nvidia.com>

* check NEMO_PATH (#7418)

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* layer selection for ia3 (#7417)

* layer selection for ia3

Signed-off-by: arendu <adithyare@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithyare@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix missing pip package 'einops' (#7397)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Fix failure of pyaudio in Google Colab (#7396)

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update README.md: output_path --> output_manifest_filepath (#7442)

Signed-off-by: Samuele Cornell <cornellsamuele@gmail.com>

* Updating FlashAttention API to match FlashAttentionV2

* Multiple fixes for mm

* Fix CI inductor issue and update to torch compile

* Remove suppress error

* Fix when conversion config uses fp16 and it complains about precision plugin

* Fixing FAv2 API usage

* Initial release of content filtering model

* Added synthetic dataloader for precached and online mode

* Mingyuanm/dreambooth opt

* Add llama2 support in neva training

* Fix sampler length

* Fix all precision issues in nemo multimodal

* Add rope dynamic linear scaling (#7437)

* Add dynamic linear scaling

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>

* Fix None dataloader issue in PTL2.0 (#7455)

* Fix None dataloader issue in PTL2.0

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updating values of self._validation_dl and self._test_dl as well

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updating values of self._validation_dl and self._test_dl as well

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [ASR] Confidence measure -> method renames (#7434)

* measure -> method

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add steps for document of getting dataset 'SF Bilingual Speech' (#7378)

* Add steps for document of getting dataset 'SF Bilingual Speech'

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update datasets.rst

added a link from a tutorial demonstrating detailed data prep steps.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* RNN-T confidence and alignment bugfix (#7381)

* new frame_confidence and alignments lists are now always created after the while loop

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* tests added

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

---------

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* Fix resume from checkpoint in exp_manager (#7424) (#7426)

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix checking of cuda/cpu device for inputs of Decoder (#7444)

* Fix checking of cuda/cpu device for inputs of Decoder

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* Update tacotron2.py

Signed-off-by: Jason <jasoli@nvidia.com>

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Signed-off-by: Jason <jasoli@nvidia.com>
Co-authored-by: Jason <jasoli@nvidia.com>

* Fix failure of ljspeech's get_data.py (#7430)

* Fix failure of ljspeech's get_data.py

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Robin Dong <robin.k.dong@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Fix audio codec type checks (#7373)

* [TTS] Fix audio codec type checks

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix audio codec tests

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Add dataset to path of logged artifacts (#7462)

* [TTS] Add dataset to path of logged artifacts

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Revert axis name back to Audio Frames

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* Fix sft dataset truncation (#7464)

* Add fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>

---------

Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Automatic Lip Reading Recognition (ALR) - ASR/CV (Visual ASR) (#7330)

* striding_conv1d_k5 and dw_striding_conv1d_k5 subsampling

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* transpose conv1d inputs

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* Update subsampling.py

change striding_conv1d_k5 to striding_conv1d

Signed-off-by: Maxime Burchi <60737204+burchim@users.noreply.github.com>

* cv branch

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* video manifest

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* add collection classes

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test_step_outputs

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* correct manifest bug when having only audio or only videos

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* correct manifest bug when having only audio or only videos

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* clean references

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* freeze unfreeze transcribe cv models

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* correct manifest get_full_path bug

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* update for PR

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* guard torchvision

Signed-off-by: mburchi <maxime.burchi@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update nemo/collections/cv/data/video_to_text_dataset.py

Co-aut…
    @stevehuang52 @jqueguiner
    @github-actions @titu1994 @pre-commit-ci @RobinDong @ericharper @cuichenx @anferico @arendu @hsiehjackson @yzhang123 @athitten @shuoer86 @XuesongYang @nithinraok @yaox12 @MaximumEntropy @Zhilin123 @mikolajblaz @ekmb @rlangman @aklife97 @uppalutkarsh @anteju @gshennvm @fayejf @yaoyu-33 @Victor49152 @milesial @suiyoubi @meatybobby @maanug-nv @michalivne @blahBlahhhJ @lhb8125 @okuchaiev @azzhipa @omahs @timmoon10 @JimmyZhang12 @jiemingz @erhoo82 @Jorjeous @PeganovAnton @karpnv @popcornell @hXl3s @KunalDhawan @GNroy @blisc @burchim @Kipok @janekl @The0nix @stas00 @redoctopus @GiacomoLeoneMaria @odelalleau @marcromeyn @hkelly33 @BestJuly @erastorgueva-nv @dimapihtar @menon92 @ahmadki @ssh-meister @yidong72 @jbaczek @jzi040941 @Davood-M @tango4j @artbataev @sanandaraj5597 @vysarge @xrennvidia @zhehuaichen @shanmugamr1992 @trias702 @Faith-Nchifor @martin0258 @orena1 @sjmikler @xuanzic @HuiyingLi @bene-ges @shan18 @pzelasko @Agoniii @stephenmcconnachie @tbartley94 @terrykong @michal2409 @itzsimpl @pablo-garay @Zhuoyao1012 @mwawrzos @chiachihchen @JRD971000 @harishankar-gopalan @layalir @hainan-xv @akoumpa @mgrafu @krishnacpuvvada @jbieniusiewi @andrusenkoau
    129 people committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    78c1e8e View commit details
    Browse the repository at this point in the history

  2. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    2e74cd1 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. for now bypass asr_model init in perception since that causes issues … 

    …in tp=2

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    95ee03c View commit details
    Browse the repository at this point in the history

  2. update doc and infer 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    5ff28a1 View commit details
    Browse the repository at this point in the history

  3. https://github.com/NVIDIA/NeMo/pull/8464/files 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    80f7439 View commit details
    Browse the repository at this point in the history

  4. update doc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    e1e825f View commit details
    Browse the repository at this point in the history

  5. update doc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    99fb448 View commit details
    Browse the repository at this point in the history

  6. update doc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    446c6d9 View commit details
    Browse the repository at this point in the history

  7. update doc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    3d78dd7 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. add a debug script 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    70ed539 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. support text-only training and speech and text joint training 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    13f03a2 View commit details
    Browse the repository at this point in the history

  2. always require text only data has question field in the data and use it 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    c0260c6 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. support prepend_to_exist_question 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    85ba4f6 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. support random_context_prob 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    fc185d5 View commit details
    Browse the repository at this point in the history

  2. apply random_context_prob for w/ and w/o canary 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    3247583 View commit details
    Browse the repository at this point in the history

  3. guard random context 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    268bb70 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. protect the case where answer is empty 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    9d69f2e View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. fix for ++model.pretrained_canary_model=$ASR_MODEL 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    b812226 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. support unfreeze_emb 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    ae9adf0 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. minor update 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    0916850 View commit details
    Browse the repository at this point in the history

  2. fix import 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    db542b4 View commit details
    Browse the repository at this point in the history

  3. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    fe7214b View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. support t5 + lhotse 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    b133332 View commit details
    Browse the repository at this point in the history

  2. add xattn 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    3f5fd1b View commit details
    Browse the repository at this point in the history

  3. CrossAttendModularizedAudioT5Model is WIP and replaced by audio_promp… 

    …t_first=False

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    2291706 View commit details
    Browse the repository at this point in the history

  4. support distributed adam 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    e6cdebf View commit details
    Browse the repository at this point in the history

  5. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    916324e View commit details
    Browse the repository at this point in the history

  6. fix pretrained info 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    98f86b5 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. support with_distributed_adam 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    2876d41 View commit details
    Browse the repository at this point in the history

  2. fix distributed adam 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    cf6deb2 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. add local_batch_size 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    b8fc008 View commit details
    Browse the repository at this point in the history

  2. support mt5 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    b6ef6a5 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into heh/modular_speechllm_pr 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    555a007 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    8f524e3 View commit details
    Browse the repository at this point in the history

  5. update dockerfile 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    619d75d View commit details
    Browse the repository at this point in the history

  6. support mt5 and bypass bos_id=-1 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    c0b9d0c View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2024

  1. support configurating legacy_tokenizer for mt5 models 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 23, 2024
    Configuration menu
    Copy the full SHA
    9a4861b View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. update for merging main 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    c3ca938 View commit details
    Browse the repository at this point in the history

  2. fix for merge main 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    76db149 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    f7afea1 View commit details
    Browse the repository at this point in the history

  4. clean up docs 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    c99ad43 View commit details
    Browse the repository at this point in the history

  5. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    7c9ded7 View commit details
    Browse the repository at this point in the history

  6. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    4c4ac20 View commit details
    Browse the repository at this point in the history

  7. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    afbc212 View commit details
    Browse the repository at this point in the history

  8. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    6bce450 View commit details
    Browse the repository at this point in the history

  9. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    b3f6156 View commit details
    Browse the repository at this point in the history

  10. update 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    f63b8b8 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    9dd72b6 View commit details
    Browse the repository at this point in the history

  2. fix speechlm test 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    11facc7 View commit details
    Browse the repository at this point in the history

  3. update doc 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    3da8282 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    179fafd View commit details
    Browse the repository at this point in the history

  5. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    14c1334 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    98a0143 View commit details
    Browse the repository at this point in the history

  2. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    7dbe84d View commit details
    Browse the repository at this point in the history

  3. fix multi-layer feat 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    3a039f5 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    55c9e04 View commit details
    Browse the repository at this point in the history

  5. update for webdataset 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    073212b View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. support setting dropout and label smoothing 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    edcf401 View commit details
    Browse the repository at this point in the history

  2. make sure the updated cfg is passed to frozen_model 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    d3a04e0 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2024

  1. mv model paths 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    3762632 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    ba86fb9 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. force str to avoid bugs with implicit conversion of str to bool type 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    fdfe7b5 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Update examples/multimodal/speech_llm/README.md 

    Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52 @nithinraok
    stevehuang52 and nithinraok committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    18b2921 View commit details
    Browse the repository at this point in the history

  2. Update examples/multimodal/speech_llm/README.md 

    Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52 @nithinraok
    stevehuang52 and nithinraok committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    fef24dc View commit details
    Browse the repository at this point in the history

  3. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    c532150 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'heh/modular_speechllm_pr' of https://github.com/NVIDIA/… 

    …NeMo into heh/modular_speechllm_pr

Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    21d4261 View commit details
    Browse the repository at this point in the history

  5. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    c2f6b78 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'canary_speechllm1_cross_t5_pr' into canary_speechllm1_c… 

    …ross_t5_pr2

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    7744144 View commit details
    Browse the repository at this point in the history

  7. update for saving nemo 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    647e184 View commit details
    Browse the repository at this point in the history

  8. update eval and ngc ckpt 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    36df825 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Update nemo/collections/multimodal/speech_llm/data/audio_text_qa_data… 

    …set.py

Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52 @nithinraok
    stevehuang52 and nithinraok committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    f6a90d1 View commit details
    Browse the repository at this point in the history

  2. Update nemo/collections/multimodal/speech_llm/modules/common/audio_te… 

    …xt_generation_utils.py

Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52 @nithinraok
    stevehuang52 and nithinraok committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    d73a684 View commit details
    Browse the repository at this point in the history

  3. Update tests/collections/multimodal/test_speechllm_models.py 

    Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
    @stevehuang52 @nithinraok
    stevehuang52 and nithinraok committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    3dea3ce View commit details
    Browse the repository at this point in the history

  4. refactor and remove nlp adapter mixin assert 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    aa4f85b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'heh/modular_speechllm_pr' of https://github.com/NVIDIA/… 

    …NeMo into heh/modular_speechllm_pr
    @stevehuang52
    stevehuang52 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    9e10694 View commit details
    Browse the repository at this point in the history

  6. remove random context augmentation 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    360acd4 View commit details
    Browse the repository at this point in the history

  7. fix docstring 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    6449924 View commit details
    Browse the repository at this point in the history

  8. add docstring 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    52617f9 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. minor refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    7c78165 View commit details
    Browse the repository at this point in the history

  2. refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    ed29843 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. fixes to be compatible with 24.01 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    19b3d9f View commit details
    Browse the repository at this point in the history

  2. refactor and fix missing import 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    5a4be92 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. fix for unfreeze llm 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    03b9e60 View commit details
    Browse the repository at this point in the history

  2. for unfreeze am 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    35f0b03 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into heh/modular_speechllm_pr

    @pablo-garay
    pablo-garay committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c991e5b View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. major refactor on input format and minor update 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    79156fc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'heh/modular_speechllm_pr' of https://github.com/NVIDIA/… 

    …NeMo into heh/modular_speechllm_pr
    @stevehuang52
    stevehuang52 committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    0268898 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. fix codeQL 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    b6cac3d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    8b19dc5 View commit details
    Browse the repository at this point in the history

  3. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    960f958 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. fix for canary prompt 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    fac3a4e View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. fix for canary prompt and support t5 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    89f0a42 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'canary_speechllm1_cross_t5_pr2' of github.com:zhehuaich… 

    …en/NeMo into canary_speechllm1_cross_t5_pr2
    @zhehuaichen
    zhehuaichen committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    790359d View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Merge remote-tracking branch 'origin/main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    2e18366 View commit details
    Browse the repository at this point in the history

  2. configurable random_context_positive_percent 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    2bf9b07 View commit details
    Browse the repository at this point in the history

  3. update default random_context_num to 8 to reduce seq len 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    566ee5a View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. inference support 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    f5e4af3 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. support TP>1 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 1, 2024
    Configuration menu
    Copy the full SHA
    a63e35d View commit details
    Browse the repository at this point in the history

Commits on May 5, 2024

  1. fix for salm decode 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 5, 2024
    Configuration menu
    Copy the full SHA
    e0f5189 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    8043262 View commit details
    Browse the repository at this point in the history

  2. update for NGC ckpt and refactor 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    55f8231 View commit details
    Browse the repository at this point in the history

  3. clean up 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    d9e2788 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. support output metainfo with audio_filepath 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    60843db View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into heh/modular_speechllm_pr

    @ericharper
    ericharper committed May 7, 2024
    Configuration menu
    Copy the full SHA
    3cd12e9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into heh/modular_speechllm_pr

    @ericharper
    ericharper committed May 7, 2024
    Configuration menu
    Copy the full SHA
    30a583a View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Merge remote-tracking branch 'upstream/heh/modular_speechllm_pr' into… 

    … canary_speechllm1_cross_t5_pr2

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b9fc1bc View commit details
    Browse the repository at this point in the history

  2. revert unrelated changes 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    e4cad0c View commit details
    Browse the repository at this point in the history

  3. revert unrelated changes 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    226c605 View commit details
    Browse the repository at this point in the history

  4. some fixes for t5 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    f9e2f94 View commit details
    Browse the repository at this point in the history

  5. clean up and test inference 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    d4a6fd8 View commit details
    Browse the repository at this point in the history

  6. move dataset code to one place 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    90887cf View commit details
    Browse the repository at this point in the history

  7. verify train and inference for bestow+gpt and salm+t5 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 8, 2024
    Configuration menu
    Copy the full SHA
    d5265bd View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into heh/modular_speechllm_pr

    @nithinraok
    nithinraok committed May 8, 2024
    Configuration menu
    Copy the full SHA
    55b270b View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    1c4cbd7 View commit details
    Browse the repository at this point in the history

  2. skip speechlm test until data moved to CI machines 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    3e88457 View commit details
    Browse the repository at this point in the history

  3. use pad_id for pad and add eos_id when enabled 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0700cdb View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    17ab55b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    6cae145 View commit details
    Browse the repository at this point in the history

  3. refactor and update to avoid changing nlp_adapter_mixin 

    Signed-off-by: stevehuang52 <heh@nvidia.com>
    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    4cfaa30 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'heh/modular_speechllm_pr' of https://github.com/NVIDIA/… 

    …NeMo into heh/modular_speechllm_pr
    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    27e33ee View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into heh/modular_speechllm_pr

    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    67ecaa1 View commit details
    Browse the repository at this point in the history

  6. Apply isort and black reformatting 

    Signed-off-by: stevehuang52 <stevehuang52@users.noreply.github.com>
    @stevehuang52
    stevehuang52 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    89926fa View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. Merge remote-tracking branch 'upstream/heh/modular_speechllm_pr' into… 

    … canary_speechllm1_cross_t5_pr3

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 11, 2024
    Configuration menu
    Copy the full SHA
    fb8914d View commit details
    Browse the repository at this point in the history

  2. minor edit 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 11, 2024
    Configuration menu
    Copy the full SHA
    9499f2e View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into canary_speechllm1_c… 

    …ross_t5_pr3

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 11, 2024
    Configuration menu
    Copy the full SHA
    e601135 View commit details
    Browse the repository at this point in the history

  4. Apply isort and black reformatting 

    Signed-off-by: zhehuaichen <zhehuaichen@users.noreply.github.com>
    @zhehuaichen
    zhehuaichen committed May 11, 2024
    Configuration menu
    Copy the full SHA
    3cc0432 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. fixes per Piotr and Steve's comments 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed May 28, 2024
    Configuration menu
    Copy the full SHA
    11407b2 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. WIP in getting rid of canary specific things in dataset 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    db2166e View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into canary_speechllm1_c… 

    …ross_t5_pr3
    @zhehuaichen
    zhehuaichen committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    d6c23a5 View commit details
    Browse the repository at this point in the history

  3. remove canary specific design; bugfix for asr/models/aed_multitask_mo… 

    …dels.py

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    9c2c4af View commit details
    Browse the repository at this point in the history

  4. remove random_context and submit it later by rewriting with augmenter 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    24c0f9f View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. remove canary specific stuffs in dataloading; use input_cfg in lhotse… 

    … to support context

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1999298 View commit details
    Browse the repository at this point in the history

  2. fix for NVIDIA#9169 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1f10bd7 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into canary_speechllm1_c… 

    …ross_t5_pr3
    @zhehuaichen
    zhehuaichen committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e96da9b View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. minor fix 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    0aa4179 View commit details
    Browse the repository at this point in the history

  2. make sure NGC inference and fix CodeQL https://github.com/NVIDIA/NeMo… 

    …/pull/9169/checks?check_run_id=25818322332

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    24daa2e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into canary_speechllm1_cross_t5_pr3

    @zhehuaichen
    zhehuaichen committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    1239b35 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. add back the assert in nlp collection and add a enforce_divisible_bat… 

    …ch flag

Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    eb7e00d View commit details
    Browse the repository at this point in the history

  2. nit 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    e4e7802 View commit details
    Browse the repository at this point in the history

  3. fixes per Som s comments NVIDIA#9169 (review) 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    0b4451b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into canary_speechllm1_cross_t5_pr3

    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    9d362e5 View commit details
    Browse the repository at this point in the history

  5. nit 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    f12fa74 View commit details
    Browse the repository at this point in the history

  6. fix split_list 

    Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com>
    @zhehuaichen
    zhehuaichen committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    18bdbe7 View commit details
    Browse the repository at this point in the history