Add NeVA_Mixtral Tutorial (with new NeVA features) #9912

* Nemotron ONNX export fixed Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Cleanup Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Addressing code review comments Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com>

* Docker cleanup

Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com>

* add slurm files to .gitignore * add differentiable decode to SDXL VAE * Optionally return predicted noise during the single step sampling process * also change `get_gamma` as a new function to use inside other functions which may interact with sampling (e.g. draft+) * debugging sdunet converter script * Added SD/SDXL conversion script from HF to NeMo * added 'from_nemo' config for VAE * tmp commit, please make changes (oci is super slow, cannot even run vim) * new inference yaml works * add logging to autoencoder * !(dont squash) Added enabling support for LinearWrapper for SDLoRA * added samples_per_batch and fsdp arguments to SDXL inference * added extra optionally wrapper to FSDP * remove unncessary comments * remove unnecessary comments * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: Rohit Jena <rohitkumarj@nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>

* add NemoQueryLLMPyTorch class for triton query of in-framework models * nemo_export.py changes to better support in-framework models * separate out in-framework version of triton deploy script * add generate() function to MegatronLLMDeployable to allow for direct use in export tests * use NemoQueryLLMPyTorch in deploy tests * add warning message for when MegatronLLMDeployable overrides transformer_engine * remove enable_streaming argument from deploy_inframework_triton.py since MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments * Apply isort and black reformatting Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> * skip trtllm support check if in_framework testing * remove unused imports * run_existing_checkpoints was passing wrong prompts argument for in-framework mode * fix unused import in query_inframework.py --------- Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

* Use FP8 in GPT TP2 test Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Add hydra options to use TE, TP overlap and FP8 Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Override presence checks in hydra Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * WIP: Add debug code Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbaczek <jbaczek@users.noreply.github.com> * Add more debug code Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbaczek <jbaczek@users.noreply.github.com> * Add more debug code Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbaczek <jbaczek@users.noreply.github.com> * Remove debug code and change underlying transformer layer to TE Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Override hydra error Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Remove tp overlap from the test Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Change runner for fp8 tests Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * fix Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Add tp overlap test Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Remove TP overlap from tests. It is unsupported in docker environment Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Adjust GPT PP2 test to use FP8. Change optimizer in TP2 test Signed-off-by: Jan Baczek <jbaczek@nvidia.com> * Remove env overrides form GPT PP2 test Signed-off-by: Jan Baczek <jbaczek@nvidia.com> --------- Signed-off-by: Jan Baczek <jbaczek@nvidia.com> Signed-off-by: jbaczek <jbaczek@users.noreply.github.com> Co-authored-by: jbaczek <jbaczek@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com>

…variety of tensors (NVIDIA#9641) * enables default data step in megatron parallel to operate on a wider variety of tensors coming out of the dataloader * handles the case where a batch is empty * Apply isort and black reformatting Signed-off-by: jomitchellnv <jomitchellnv@users.noreply.github.com> * Allows the default data step to operate on more types than just dictionaries Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> --------- Signed-off-by: jomitchellnv <jomitchellnv@users.noreply.github.com> Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> Co-authored-by: jomitchellnv <jomitchellnv@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com>

…a wider …" (NVIDIA#9666)

* wip contrastive reranker Signed-off-by: arendu <adithya.r@gmail.com> * wip Signed-off-by: arendu <adithya.r@gmail.com> * wip Signed-off-by: arendu <adithya.r@gmail.com> * working reranker training and validation Signed-off-by: arendu <adithya.r@gmail.com> * default peft for reranker Signed-off-by: arendu <adithya.r@gmail.com> * validation time update Signed-off-by: arendu <adithya.r@gmail.com> * reranker test Signed-off-by: arendu <adithya.r@gmail.com> * reranker inference Signed-off-by: arendu <adithya.r@gmail.com> * reranker inference Signed-off-by: arendu <adithya.r@gmail.com> * Apply isort and black reformatting Signed-off-by: arendu <arendu@users.noreply.github.com> * updates Signed-off-by: arendu <adithya.r@gmail.com> * Apply isort and black reformatting Signed-off-by: arendu <arendu@users.noreply.github.com> * updates Signed-off-by: arendu <adithya.r@gmail.com> * Apply isort and black reformatting Signed-off-by: arendu <arendu@users.noreply.github.com> * also can support rlhf style reward model loss Signed-off-by: arendu <adithya.r@gmail.com> * Apply isort and black reformatting Signed-off-by: arendu <arendu@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: arendu <arendu@users.noreply.github.com> * typo in cicd Signed-off-by: arendu <adithya.r@gmail.com> --------- Signed-off-by: arendu <adithya.r@gmail.com> Signed-off-by: arendu <arendu@users.noreply.github.com> Signed-off-by: Adi Renduchintala <adithya.r@gmail.com> Co-authored-by: arendu <arendu@users.noreply.github.com>

* unpin transformers Signed-off-by: dimapihtar <dpihtar@gmail.com> * guard deprecated imports Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * fix import guards Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix import guards Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * try fixing Signed-off-by: Chen Cui <chcui@nvidia.com> * disable HF tests Signed-off-by: Dmytro Pykhtar <dpykhtar@login-eos01.eos.clusters.nvidia.com> * try fixing Signed-off-by: Chen Cui <chcui@nvidia.com> * hard code model lists Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * hard code model lists Signed-off-by: Chen Cui <chcui@nvidia.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@login-eos01.eos.clusters.nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Dmytro Pykhtar <dpykhtar@login-eos01.eos.clusters.nvidia.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com>

* Added CPU offloading docs Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> * Tech writer review Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> --------- Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Co-authored-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

* Update llama-3 PEFT notebook to download model from NGC Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Fix broken link in llama-3 PEFT tutorial README Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Fix broken code block in llama 3 PEFT tutorial README Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Copy-edits to Llama-3 8B PEFT tutorial README Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Fix broken link Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Minor formatting fixes Signed-off-by: Shashank Verma <shashank3959@gmail.com> --------- Signed-off-by: Shashank Verma <shashank3959@gmail.com>

Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: ashors1 <ashors@nvidia.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

* Parametrize FPS group * Apply isort and black reformatting * Change deafult to False * Add logic to new ckptIO * Turn on parallel save by default --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

* huvu/mcore_t5 first commit from local * removing DEBUGGING prints * cleaning megatron_lm_encoder_decoder_model.py code * cleaning code * adding Github action test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * reset .github/workflows/cicd-main.yml * reset .github/workflows/cicd-main.yml * adding condition self.mcore_t5 when running self.build_transformer_config() * refractor megatron_lm_encoder_decoder_model.py to not use self.model * only run T5-related tests * remove all self.model * reset cicd file * reset cicd file * updating codes remove duplicate if/else; adding mcore/transformer_engine to config file * adjust +model.mcore_t5=True * fix training for non-mcore, bf16, O2 * reset cicd-main.yml --------- Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

* adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * add mamba dependancies * add mcore tag * modify dockerfile ci * modify dockerfile ci * fix TP>1 to TP1 * add inference, update based on latest mcore commits * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * minor fix * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * minor fix * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * bug fix, tutorial update --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com>

Signed-off-by: Ryan <rlangman@nvidia.com>

* commit to eval/sft/peft * update MCORE_COMMIT * address Chen's comments, updating retro unit test * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> --------- Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

…IDIA#9715) * Allow non-strict load * Point to non-stric load MCore branch * Avoid module level StrictHandling * Use MCore fork * Update to MCore fix * Restore ackward compatibility * Update flag defaults * Update MCore tag * Update PyT Dist interface * Update to latest core_r0.8.0 --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

* fix legacy ds padding bug Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * avoid code repetition Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix typo Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com>

…variety of tensors - second try (NVIDIA#9671) * enables default data step in megatron parallel to operate on a wider variety of tensors coming out of the dataloader Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> * handles the case where a batch is empty Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> * Apply isort and black reformatting Signed-off-by: jomitchellnv <jomitchellnv@users.noreply.github.com> Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> * Allows the default data step to operate on more types than just dictionaries Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> * Apply isort and black reformatting Signed-off-by: jomitchellnv <jomitchellnv@users.noreply.github.com> --------- Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com> Signed-off-by: jomitchellnv <jomitchellnv@users.noreply.github.com> Co-authored-by: jomitchellnv <jomitchellnv@users.noreply.github.com> Co-authored-by: John St. John <jstjohn@users.noreply.github.com>

…A#9647) * Fix when optimizers are setup for PEFT * Apply isort and black reformatting * Init DDP inside PEFT * Apply isort and black reformatting * Some fixes, loss seems to become nan with peft for some reason * Apply isort and black reformatting * Loss goes down on fp32 * Apply isort and black reformatting * Simplifying FNMixin * Apply isort and black reformatting * Fix bug with new checkpoint-io * Apply isort and black reformatting * Fix failing test: test_peft_on_train_epoch_start_with_adapter * Apply isort and black reformatting --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: ashors1 <ashors@nvidia.com>

* refactor: README * refactor: Use new README in `setup.py` Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

* Remove mask if use fusion mask Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> * Apply isort and black reformatting Signed-off-by: hsiehjackson <hsiehjackson@users.noreply.github.com> --------- Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> Signed-off-by: hsiehjackson <hsiehjackson@users.noreply.github.com> Co-authored-by: hsiehjackson <hsiehjackson@users.noreply.github.com>

…DIA#9690) (NVIDIA#9694) * Move tensorstore import inline * Moving AsyncFinalizableCheckpointIO import inline * Wrap AsyncCompatibleCheckpointIO in try/catch inside pl.py * Moving gpt_layer_specs import inline * Apply isort and black reformatting --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com>

* add contianer * modify tutorial * modify tutorial * modify tutorial --------- Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com>

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>

…#9650) (NVIDIA#9691) * Nemotron export - fixing megatron_export.py (NVIDIA#9625) * Nemotron ONNX export fixed * Cleanup * Addressing code review comments --------- * Including all trainable-params in a PEFT-checkpoint * Apply isort and black reformatting * Small fixes to make model-importer work * Fixing failing tests --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: ashors1 <ashors@nvidia.com>

* [NeMo-UX] Make TE and Apex dependencies optional (NVIDIA#9550) * Provide a pure pytorch/jit path to avoid required dependency on TE and Apex Signed-off-by: ashors1 <ashors@nvidia.com> * add missing file Signed-off-by: ashors1 <ashors@nvidia.com> * add minimal gpt pretraining example Signed-off-by: ashors1 <ashors@nvidia.com> * fix pre-training datamodule initialization Signed-off-by: ashors1 <ashors@nvidia.com> * add non-te/non-apex test Signed-off-by: ashors1 <ashors@nvidia.com> * add comment to pretraining script Signed-off-by: ashors1 <ashors@nvidia.com> * use microbatch calculator from mcore Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix nemo 2 test name Signed-off-by: ashors1 <ashors@nvidia.com> * update Mcore commit for CI Signed-off-by: ashors1 <ashors@nvidia.com> * replace apex microbatch calculator with megatron's in more places Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix missing import Signed-off-by: ashors1 <ashors@nvidia.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * fix missed apex import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> * move imports Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add types to command-line args Signed-off-by: ashors1 <ashors@nvidia.com> * bug fix Signed-off-by: ashors1 <ashors@nvidia.com> * fix path Signed-off-by: ashors1 <ashors@nvidia.com> * Disable distributed optimizer in nemo 2.0 test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix optimizer config Signed-off-by: ashors1 <ashors@nvidia.com> * update checkpointing Signed-off-by: ashors1 <ashors@nvidia.com> * move import Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix failing unit test Signed-off-by: ashors1 <ashors@nvidia.com> * fix failing test Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * Updating num_weights check of RETRO due to underlying changes from mcore RETRO MLM Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix typo Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * remove stale warning Signed-off-by: ashors1 <ashors@nvidia.com> * fix lora notebook Signed-off-by: ashors1 <ashors@nvidia.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * add import guards to gemma2 Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> * fix cherry-pick Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: huvunvidia <86480512+huvunvidia@users.noreply.github.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com>

* minor 2.0 bug fix when TE/Apex not installed Signed-off-by: ashors1 <ashors@nvidia.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com>

Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Co-authored-by: ashors1 <ashors@nvidia.com>

…v variable (NVIDIA#9736) (NVIDIA#9750) Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Fix issue with prompt_defaults Signed-off-by: smajumdar <titu1994@gmail.com> * Add core level support for grad map tracking Signed-off-by: smajumdar <titu1994@gmail.com> * Add core level support for grad map tracking Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Add tutorial and update repr of formatters Signed-off-by: smajumdar <titu1994@gmail.com> * Update docs Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com>

…al_batch_size (NVIDIA#9707) (NVIDIA#9753) Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

* fix serialization of partial function * update serialization to handle value.args Signed-off-by: srabhi <srabhi@nvidia.com> * add unit test Signed-off-by: srabhi <srabhi@nvidia.com> * remove redundant code from unit-test Signed-off-by: srabhi <srabhi@nvidia.com> --------- Signed-off-by: srabhi <srabhi@nvidia.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

…or (NVIDIA#9682) * Speeds up copying of neccesary artifact files with SaveRestoreConnector Previously, the SaveRestoreConnector would copy and untar entire checkpoints just to copy out a tokenizer. For models in the >100GB, this led to timeouts since only rank=0 did this work, while other ranks moved on and waited at an all-gather barrier (observed NCCL timeout at 10min). Signed-off-by: Terry Kong <terryk@nvidia.com> * cleanup Signed-off-by: Terry Kong <terryk@nvidia.com> * black formatting Signed-off-by: Terry Kong <terryk@nvidia.com> * Apply isort and black reformatting Signed-off-by: terrykong <terrykong@users.noreply.github.com> Signed-off-by: Terry Kong <terryk@nvidia.com> * restoring logic to previous tempdir logic Signed-off-by: Terry Kong <terryk@nvidia.com> * nlp overrides too Signed-off-by: Terry Kong <terryk@nvidia.com> * respect return_config Signed-off-by: Terry Kong <terryk@nvidia.com> * some unit tests Signed-off-by: Terry Kong <terryk@nvidia.com> * nodbg Signed-off-by: Terry Kong <terryk@nvidia.com> * Apply isort and black reformatting Signed-off-by: terrykong <terrykong@users.noreply.github.com> * correct typing Signed-off-by: Terry Kong <terryk@nvidia.com> * Fixes directory issue Signed-off-by: Terry Kong <terryk@nvidia.com> * Apply isort and black reformatting Signed-off-by: terrykong <terrykong@users.noreply.github.com> --------- Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: terrykong <terrykong@users.noreply.github.com> Co-authored-by: terrykong <terrykong@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com>

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

Signed-off-by: ashors1 <ashors@nvidia.com>

* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (NVIDIA#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (NVIDIA#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com>

* ci: Add workflow for code-freeze Signed-off-by: Oliver Koenig <okoenig@nvidia.com> * ci: Add workflow for releasing NeMo Tookit Signed-off-by: Oliver Koenig <okoenig@nvidia.com> --------- Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

* Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com>

This fixes the template such that collapsable sections are properly rendered. Signed-off-by: Oliver Koenig <okoenig@nvidia.com>

…9548) * Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com>

) * leave space for generating tokens Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> * Apply isort and black reformatting Signed-off-by: hsiehjackson <hsiehjackson@users.noreply.github.com> * Refactoring Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> * Update comment Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> * Apply isort and black reformatting Signed-off-by: hsiehjackson <hsiehjackson@users.noreply.github.com> --------- Signed-off-by: Cheng-Ping Hsieh <chsieh@nvidia.com> Signed-off-by: hsiehjackson <hsiehjackson@users.noreply.github.com> Co-authored-by: hsiehjackson <hsiehjackson@users.noreply.github.com>

* Updated the vLLM integration for version 0.5.1. * Updated the vllm version in requirements_vllm.txt. * Apply isort and black reformatting --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Alexey Panteleev <alpanteleev@nvidia.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

* minor fix tutorial * minor fix tutorial --------- Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com>

* Adding context- & expert-parallism to MegatronStrategy (#9525) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add CICD test for Stable Diffusion (#9464) * Add CICD test for Stable Diffusion Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Update cicd-main.yml Signed-off-by: Michal Futrega <mfutrega@nvidia.com> * Use single gpu runner Signed-off-by: Michal Futrega <mfutrega@nvidia.com> --------- Signed-off-by: Michal Futrega <mfutrega@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/nemo ux mixtral (#9446) * use default collate if dataset does not have one Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mixtral config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add convert_state Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix StateDictTransform for 2D layers, e.g. MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass num_moe_experts to specs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * udpate MixtralModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mini docstring Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * update mcoreddp call (#9345) * update mcoreddp call Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update mcore commits Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Llama and Gemma (#9528) * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add llama3 Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix typo Signed-off-by: Chen Cui <chcui@nvidia.com> * enable importers with multiple models Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * add gemma Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * checks Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] minor logging bug fixes (#9529) * minor exp_manager bug fixes * remove print statement * fix docstring * fix AppState defaults --------- Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mcore distOpt restore fix (#9421) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Custom Tiktoken tokenizer. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fixed the tokenizer decoding on special tokens. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Apply isort and black reformatting Signed-off-by: ertkonuk <ertkonuk@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Added token_to_id() method. Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update neva conversion script from and to HF (#9296) * Update NeMo script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix example scripts Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * Update convert_llava_nemo_to_hf.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Support (#9381) * Export implementation for vLLM 0.4.3. Supports LLAMA2, Mistral, Mixtral (unverified), Gemma and StarCoder2 models. The nemo.export.tensorrt_llm alias was removed to avoid initializing TRT-LLM when importing anything from nemo.export. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed some CodeQL warnings. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Removed empty files. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Updated the integration for vLLM 0.5.0. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Updated the vLLM deployment interface to use max_output_len instead of max_output_token. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the Exporter class to nemo/export and renamed its file to vllm_exporter.py, to be more similar to TRT-LLM. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Implemented vLLM support in the export tests, added functional testing, implemented forward evaluation on vLLM without Triton. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Moved the vLLM deployment functionality to the common deploy_triton.py script. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed the CodeQL discovered issues. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Fixed one more return of a wrong dimensionality... Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * More wrong dimensionality returns. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PL: Delete precision if using plugin. TODO switch to MegatronTrainerBuilder (#9535) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add page context fmha (#9526) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * extend get_gpt_layer_modelopt_spec to support MoE (#9532) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix mock data generation for legacy dataset (#9530) Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] IO fixes (#9512) * Improve IOMixin.io_transform_args to handle dataclasses better * Dump task json + img inside NeMoLogger * Adding store_io to train task * Update opt.connect to also propagate to __io__ * Rename opt to optim for consistency * Moving to using safe serialization using fiddle, only use cloudpickle when needed * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Using Config from fiddle instead of sdk for now * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move enable_nemo_ckpt_io from MegatronStrategy to ModelCheckpoint * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move nemo-ckpt to _get_finalize_save_checkpoint_callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Update TrainerContext & io.load_ckpt * Use renamed TrainerContext inside ModelCheckpoint * Remove double io saving * Rename lightning.pytorch.opt -> optim * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove store_io from train-task * Adding fiddle-extension for torch * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Move fdl_torch import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding dtype to serialization * Some fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Make TransformerConfig inherit from IOMixin to fix serialization error * Make TransformerConfig inherit from IOMixin to fix serialization error * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add support for BuiltinFunctionType * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Add missing import * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix dataclass fields --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Test C++ runtime on demand in nemo_export.py to avoid possible OOMs (#9544) * Add test_cpp_runtime flag Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix lhotse tests for v1.24.2 (#9546) * Fix lhotse tests for v1.24.0 Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Fix RIR test Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * gpu_unitTests_notOptional (#9551) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * add reset learning rate functionality (#9372) * add reset_lr functionality Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * move reset_lr from optim section Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * add reset_lr value to config Signed-off-by: dimapihtar <dpihtar@gmail.com> * set reset_lr False by default Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra line Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * add reset_lr test Signed-off-by: dimapihtar <dpihtar@gmail.com> * remove extra quote Signed-off-by: dimapihtar <dpihtar@gmail.com> * add ability to reset schedule's max_steps and decay_steps Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * change scheduler's first step logic when using reset_lr Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix reset_lr logic Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * update reset_lr comments Signed-off-by: dimapihtar <dpihtar@gmail.com> * add use cases for reset_lr feature Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Python AIStore SDK to container and bump min Lhotse version (#9537) * Add Python AIStore SDK to requirements and bump min Lhotse version Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Move AIStore Python SDK to Dockerfile, remove matplotlib/ipywidgets deps Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() (#9147) * Ininial WARs to implement dynamo option for export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * including weights in .onnx Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * dynamo_export works for many small models Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * External weights behaviour fixed Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Cleanup Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Apply isort and black reformatting Signed-off-by: borisfom <borisfom@users.noreply.github.com> * print cleaned up Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Added overloadable dynamic_shapes_for_export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Addressing code review Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI issues Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Fixing CI test failure Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> * Eliminated test cross-contamination Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> --------- Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com> Signed-off-by: borisfom <borisfom@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix tokenizer IO (#9555) * Adding tokenizer to io-test + making it pass * Handling tokenizer correctly inside dump_io * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Removing not used import --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo UX] Move mistral_7b.py to mistral.py (#9545) * Move mistral_7b.py to mistral.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename MixtralConfig to MixtralConfig8x7B Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * mistral rename: mistralconfig7b & mistralmodel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Use closed-formula to round by multiple (#9307) * Use closed-formula to round by multiple Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * ci: Do not attempt to send slack on fork (#9556) * ci: Do not attempt to send slack on fork Signed-off-by: Oliver Koenig <okoenig@nvidia.com> * test Signed-off-by: Oliver Koenig <okoenig@nvidia.com> --------- Signed-off-by: Oliver Koenig <okoenig@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix nemo export test (#9547) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix SDXL incorrect name in docs (#9534) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * GPU unit tests: Mark flaky tests to be fixed (#9559) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * switch to torch_dist as default dist checkpointing backend (#9541) Signed-off-by: ashors1 <ashors@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Checkpointing bug fixes (#9562) * fix checkpoint loading * fix * fixes * another fix * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add tps and pps params to the export script (#9558) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix export test Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove n_gpus param Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add and fix parameters Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * fix deploy script Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * rename tps and pps params Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Consolidate gpt continue training script into pretraining script (#9413) * Consolidate gpt continue training with pretraining Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix default config Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add github action cicd Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * extract _integrate_original_checkpoint_data as a method Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix getattr Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Revert "Add github action cicd" This reverts commit a453f16ba2be6413db932623009da893208acdd5. * Update comments in nlp_overrides.py Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: yaoyu-33 <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add support to change Multi task model prompt (#9542) * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Add support to change Multi task model prompt Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Update nemo/collections/common/prompts/formatter.py Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Address comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add Multimodal Exporter (#9256) * Add video-neva TRT export * Add TRT inference * Change config * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change export params * Remove unused import * Add neva export * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change unpack nemo * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add trt infer config * Fix neva trt inference * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Add exporter * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix infer * Add PyTriton * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix deploy wrong dim * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Change to pass PIL Image * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix video neva deploy * Change query * Change deploy * Remove unused import * Change ptuning * Change to mm exporter * Add script * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> * Fix script --------- Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> Co-authored-by: meatybobby <meatybobby@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable encoder adapters for Canary and MultiTaskAED models (#9409) * Fix assertions for adapter types Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Cleanup Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Finalize support for decoder adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * fix the freeze/unfreeze problem by replacing as_frozen with torch.inference_mode * Apply isort and black reformatting Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> * Update tests to new generic way of module update Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize code for update module Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Fix variable name Signed-off-by: smajumdar <titu1994@gmail.com> * Finalize projection support for transformer mha adapters Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Correct implementation of freeze restore Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Corrects the implementation of replace_adapter_modules to limit to just the top level modules Signed-off-by: smajumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Remove registration of Transformer MHA Signed-off-by: smajumdar <titu1994@gmail.com> * Address reviewer comments Signed-off-by: smajumdar <titu1994@gmail.com> --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Co-authored-by: Weiqing Wang <weiqingw@nvidia.com> Co-authored-by: weiqingw4ng <weiqingw4ng@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * pass option through (#9570) Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * PTQ refinements (#9574) * Rename megatron_gpt_quantization -> megatron_gpt_ptq Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Configure export.save_path as dir or tarball Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * PTQ docs update Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Make model_type optional in case of quantized checkpoints Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Drop unused save_nemo_model_config argument Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Audio model collection (#9263) * Audio model collection Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> * Fix imports Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Addressed PR comments Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Apply isort and black reformatting Signed-off-by: anteju <anteju@users.noreply.github.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com> Signed-off-by: anteju <anteju@users.noreply.github.com> Co-authored-by: anteju <anteju@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Fix Trainer serialization (#9571) * Fix Trainer serialization * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update click version requirement (#9580) Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Fault tolerance] Heartbeat detection (#9352) * Fault tolerance related changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Cosmetic changes in documentation Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Doc update round2 Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add ModelOpt QAT example for Llama2 SFT model (#9326) * add INT4 QAT example for Llama2 SFT model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Add config parameter to control kv cache quantization Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * Fix typo in cicd-main.yml for QAT test Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * fix nlp_overrides.py Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * address reviewer feedback Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * quantize unwrapped model Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> * add compress export argument for qat config Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set TE flag in legacy -> mcore conversion script (#9585) * set TE flag Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add fabric-API for manual forward-pass (#9577) * First pass over fabric-API * Adding Trainer -> Fabric conversion * Some small fixes to get a forward-pass in Fabric working * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding doc-string to Fabric.import_model * Adding track_io to io_init of Fabric * Fix Fabric.load_model + add doc-string * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove unused import * Some small fixes * Fix failing test --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Add SDK-factories to llm-collection (#9589) * Adding sdk-factories to llm-collection * Removing _model from mistral + mixtral * Expose lr_scheduler inside lightning * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Multimodal projection layer adapter fix for PP>1 (#9445) * enabling multimodal adapters to load in PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * parameterizing validate_access_integrity, set to false when PP>1 Signed-off-by: paul-gibbons <paul@gibbonspaul.com> formatting fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update nlp_model.py Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * update modelPT with validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating save_restore_connector w/ validate_access_integrity Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * addressing comment Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adding validate_access_integrity to super().load_config_and_state_dict() Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * testing reorder of validate_access_integrity for CI failures Signed-off-by: paul-gibbons <paul@gibbonspaul.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Add offline quantization script for QLoRA deployment (#9455) * add qlora offline quantization script Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * clean Signed-off-by: Chen Cui <chcui@nvidia.com> * docstring Signed-off-by: Chen Cui <chcui@nvidia.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * qlora support more models (#9488) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Some improvements to NeMoLogger (#9591) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set n_gpu to None in nemo export (#9593) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * set ngpus to None Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Inflight nemo model export support (#9527) * online model conversion and refit Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * clean code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * add refit, cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * combine weight conversion functions Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> * remove debug print Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * fix single gpu and cleanup code Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com> * Apply isort and black reformatting Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> --------- Signed-off-by: JimmyZhang12 <JimmyZhang12@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * vLLM Export Improvements (#9596) * Separated the vLLM export functionality from the common deployment script into deploy_vllm_triton.py. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Fixed vocab_size for LLAMA3. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Export test: fixed deployment testing w/o Megatron, made functional tests optional, added --gpu_memory_utilization. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> * Apply isort and black reformatting Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> * Addressing review and CodeQL comments. Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> --------- Signed-off-by: Alexey Panteleev <alpanteleev@nvidia.com> Signed-off-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: apanteleev <apanteleev@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set finalize_model_grads_func in on_fit_start instead to make sure it's being called (#9599) Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Set no_sync_func & grad_sync_fucn (#9601) * Set no_sync_func & grad_sync_fucn Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * set overlap_param_sync Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * small nemo logger bug fix (#9607) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix the dict format returned by scheduler method (#9609) Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Dataloading enhancements and bug fixes (#9595) * fix dataloading + checkpoint restore * clean up data sampler * fix typo * support passing multiple paths to data module * fix validation dataloader * fix dataloader len when using gradient accumulation * fix progress bar * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix step count in loggers * fix blended dataset * address comments * address comment * move step logging into strategy * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix serialization of AutoResume (#9616) * fix serialization of autoresume * update undefined variables Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Chat template support for megatron_gpt_eval.py (#9354) * Bump PTL version (#9557) Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [Resiliency] Straggler detection (#9473) * Initial straggler det impl Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed CI code checks Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Removed unused import Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * remove submodule Signed-off-by: Maanu Grover <maanug@nvidia.com> * Updated documentation; Updated callback params; Cosmetic changes Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixed straggler det config; Added basic test Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * Fixes in test_straggler_det.py Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Updated straggler callback API Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> * Apply isort and black reformatting Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> * stop_if_detected=False by default Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> --------- Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move model loading to separate function; call toContainer once; pad using closed formula Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * read prompts from file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * If input prompt contains dict, apply model.tokenizer.chat_template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * apply @Gal Leibovich's patch Taken from: https://github.com/NVIDIA/NeMo/commit/17572905344db4692583e72799d55801a8860f35 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename prompts_file to prompts_jsonl Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat_template param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add ChatTemplateMixin to SentencePieceTokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat-template to text-gen-strat Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move load prompts to separate file Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove chat-template from text-gen-utils Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make chat-template more generic Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add assert message Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor for chat_template_mixin Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * undo ckpt conv changes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rounding to function Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Jacek Bieniusiewicz <jbieniusiewi@nvidia.com> Signed-off-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: jbieniusiewi <152396322+jbieniusiewi@users.noreply.github.com> Co-authored-by: jbieniusiewi <jbieniusiewi@users.noreply.github.com> Co-authored-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Jsonl support (#9611) * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Adding support to preprocess .jsonl and .jsonl.gz files in input directory Signed-off-by: adityavavre <avavre@nvidia.com> * Apply isort and black reformatting Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> --------- Signed-off-by: adityavavre <avavre@nvidia.com> Signed-off-by: adityavavre <adityavavre@users.noreply.github.com> Co-authored-by: adityavavre <adityavavre@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] Add PEFT (#9490) * initial commit for PEFT in nemo2 * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * make import easier Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * address comments Signed-off-by: Chen Cui <chcui@nvidia.com> * Update nemo/collections/llm/peft/lora.py Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Some small fixes + adding more doc-strings * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Adding ModelTransform callback * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing type-hint for model_transform * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * fix import Signed-off-by: Chen Cui <chcui@nvidia.com> * model transform for gemma llama Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * fix model transform Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * change lora target default to all linear modules Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * Small fix in mixtral * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Integrating PEFT to the public-API + some fixes * Big refactor to allow to load adapter-states * Some fixes to support adapter_path * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Disabling ckpt reloading when adapter_path is passed * Fix CLI * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Remove commented-out code * Remove commented-out code * Remove un-used import * Fix callback imports * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fixing llm.pretrain * Some small fixes * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix missing import + type-hint in finetune * Adding PreemptionCallback + some more tests * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Clean up imports & clean up llm.api * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Trying to fix failing tests * Remove __init__.py 2 * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Fix failing test * Trying to fix last failing test --------- Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Akoumparouli/mistral import instruct chat template fix (#9567) * use bf16 by defualt mistral conv Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add chat template Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use capitalized role names Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Remove .cuda calls, use device isntead (#9602) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix converter defautl args (#9565) * fix converter defautl args Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * mixtral export (#9603) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix: remove non_blocking from PTL's .cuda call (#9618) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba tmp (#9612) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add mamba_tmp * remove mamba import * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * TitaNet Batch Verify Speaker (#9337) * add batch_inference for verify_speakers method Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * remove not used package Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change batch inference logic Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * fixup Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * requested changes Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add verify_speakers_batch to docs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * handle None durations in manifest Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * change logging text Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * Apply isort and black reformatting Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> * check duration presence Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> * add channel_selector to dataset configs Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> --------- Signed-off-by: msekoyan@nvidia.com <msekoyan@nvidia.com> Signed-off-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: monica-sekoyan <monica-sekoyan@users.noreply.github.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Enable MCore checkpointing optimizations (#9505) * Expose num processes in PyT Dist Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add parallel save/load optimizations from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Remove async utils from MCore Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Enable PyT Dist caching Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Small fixes Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Make sure DistCkptIO is instantiated from config Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Bump MCore version to v0.7 Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Print load strategy Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Forward MCore to model space DistOpt Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add separate flag to control DistOpt paralell R/W Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Turn off parallel save by default Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change mixtral moe key name for trt-llm (#9620) * fix minor import bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * change moe key values Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add weight to the key Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix ckpt load bug (#9621) * fix ckpt load bug Signed-off-by: dimapihtar <dpihtar@gmail.com> * Apply isort and black reformatting Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: dimapihtar <dimapihtar@users.noreply.github.com> Co-authored-by: dimapihtar <dimapihtar@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * NeVA Minor Fixes (#9608) * fix neva resume with empty param loaded for some pp stage Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix crop size check Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * fix pretrianing data sizes and weights (#9627) Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Alit/mamba (#9575) * adding mamba support * fix import mixins * rm convert jamba * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * more cleanups * use GPT text gen * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * fixing gbs in TP convetor * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * add reqs * add tutorial * minor fix to tutorial * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * moving finetuning files Signed-off-by: arendu <adithya.r@gmail.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> * address comments * add mamba dependancies * add mcore tag * modify dockerfile ci * modify dockerfile ci --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Signed-off-by: arendu <adithya.r@gmail.com> Co-authored-by: Ali Taghibakhshi <ataghibakhsh@login-eos01.eos.clusters.nvidia.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: arendu <adithya.r@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [NeMo-UX] async checkpointing support (#9466) * add async checkpointing support * fixes * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * add parallel read/write support and other optimizations * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * address comments, make dist checkpointing args configurable * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> * fix small typo Signed-off-by: ashors1 <ashors@nvidia.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Update default sharding type Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: ashors1 <ashors1@users.noreply.github.com> --------- Signed-off-by: ashors1 <ashors1@users.noreply.github.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Anna Shors <71393111+ashors1@users.noreply.github.com> Co-authored-by: ashors1 <ashors1@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Fix the arguments of forward_for_export function in msdd_models (#9624) * Fix the arguments of forward_for_export function Signed-off-by: Taejin Park <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: Taejin Park <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Change default parallel_save to False (#9632) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Unwrap ckpt_io for model opt (async save) (#9622) Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * MCore T5 support for NeMo - Training (#9432) * huvu/mcore_t5 first commit from local * removing DEBUGGING prints * cleaning megatron_lm_encoder_decoder_model.py code * cleaning code * adding Github action test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * only run mcore T5 test * reset .github/workflows/cicd-main.yml * reset .github/workflows/cicd-main.yml * adding condition self.mcore_t5 when running self.build_transformer_config() * refractor megatron_lm_encoder_decoder_model.py to not use self.model * only run T5-related tests * remove all self.model * reset cicd file * reset cicd file * updating codes remove duplicate if/else; adding mcore/transformer_engine to config file * adjust +model.mcore_t5=True * Apply isort and black reformatting Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> --------- Signed-off-by: huvunvidia <huvunvidia@users.noreply.github.com> Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com> Co-authored-by: huvunvidia <huvunvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * [Nemo-UX] Expose transformer_layer_spec inside GPTConfig (#9592) * Expose transformer_layer_spec inside GPTConfig * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> * Expose layer-specs * Apply isort and black reformatting Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> --------- Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com> Co-authored-by: marcromeyn <marcromeyn@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com> * Update NeMo Clip to Use MCore Modules (#9594) * update clip model and config file Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update clip for mcore Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * MCore CLIP Fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix no mask Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * few neva fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip module Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add siglip loss Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix collate fn Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip conversion script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update siglip convert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * clean up script Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * clip fixe…

* typos and branch name update to r2.0.0rc1 (NVIDIA#9846) Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao Koluguri <nithinraok> * restoring main branch name Signed-off-by: Nithin Rao Koluguri <nithinraok> --------- Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: Nithin Rao Koluguri <nithinraok>

* Spectrogram transforms without torchaudio Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Use einops, added comments Signed-off-by: Ante Jukić <ajukic@nvidia.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com> Co-authored-by: Roman Korostik <roman.korostik@pm.me>

* add "offline" data cache generation support Signed-off-by: dimapihtar <dpihtar@gmail.com> * revert config Signed-off-by: dimapihtar <dpihtar@gmail.com> * add comment for data_cache_generation_only usage Signed-off-by: dimapihtar <dpihtar@gmail.com> * add num_dataset_builder_threads param Signed-off-by: dimapihtar <dpihtar@gmail.com> * fix comment Signed-off-by: dimapihtar <dpihtar@gmail.com> --------- Signed-off-by: dimapihtar <dpihtar@gmail.com>

…VIDIA#9808) * Fix clip model * fix neva loading due to fp8 change * Fix CLIP comment issues * update docs * Fix neva tutorial * Apply isort and black reformatting * fix siglip few things * Apply isort and black reformatting * Update convert_siglip_hf_to_nemo.py * fix comments --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>

Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slynedeng@gmail.com> Co-authored-by: slyne deng <slyned@nvidia.com>

* add docs Signed-off-by: stevehuang52 <heh@nvidia.com> * add lhotse specific info Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com> * move images to github release 1.23 Signed-off-by: stevehuang52 <heh@nvidia.com> * clean up Signed-off-by: stevehuang52 <heh@nvidia.com> --------- Signed-off-by: stevehuang52 <heh@nvidia.com> Signed-off-by: zhehuaichen <dian.chenzhehuai@gmail.com> Co-authored-by: zhehuaichen <dian.chenzhehuai@gmail.com>

* Move PubMedQA example to dedicated folder Signed-off-by: Shashank Verma <shashank3959@gmail.com> * Add llama 3.1 SDG LoRA and NIM deployment tutorial Signed-off-by: Shashank Verma <shashankv@nvidia.com> * Minor updates to Llama-3.1 PEFT tutorial README Signed-off-by: Shashank Verma <shashankv@nvidia.com> * Minor updates to Llama 3.1 PEFT tutorial README Signed-off-by: Shashank Verma <shashankv@nvidia.com> * Updates to NIM deployment instructions Signed-off-by: Shashank Verma <shashankv@nvidia.com> * Minor update to README --------- Signed-off-by: Shashank Verma <shashank3959@gmail.com> Signed-off-by: Shashank Verma <shashankv@nvidia.com>

* Metric with Squim Objective and MOS Signed-off-by: Ante Jukić <ajukic@nvidia.com> * Removed utility functions Signed-off-by: Ante Jukić <ajukic@nvidia.com> --------- Signed-off-by: Ante Jukić <ajukic@nvidia.com>

…se) (NVIDIA#9699) (NVIDIA#9827) * add dummy vision and text transformer config (assumed mcore to be false) * Apply isort and black reformatting --------- Signed-off-by: Rohit Jena <rohit.rango@gmail.com> Signed-off-by: rohitrango <rohitrango@users.noreply.github.com> Co-authored-by: Rohit Jena <rohitrango@users.noreply.github.com>

* adding japanese text preprocessing * japanese phoneme tokenizer * japanese tests * japanese g2p model * japanese word to ipa dictionary * add requirements Signed-off-by: Alex Cui <alexcui1994@gmail.com> --------- Signed-off-by: Alex Cui <alexcui1994@gmail.com> Signed-off-by: BuyuanCui <BuyuanCui@users.noreply.github.com> Co-authored-by: BuyuanCui <BuyuanCui@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

) * Query TransformerConfig attributes when copying btw configs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * test Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com>

* MoE docs Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * additional fixes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

* Update Huggingface Hub support * Update hf hub * Update hf hub * Apply isort and black reformatting --------- Signed-off-by: smajumdar <titu1994@gmail.com> Signed-off-by: Somshubra Majumdar <titu1994@gmail.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Make alignments tests work on any machine --------- Signed-off-by: Vladimir Bataev <vbataev@nvidia.com> Signed-off-by: artbataev <artbataev@users.noreply.github.com> Co-authored-by: artbataev <artbataev@users.noreply.github.com>

* Update arch check for SD Signed-off-by: Jaemin Choi <jaeminc@nvidia.com> * Apply isort and black reformatting Signed-off-by: minitu <minitu@users.noreply.github.com> --------- Signed-off-by: Jaemin Choi <jaeminc@nvidia.com> Signed-off-by: minitu <minitu@users.noreply.github.com> Co-authored-by: Jaemin Choi <jaeminc@nvidia.com> Co-authored-by: minitu <minitu@users.noreply.github.com>

* Revert "Jpg2p jun18 (NVIDIA#9538)" This reverts commit 53d7a91. * Apply isort and black reformatting Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> --------- Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: pablo-garay <pablo-garay@users.noreply.github.com>

* Revert "Jpg2p jun18 (NVIDIA#9538)" This reverts commit 53d7a91. * Need first jobs to succeed * Make failing jobs optional * Apply isort and black reformatting Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> --------- Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: pablo-garay <pablo-garay@users.noreply.github.com>

* Change decord to guard import * Apply isort and black reformatting Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> --------- Signed-off-by: meatybobby <meatybobby@users.noreply.github.com> Co-authored-by: meatybobby <meatybobby@users.noreply.github.com>

) Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>

* add nemo fundamentals page * remove unused reference tag * add link to checkpoints intro * clarify postprocessing and mention loss function * rephrase key parameters * fix typo * mention trainer accelerator param * fix bulletpoint formatting * fix bullet points part 2 * quick formatting fixes * fix phrasing * update based on review plus other small fixes --------- Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com> Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>

* Torch major and minor versions set to current year and month if YY.MM formatting is not met Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Update nvidia torch version check Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Remove redundant import Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> * Formatting fix Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> --------- Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com> Co-authored-by: Dong Hyuk Chang <donghyukc@nvidia.com>

* fix arg name Signed-off-by: Sangkug Lym <slym@nvidia.com> * cleanup Signed-off-by: Sangkug Lym <slym@nvidia.com> * cleanup Signed-off-by: Sangkug Lym <slym@nvidia.com> --------- Signed-off-by: Sangkug Lym <slym@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>

* Added defer wgrad support with mcore optim Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> * Apply isort and black reformatting Signed-off-by: sanandaraj5597 <sanandaraj5597@users.noreply.github.com> --------- Signed-off-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Signed-off-by: sanandaraj5597 <sanandaraj5597@users.noreply.github.com> Co-authored-by: Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> Co-authored-by: sanandaraj5597 <sanandaraj5597@users.noreply.github.com>

…, videoneva Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add NeVA_Mixtral Tutorial (with new NeVA features) #9912

Add NeVA_Mixtral Tutorial (with new NeVA features) #9912

Commits on Jul 8, 2024

Commits on Jul 9, 2024

Commits on Jul 10, 2024

Commits on Jul 11, 2024

Commits on Jul 12, 2024

Commits on Jul 15, 2024

Commits on Jul 16, 2024

Commits on Jul 17, 2024

Commits on Jul 18, 2024

Commits on Jul 19, 2024

Commits on Jul 22, 2024

Commits on Jul 23, 2024

Commits on Jul 24, 2024

Commits on Jul 25, 2024

Commits on Jul 26, 2024