add support to convert hf to nemo loras with tp > 1 #9303
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
* Add params like max_num_tokens and opt_num_tokens Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove padding param added * update params like max_num_token Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * remove context context_fmha param for now Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * add params like max num token to the script Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com>
* sum-reudce grad_norm in DP+CP domain Signed-off-by: Sangkug Lym <slym@nvidia.com> * Apply isort and black reformatting Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> --------- Signed-off-by: Sangkug Lym <slym@nvidia.com> Signed-off-by: pablo-garay <pablo-garay@users.noreply.github.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Co-authored-by: pablo-garay <pablo-garay@users.noreply.github.com>
* temp save Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * temp save 2 Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update code Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable seq packing Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix neva and clip Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Enable parallel seq packing algo and few other fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Pipeline parallel support Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update data preprocess Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix few pp issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * enable sequence packing w/ PP Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix cu_seqlens in inputs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * add assert Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Depend on PP to decide whether do padding Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add docstring Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix few PP evaluation issues Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add llama3 template Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix license Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix llama3 Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Few fixes Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Few neva bugs Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * llama3 inference fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Force vision encoder to run in fp32 Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Revert "Force vision encoder to run in fp32" This reverts commit 9d2160d. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Try adding distributed format of checkpoint Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow dist checkpoint to be non-strict Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Some fixes for PP + dist ckpt in Neva Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix peft Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * few fixes for lora Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * checkpoint updates Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * bug fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add neva dist checkpoint converter Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * resolve comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update neva dist ckpt apis Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Apply isort and black reformatting Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> * fix return Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>
* first commit * working pipeline rag_indexing; rag_eval with rag.yaml * udpate RAG documentation * add image to documents * cleaning docs * before merge from main * refactor code to make it easier to support more customized embedder and LLMs in future * addressing Ali's comments * addressing Ali's comments * addressing Ali's comments * fix Code scanning results / CodeQL --------- Co-authored-by: Huy Vu2 <huvu@login-eos01.eos.clusters.nvidia.com>
* update branch * pin --------- Signed-off-by: eharper <eharper@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com>
* Mcore dist opt ckpt fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass dp_zero_gather_scatter to starded-state-dict Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * introduce dist_ckpt_parallel_save option Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * determine sharding type from dist_ckpt_parallel_save Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * read model.disk_ckpt_parallel_save from cfg and pass it to mcore dist ckpt Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * Pass is_loading to mcore_optim.py's sharded_state_dict Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * Update nemo/core/optim/mcore_optim.py Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com>
Signed-off-by: Nithin Rao Koluguri <nithinraok> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
…r::forward (NVIDIA#9278) * Accept None as an argument to decoder_lengths in GreedyBatchedCTCInfer::forward (NVIDIA#9246) * Accept None as an argument to decoder_lengths in GreedyBatchedCTCInfer::forward GreedyCTCInfer::forward already allowed for this, so they did not implement the exact same interface. Now, they do. Also warn about not passing in the decoder_lengths argument. It is likely an error on the user's part not to pass it in explicitly. Signed-off-by: Daniel Galvez <dgalvez@nvidia.com> * Apply isort and black reformatting Signed-off-by: titu1994 <titu1994@users.noreply.github.com> * Log warning only once for sanity. Signed-off-by: Daniel Galvez <dgalvez@nvidia.com> --------- Signed-off-by: Daniel Galvez <dgalvez@nvidia.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Co-authored-by: titu1994 <titu1994@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> * Apply isort and black reformatting Signed-off-by: nithinraok <nithinraok@users.noreply.github.com> --------- Signed-off-by: Daniel Galvez <dgalvez@nvidia.com> Signed-off-by: titu1994 <titu1994@users.noreply.github.com> Signed-off-by: nithinraok <nithinraok@users.noreply.github.com> Co-authored-by: Daniel Galvez <galv@users.noreply.github.com> Co-authored-by: titu1994 <titu1994@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: nithinraok <nithinraok@users.noreply.github.com>
* fix extra state and post process * move to args * Apply isort and black reformatting Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> --------- Signed-off-by: JRD971000 <JRD971000@users.noreply.github.com> Co-authored-by: JRD971000 <JRD971000@users.noreply.github.com>
* Remove .nemo instead of renaming Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add ignore_errors=True flag Signed-off-by: dimapihtar <dpihtar@gmail.com> * Revert "Remove .nemo instead of renaming" This reverts commit b836410. Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Remove backup .nemo after success Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Update tests Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Backup .nemo imediately before save_to Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: mikolajblaz <mikolajblaz@users.noreply.github.com> * Fix CTC import Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: dimapihtar <dpihtar@gmail.com>
* refactor pack seq script Signed-off-by: Chen Cui <chcui@nvidia.com> * add copyright header Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> * update doc Signed-off-by: Chen Cui <chcui@nvidia.com> * minor Signed-off-by: Chen Cui <chcui@nvidia.com> * Apply isort and black reformatting Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> --------- Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: cuichenx <cuichenx@users.noreply.github.com> Co-authored-by: cuichenx <cuichenx@users.noreply.github.com>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information