-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Binarized memmap dataloader for Megatron NMT, Inference and checkpoint -> nemo #4137
Conversation
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
…mt_memmap_dataloader
This pull request fixes 1 alert when merging 590f40e into 470587a - view on LGTM.com fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
This pull request fixes 1 alert when merging 968938b into 470587a - view on LGTM.com fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
…mt_memmap_dataloader
This pull request introduces 5 alerts and fixes 2 when merging fd0b976 into e838862 - view on LGTM.com new alerts:
fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
This pull request introduces 5 alerts and fixes 2 when merging 09884f3 into e838862 - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 5 alerts and fixes 2 when merging ff3df3d into f6936ce - view on LGTM.com new alerts:
fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
…mt_memmap_dataloader
This pull request introduces 5 alerts and fixes 2 when merging a6c9dda into f6936ce - view on LGTM.com new alerts:
fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
This pull request introduces 5 alerts and fixes 2 when merging 2d3ba1a into 2af3786 - view on LGTM.com new alerts:
fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
This pull request introduces 5 alerts and fixes 2 when merging 4214909 into f9d45db - view on LGTM.com new alerts:
fixed alerts:
|
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
This pull request introduces 5 alerts and fixes 2 when merging 37cda3b into ed35577 - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 5 alerts and fixes 2 when merging 86137df into 760e628 - view on LGTM.com new alerts:
fixed alerts:
|
…t -> nemo (NVIDIA#4137) * Temp Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add megatron dataset Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update config and fix global batch fetcher Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add dataset class Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update yaml Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix duplicate yaml key Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Translate method and preprocess script for raw text Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove pdb Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix arg name Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix other arg Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Change sampler back Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Move back to global batch fetcher to use distributed sampler Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add text memmap data Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Update monitor Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fixes for PP Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove unused import Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Truncate examples in text memmap Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * NMT training batch interpolation key Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * tarred data fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Change dataset type check Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix sampler Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Pass dataset cfg to determine type Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Log global step on validation step as well Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix NMT model saving with artifacts Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Initialize DDP in decode if not initialized. Needed for inference only mode Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Megatron NMT inference script Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Inference config file Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * hardcode max delta temporarily Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * detokenizer if processor is not none Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Sampler config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Compat with configs without sampler arg Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Comment for validation dataset type Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix tokenizer building Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * CI test for megatron nmt Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix tokenizer in restore Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * O2 restore from fix Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove print Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Change tokenizer model name in config Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Logging Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Set seed for distributed sampler Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Cluster debugging messages Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix max generation delta Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * No LM Init Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Use nlp save restore connector Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Remove useless infer args Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Typo Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * UTF8 safe print of translation result Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Style Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add save restore connector back with comment Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Refactor Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Add missing args Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Address comments Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Empty to restart * Fix CI test Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Check for test ds Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * set fusion to false Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Micha Livne <michalivne@users.noreply.github.com> Co-authored-by: Abhinav Khattar <aklife97@gmail.com> Signed-off-by: Georg Kucsko <gkucsko@gmail.com>
What does this PR do ?
Collection: NLP
Changelog
use_tarred_dataset
arg to a genericdataset_type
arg that can take[text, tarred, bin_memmap, text_memmap]
Usage
dataset_type: bin_memmap
.Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information