Synthetic dataloading for Megatron T5 and GPT models #6181
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vysarge
force-pushed
the
synthetic_dataloading
branch
2 times, most recently
from
1135923
to
2210856
Compare
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
vysarge
force-pushed
the
synthetic_dataloading
branch
from
2210856
to
28893ee
Compare
for more information, see https://pre-commit.ci
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
for more information, see https://pre-commit.ci
… up code Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
vysarge
force-pushed
the
synthetic_dataloading
branch
from
f4695ee
to
a40b01a
Compare
for more information, see https://pre-commit.ci
vysarge
changed the title
yidong72
reviewed
Mar 21, 2023
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
yidong72
previously approved these changes
Mar 21, 2023
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Files from this location will not be read; mock data will be generated instead. | ||
| logging.warning(f"Requested data_impl={data_impl}, so ignoring data_prefix setting: {data_prefix}") | ||
| if dataset_type == DSET_TYPE_T5: | ||
| from nemo.collections.nlp.data.language_modeling.megatron.t5_dataset import MockT5Dataset |
Check notice
Code scanning / CodeQL
Cyclic import
hsiehjackson
pushed a commit
to hsiehjackson/NeMo
that referenced
this pull request
Jun 2, 2023
* Added dataset producing synthetic data in the same shape as GPTDataset Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added dataset producing synthetic data in the same shape as T5Dataset Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added some config checks, simplified SyntheticGPTDataset, and cleaned up code Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename synthetic -> mock Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Added Jenkins test Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Alter random generation seed based on currently requested idx Signed-off-by: Valerie Sarge <vsarge@nvidia.com> --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com> Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Enables testing functionality and performance of training code using a synthetic version of existing datasets.
Collection: NLP (language_modeling)
Changelog
Usage
HYDRA_FULL_ERROR=1 python NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py \ model.data.data_impl=synthetic \ model.data.data_prefix=[] HYDRA_FULL_ERROR=1 python NeMo/examples/nlp/language_modeling/megatron_t5_pretraining.py \ model.data.data_impl=synthetic \ model.data.data_prefix=[]Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information