Implements the UL2 Dataset and config

[MaximumEntropy]

What does this PR do ?

1. Implements the UL2 Dataset from https://arxiv.org/abs/2205.05131.
2. Adds a megatron_ul2_config.yaml to train an encoder-decoder model on this data using t5 pretraining script.

Collection: NLP

Changelog

• UL2 dataset class that inherits from T5Dataset
• UL2 yaml config
• Minor refactor of LMAdaptedT5Dataset

Usage

python megatron_t5_pretraining.py -cn megatron_ul2_config

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[lgtm-com]

This pull request introduces 3 alerts when merging e90094e into 8318980 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment
• 1 for Unused local variable
• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 8cbcd1c into 84265ac - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 8 alerts and fixes 1 when merging d2e984d into 84265ac - view on LGTM.com

new alerts:

• 7 for Module-level cyclic import
• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 3fdb609 into 84265ac - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[yidong72]

LGTM

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging f2c1919 into 7a9a8f0 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 56c7a71 into 7a9a8f0 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging 48a17f3 into 7a9a8f0 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging cc940b7 into e838862 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert and fixes 1 when merging c983d20 into f6936ce - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment

fixed alerts:

• 1 for Unused import