Add two shuffling algos: naive (globally) and py1b (fixed-size blocks).

[knighton]

Description of changes:

py1b is a shuffling algorithm that performs the final shuffle over fixed-size blocks instead of intra-shard like py1s. These units are presumably larger or much larger than single shards, leading to better shuffledness at the cost of having to download more shards to make progress.

naive is a shuffling algorithm that naively shuffles all-to-all. This is useful for single-node training on small data, where you want the most random shuffle possible. Statistically, this algorithm will result in all nodes downloading all shards, with those downloads all happening at the start of the epoch, bringing training to a crawl.

Also update the default shuffle algo from py1s to py1b.

Issue #, if available:

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

[karan6181]

Overall looks good. Can you also add a unit test for each shuffling algorithm (naive, py1b, py1s, py2s) to validate the output sample order with an expected sample order?

[karan6181]

Can you add more information here on what shuffle_block_size do ?

[karan6181]

Can you add a one to line line of brief statement explaining when to use which naive, py1b, py1s, py2s algorithm ?

[abhi-mosaic]

If we change the default to py1b, will all the StreamingDataset unit tests (aka for correctness, presence of samples, etc) rerun with py1b ? That would be great if so.

[knighton]

If we change the default to py1b, will all the StreamingDataset unit tests (aka for correctness, presence of samples, etc) rerun with py1b ? That would be great if so.

pytest is happy :)

[karan6181]

Based on the offline discussion, James will be creating a follow up PR to address the above two comments

1. Add a one to two line of brief statement explaining when to use which naive, py1b, py1s, py2s algorithm in the StreamingDataset class documentation.
2. Add a unit test for each shuffling algorithm (naive, py1b, py1s, py2s) to validate the output sample order with an expected sample order.