Add feature to slice, subsample and split dataset #161

deependujha · 2024-06-10T05:51:57Z

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure to update the docs?
Did you write any new necessary tests?

What does this PR do?

Fixes #135 & fixes #145.

Adds support to slice StreamingDataset
Adds support to Subsample StreamingDataset
Adds support to train_test_split StreamingDataset

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

for more information, see https://pre-commit.ci

codecov · 2024-06-10T06:00:11Z

Codecov Report

Attention: Patch coverage is 94.09091% with 13 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@b51b597). Learn more about missing BASE report.

Additional details and impacted files

@@          Coverage Diff          @@
##             main   #161   +/-   ##
=====================================
  Coverage        ?    78%           
=====================================
  Files           ?     33           
  Lines           ?   4324           
  Branches        ?      0           
=====================================
  Hits            ?   3363           
  Misses          ?    961           
  Partials        ?      0

for more information, see https://pre-commit.ci

src/litdata/streaming/config.py

src/litdata/utilities/train_test_split.py

tests/utilities/test_shuffle.py

tests/utilities/test_train_test_split.py

src/litdata/utilities/train_test_split.py

src/litdata/streaming/dataset.py

src/litdata/streaming/cache.py

src/litdata/utilities/train_test_split.py

for train_test_split

are of different sizes

for more information, see https://pre-commit.ci

…und error

for more information, see https://pre-commit.ci

src/litdata/streaming/dataset.py

src/litdata/streaming/config.py

src/litdata/streaming/dataset.py

src/litdata/utilities/dataset_utilities.py

for more information, see https://pre-commit.ci

src/litdata/utilities/subsample.py

lantiga

Looks great! Just minor nits on my part, let’s try this out with redpajama

src/litdata/streaming/config.py

src/litdata/streaming/dataset.py

src/litdata/streaming/item_loader.py

src/litdata/utilities/dataset_utilities.py

src/litdata/streaming/reader.py

src/litdata/utilities/dataset_utilities.py

for more information, see https://pre-commit.ci

Co-authored-by: Luca Antiga <luca@lightning.ai>

…subsampled chunks

for more information, see https://pre-commit.ci

…om/deependujha/litdata into feat/slice-subsample-split-dataset

for more information, see https://pre-commit.ci

…om/deependujha/litdata into feat/slice-subsample-split-dataset

for more information, see https://pre-commit.ci

tchaton

Awesome contribution. Thanks you :)

Add feature to slice, subsample and split dataset

0ec0bcf

deependujha requested review from tchaton and awaelchli as code owners June 10, 2024 05:51

[pre-commit.ci] auto fixes from pre-commit.com hooks

b1cfb60

for more information, see https://pre-commit.ci

deependujha and others added 7 commits June 11, 2024 15:48

fixed CacheBatchSampler issue

e48fd22

[pre-commit.ci] auto fixes from pre-commit.com hooks

def6509

for more information, see https://pre-commit.ci

fixed pre-commit-ci errors

9eaa83b

fix mypy typing errors

ac14be8

[pre-commit.ci] auto fixes from pre-commit.com hooks

e9cb7c2

for more information, see https://pre-commit.ci

Merge branch 'main' into feat/slice-subsample-split-dataset

65b3f8d

update

1d1cb63

tchaton reviewed Jun 13, 2024

View reviewed changes

tchaton added 3 commits June 13, 2024 07:46

update

35b6c03

update

ed63866

update

1c7ec87

tchaton mentioned this pull request Jun 13, 2024

add support for slicing, subsampling and splitting StreamingDataset #156

Closed

4 tasks

yhl48 reviewed Jun 13, 2024

View reviewed changes

src/litdata/streaming/cache.py Outdated Show resolved Hide resolved

src/litdata/utilities/train_test_split.py Outdated Show resolved Hide resolved

src/litdata/utilities/train_test_split.py Outdated Show resolved Hide resolved

deependujha and others added 11 commits June 14, 2024 08:15

Made dataset.py file more readable by separating concerns, and added doc

b204d30

for train_test_split

instead of loading sampled chunks, now we keep sampled chunk files

b63badc

modified subsampling and splitting to work perfectly even when chunks

8a9fe87

are of different sizes

[pre-commit.ci] auto fixes from pre-commit.com hooks

57d9f7e

for more information, see https://pre-commit.ci

fixed pre-commit ci errors

b157269

[pre-commit.ci] auto fixes from pre-commit.com hooks

ca6d771

for more information, see https://pre-commit.ci

fix pre-commit ci fails

23153e6

fixed test_s3_streaming_dataset get_downloader_cls attribute not fo…

b1066b4

…und error

[pre-commit.ci] auto fixes from pre-commit.com hooks

e17392d

for more information, see https://pre-commit.ci

added noqa to prevent pre-commit from removing get_downloader_cls import

3707f80

[pre-commit.ci] auto fixes from pre-commit.com hooks

91cae4b

for more information, see https://pre-commit.ci

tchaton reviewed Jun 15, 2024

View reviewed changes

deependujha and others added 2 commits June 15, 2024 21:19

Reset method implemented for streaming dataset

42df0c7

[pre-commit.ci] auto fixes from pre-commit.com hooks

348ca80

for more information, see https://pre-commit.ci

tchaton reviewed Jun 16, 2024

View reviewed changes

src/litdata/utilities/subsample.py Outdated Show resolved Hide resolved

lantiga approved these changes Jun 16, 2024

View reviewed changes

deependujha and others added 23 commits June 17, 2024 05:37

pass seed argument in shuffle_lists_together

9c80e4d

[pre-commit.ci] auto fixes from pre-commit.com hooks

6cc50db

for more information, see https://pre-commit.ci

Update src/litdata/streaming/config.py

35c307d

Co-authored-by: Luca Antiga <luca@lightning.ai>

Update src/litdata/streaming/item_loader.py

6cdb1c6

Co-authored-by: Luca Antiga <luca@lightning.ai>

Update src/litdata/streaming/item_loader.py

b46d383

Co-authored-by: Luca Antiga <luca@lightning.ai>

Update src/litdata/utilities/dataset_utilities.py

c5531b5

Co-authored-by: Luca Antiga <luca@lightning.ai>

Update src/litdata/utilities/dataset_utilities.py

ffbd072

Co-authored-by: Luca Antiga <luca@lightning.ai>

Optimize loading_subsampled_chunks & fix dangerous initialization of …

d46ee2a

…subsampled chunks

update

5d2faa1

[pre-commit.ci] auto fixes from pre-commit.com hooks

354837c

for more information, see https://pre-commit.ci

update

7fbca74

Merge branch 'feat/slice-subsample-split-dataset' of https://github.c…

b6c4fdd

…om/deependujha/litdata into feat/slice-subsample-split-dataset

update

c1685d1

[pre-commit.ci] auto fixes from pre-commit.com hooks

efed036

for more information, see https://pre-commit.ci

update

b81e045

update

894159a

[pre-commit.ci] auto fixes from pre-commit.com hooks

b4de189

for more information, see https://pre-commit.ci

update

2c94ad8

Use Interval (namedtuple) to keep track of chunks relevant indices

3b12c8f

[pre-commit.ci] auto fixes from pre-commit.com hooks

bf338d9

for more information, see https://pre-commit.ci

update

ff8cb5f

Merge branch 'feat/slice-subsample-split-dataset' of https://github.c…

f01ddc8

…om/deependujha/litdata into feat/slice-subsample-split-dataset

[pre-commit.ci] auto fixes from pre-commit.com hooks

42c5f24

for more information, see https://pre-commit.ci

tchaton approved these changes Jun 17, 2024

View reviewed changes

tchaton merged commit 5c242b4 into Lightning-AI:main Jun 17, 2024
28 checks passed

deependujha deleted the feat/slice-subsample-split-dataset branch June 17, 2024 09:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add feature to slice, subsample and split dataset #161

Add feature to slice, subsample and split dataset #161

deependujha commented Jun 10, 2024

codecov bot commented Jun 10, 2024 •

edited

Loading

lantiga left a comment

tchaton left a comment

Add feature to slice, subsample and split dataset #161

Add feature to slice, subsample and split dataset #161

Conversation

deependujha commented Jun 10, 2024

What does this PR do?

PR review

Did you have fun?

codecov bot commented Jun 10, 2024 • edited Loading

Codecov Report

lantiga left a comment

Choose a reason for hiding this comment

tchaton left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 10, 2024 •

edited

Loading