LitData v0.2.53

Lightning AI ⚡ is excited to announce the release of LitData v0.2.53

Highlights

Lightning AI Storage - Direct download and upload

Lightning Studios have special directories for data connections that are available to an entire teamspace. LitData functions that reference those directories will experience a significant performance increase as uploads and downloads will happen directly from the bucket that backs the folder. LitData has supported existing folder types like S3 and GCS folders, and this release introduces support for lightning_storage folders which were recently launched.

For example, output artifacts from this code will be directly uploaded to the my-data-1 Lighting Storage bucket.

from litdata import optimize

def should_keep(data):
    if data % 2 == 0:
        yield data

if __name__ == "__main__":
    optimize(
        fn=should_keep,
        inputs=list(range(1000)),
        output_dir="/teamspace/lightning_storage/my-data-1/output",
        chunk_bytes="64MB",
        num_workers=1
    )

Similarly, data will be downloaded directly from the my-data-1 Lightning Storage bucket in this example code.

from litdata import StreamingRawDataset

if __name__ == "__main__":
    data_dir = "/teamspace/lightning_storage/my-bucket-1/data"

    raw_dataset = StreamingRawDataset(data_dir)

    data = list(raw_dataset)
    print(data)

References to any of the following directories will work similarly:

1. /teamspace/lightning_storage/...
2. /teamspace/s3_connections/...
3. /teamspace/gcs_connections/...
4. /teamspace/s3_folders/...
5. /teamspace/gcs_folders/...

Changes

Added

• Add support for resolving directories in /teamspace/lightning_storage by @bhimrazy in #695
• Add support for direct upload to r2 buckets by @pwgardipee in #705
• Add readme docs for references to data connection dirs by @pwgardipee in #708

Changed

• Remove unnecessary fixed sleep by adding predicate-based path check by @Red-Eyed in #700
• ref(resolver): Refactors data connection resolution by adding a helper function and eliminating code duplication. by @bhimrazy in #706

Chores

• chore(deps): bump actions/first-interaction from 2 to 3 in the gha-updates group by @dependabot[bot] in #693
• chore(deps): update coverage requirement from ==7.8.* to ==7.10.* by @dependabot[bot] in #701
• chore(deps): bump pytest-random-order from 1.1.1 to 1.2.0 by @dependabot[bot] in #703
• chore(deps): bump cryptography from 45.0.4 to 45.0.7 by @dependabot[bot] in #704
• chore(deps): bump the gha-updates group with 3 updates by @dependabot[bot] in #707

Full Changelog: v0.2.52...v0.2.53

🧑‍💻 Contributors

We thank all folks who submitted issues, features, fixes and doc changes. It's the only way we can collectively make LitData better for everyone, nice job!

Key Contributors

@bhimrazy, @pwgardipee

New Contributors

• @Red-Eyed made their first contribution in #700
• @pwgardipee made their first contribution in #705

Thank you ❤️ and we hope you'll keep them coming!