Inject custom s3 connection configs (credentials)

[csy1204]

🚀 Feature

• Inject custom s3 connection config into the StreamingDataset

Motivation

Thanks you for the awesome project! I primarily use litdata in a local environment, connecting to S3 to fetch data. However, managing multiple S3 connections for services like mlflow has become quite challenging, especially when handling AWS S3 credentials through environment variables.

To address this, it would be great if we could inject custom S3 connection configurations into the streaming dataset. Currently, it appears that the S3Client is running globally.

Pitch

Alternatives

Additional context

[tchaton]

Hey @csy1204. Great request. Would you be interested into making a contribution ?

[csy1204]

@tchaton
100% Sure. I will make a pull request for this soon.