custom cache directory for local path

[csy1204]

🚀 Feature

There have been many situations where it would be beneficial for users to specify the Cache Directory. I would like to contribute by developing a feature that allows passing the desired path as a paramter.

Motivation

We use several file systems that have different performances and features. so we need to choose cache directory for different use cases.

Pitch

from litdata import StreamingDataset

dataset = StreamingDataset(...,  cache_dir="./chunks", max_cache_size="10GB")

• add a parameter in StreamingDataset

Alternatives

Additional context

[github-actions]

Hi! thanks for your contribution!, great first issue!

[tchaton]

Hey @csy1204, You can already do it right now.

from litdata import StreamingDataset
from litdata.streaming.cache import Dir

dataset = StreamingDataset(input_dir=Dir(cache_dir, data_dir), max_cache_size="10GB")

But I do agree this isn't very straightforward. Feel free to make a PR expose it on the StreamingDataset. This should be fairly simple.

[csy1204]

@tchaton

Thanks! I would like to use a remote storage for input_dir and store the cache in a custom directory as well.
so, the following code is more accurate. I will work on this feature. 🙋🏻‍♂️

from litdata import StreamingDataset
from litdata.streaming.cache import Dir

dataset = StreamingDataset(input_dir="s3://data-bucket/train", cache_dir="/fast_fs/.cache" max_cache_size="10GB")

https://github.com/Lightning-AI/litdata/blob/df8dcd10bdae27a5b32a7b16a49989508d26c66a/src/litdata/utilities/dataset_utilities.py#L93-L101

[deependujha]

Have you tried:

ds= StreamingDataset(input_dir=Dir(path="/fast_fs/.cache", url="s3://data-bucket/train"), max_cache_size="10GB")

[csy1204]

@deependujha cc. @tchaton

While working on this feature myself, I gained a precise understanding of its meaning. I realized that in the directory, path and URL can be utilized differently. Fortunately, this helped me to deepen my understanding of litdata. 😂 Thank you for the excellent explanation.