Moving large files between tasks?

[stwerner97]

Hi, I am looking into Flyte at the moment, but am unsure if Flyte is suited for me and would like to get some guidance from the community 😄

I want to use Flyte, among other things, for training neural networks on small- to medium-sized datasets of up to ~1-3 TBs.

Is it possible to move files of this size around between tasks? How would I go about training a neural network then? Assuming that data is stored in a S3 bucket, I would need to download the data first and then invoke a training, right? Is it possible to load the data onto some persistent volume then, or is it essential that the data fits into RAM? Is it possible to share a filesystem between two tasks or are these isolated?

[kumare3]

hi @stwerner97 thank you for reaching out. I think Flyte is absolutely fine in handling large files. You do not need to always download a file, you can ideally stream a file. here is the docs for this api - https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.types.file.FlyteFile.html#flytekit.types.file.FlyteFile.open

Also Flyte does not really move data, it just copies pointers to S3 over. Here is some more docs to understand this - https://docs.flyte.org/en/latest/concepts/data_management.html#divedeep-data-management

Data does not need to fit into ram, best would be to stream and if you are using torch using a streaming dataloader. An example for this could be https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd.
Also we are more than happy to help build such a loader as a plugin in flytekit (these will be represented as types). You can extend flyte types to support any of these modalities.

Docs on how to extend flyte types - https://docs.flyte.org/projects/cookbook/en/latest/getting_started/extending_flyte.html#id1

[kumare3]

Let me know if this helps

[stwerner97]

Thanks for the extensive answer @kumare3 ! 🙆‍♂️

That makes a lot of sense, although I sometimes need to deal with some data sources that do not support steaming.

Is it possible to use persistent volumes or another solution for such cases in Flyte?

[kumare3]

Yes absolutely- you can use ephemeral
Storage if your node supports it by simply downloading or
Use pvc's but you will have to manage the pvs on your own.
You can use pod templates to mount them.

https://docs.flyte.org/en/latest/deployment/configuration/general.html#compile-time-podtemplates