Efficient data loading functionality

[shubhamagarwal92]

🚀 Feature

Efficient data loader for huge dataset with lazy loading!

Motivation

I am working with a huge dataset consisting of 120m examples (~40G raw text) in a single csv file. I tried to follow the run_glue distributed training example, however this is too slow as it first creates all the examples and cache it here. Basically only first process in distributed training process the dataset and others just use the cache.

Is there any data loader (or a working example) that would be efficient for training the model on such a huge dataset?

Additional context

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shubhamagarwal92]

@thomwolf Is there an efficient implementation for this? Could you re-open the issue please.

[Santosh-Gupta]

CSV generally has slower load times. How about benchmarking with pickle, parquet, and feather? Pytorch's dataloader can handle multiple files and multiple lines per file
https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769