Shared Dataset Functionality

[vincentqb]

🚀 Feature

We want to build a unified data pipeline interface that offers building blocks for others to build on with the following objectives:

• Standardize datasets across domains.
• Offer flexible building blocks that can be combine to obtain other datasets.
• Enable datasets that do not fit in memory.
• Share code among domains.
• Facilitate parallel loading and processing of data.
• Decouple data loading and preprocessing/transformation.
• Offer static typing for datasets

Motivation

• The Domains currently each have their own non-standard dataset structure that may also download the data. This duplicate efforts and adds complexity to the user.
• A common bottleneck when generating datasets is reading the data. We want to offer an interface that enables reading the data and running initial preprocessing while maximizing available computing resources utilization.
• We may want to leverage specialize libraries such as NVIDIA DALI.

Additional Information

• torch.utils.data
• tf.data (e.g. uses dictionary for data point iteration)
• fast.ai's basic_data and data_block
• tnt
• torchnet
• torchdata

Datasets:

• Re-write language_modeling datasets (PennTreebank, WikiText103, WikiText2) text#624 Add a new dataset - Enwik9 text#610 new dataset format with librispeech and commonvoice audio#303 new datasets in domains
• Add VGGface2 dataset vision#1193 wants to select which metadata to return
• Internal: overview torchtext core torchvision
• safe datasets

Dataloader:

• torchaudio background iterator
• Shared Dataset Functionality #24915 wants to re-use worker processes
• FastDataLoader
• python 3.8 shared memory
• Internal: torchdata gil experiment DataLoader+Iterable

Features:

• Move collate_fn functionality / responsibility into Dataset object #12672 wants to move collate_fn functionality to datasets
• ChunkDataset API proposal #26547 wants distributed random sampling
• Sampler for IterableDataset #28743 for sampler for iterable datasets
• MultiCompose and SegmentationCompose [proof-of-concept] vision#1315 wants to apply an instance of random transform sequence to many images

cc @ssnl @fmassa @zhangguanheng66 @vincentqb @mrshenli

[vadimkantorov]

An additional wish: unify transforms interface. One idea: make them regular autograd functions / modules. This enables writing them more shared between domains (imagine various forms of cutout / warps, synthetic noise), moving them from cpu to gpu, performing them at other parts of the model, etc

[soumith]

all datasets are not moving to core pytorch. If you want, create a torchdata for it and make domain APIs depend on torchdata.

[zhangguanheng66]

all datasets are not moving to core pytorch. If you want, create a torchdata for it and make domain APIs depend on torchdata.

One of the plans is to keep the raw data links and utils in pytorch/data and have the preprocessing func and associated ops in each domain.

[cpuhrsch]

cc @ssnl - what is your take on this?

[ssnl]

I'm wonder what functionalities are planned for torchdata. Most of the proposed features sound already doable nowadays. I think additional explanation on what the exact needs are and why the current code can't do the job will be helpful for discussion.

For example,

a core download feature, within PyTorch

sounds similar to torch.hub's download utility function

a core raw data loading (e.g. ByteTensor iterator), within PyTorch

sounds doable just with the current data loading infra.

[vadimkantorov]

@soumith Any thoughts on my suggestion about transforms interface? (i.e. making them normal pytorch functions / modules to support better reuse)

[fmassa]

@vadimkantorov

Any thoughts on my suggestion about transforms interface? (i.e. making them normal pytorch functions / modules to support better reuse)

Yes, I agree, and torchvision is going to be doing this, see pytorch/vision#1375

[vincentqb]

Thoughts from offline chat with @taylorgordon20

How to help in debugging in a natural python way?

• Could use maps (e.g. Map(read, get_line)) instead of generators.
• Do we want static type analysis?
• Enforce structure between generators?
• tensorflow_datasets/image/coco.py has "features" for typing
• facebook pyre may support templating and typed dictionary check in functions?

Wishes?

• Batched
• Lazy better than Eager for data loading
• Async preferred over Sync (e.g. use CPU while GPU is busy, mutiple hosts)
• Prefer imperative over declarative
• Serializable (for pre-processing in particular) can be done in a few ways: torchscript, etc
• Shuffle, Sample
• Save, cache transformed data, memoization (available for free for flows?)