Length check fails for IterableDataset

[snie2012]

I am using IterableDataset with unknown length. It fails inside the following if clause:

ignite/ignite/engine/engine.py

Line 642 in cc78b3b

if hasattr(data, "__len__"):

Here is the logic why it fails:

1. Data is PyTorch DataLoader, which has the __len__ attr, so it goes into the if clause
2. It runs the code epoch_length = len(data), and leads to length = self._IterableDataset_len_called = len(self.dataset) in dataloader.py in pytorch
3. Since dataset is an IterableDataset, it doesn't have length so it throws sth like TypeError: object of type 'IterableDataset' has no len()

My concern is that we shouldn't expect IterableDataset to have __len__ and the code should not fail because of it. Any thoughts on this?

[vfdev-5]

@snie2012 thanks for the report! Yes, you are right about that. We have a test with IterableDataset but with defined epoch_length:

ignite/tests/ignite/engine/test_engine.py

Line 511 in a2f302b

def test_engine_with_iterable_dataloader():

Yes, definitely, we need to support IterableDataset with unknown length !

[vfdev-5]

@snie2012 the bug should be fixed in the next nightly release

[SantoshGuptaML]

Should DistributedSampler also support IterableDataset ? I get this error message

    train_sampler = DistributedSampler(TrainDataset(config))
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/distributed.py", line 63, in __init__
    self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.num_replicas))
TypeError: object of type 'TrainDataset' has no len()

Where TrainDataset is an extension of IterableDataset

[vfdev-5]

@SantoshGuptaML no, torch's DistributedSampler does not support iterable datasets, only map-style datasets.

There are various way to route samples depending on process (distributed or simply mp):

• https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading
• https://discuss.pytorch.org/t/ddp-and-iterable-datasets/79180

In case of distributed training, the simplest way to distribute it (if possible) is to set up your iterable dataset depending on the rank:

import ignite.distributed as idist
rank = idist.get_rank()
ws = idist.get_world_size()
dataset = get_iterable_dataset(..., rank=rank, world_size=ws, ...)

where get_iterable_dataset is your custom method that fetches unique data for a rank.