Working with Lightning: which num_workers for DataLoaders?

[arthursw]

I created my pytorch-lightning DataModule this way (as in TorchIO + MONAI + PyTorch Lightning notebook):

class DataModule(pl.LightningDataModule):
	def __init__(self, data_dir: Path, config: Config):
		super().__init__()

[...]

	def train_dataloader(self):
		patches_training_set = tio.Queue(
			subjects_dataset=self.train_set,
			max_length=self.config.max_queue_length,
			samples_per_volume=self.config.samples_per_volume,
			sampler=self.config.sampler,
			num_workers=56,                   # on a 56 cores CPU
			shuffle_subjects=True,
			shuffle_patches=True,
		)
		return DataLoader(patches_training_set, batch_size=self.config.training_batch_size)

But I get the following lightning warning about the returned DataLoader:

PossibleUserWarning: The dataloader, val_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument`

And if I add num_workers=56 (or other value > 0) to the DataLoader parameters, I get the torchio error:

The number of workers for the data loader used to pop patches from the queue should be 0. Is it?

So, I guess I should ignore the lightning warning on DataLoaders.

I guess torchio does the job of loading and processing the patches in parallel, so the DataLoader do not need multiple workers and batches are ready to be consumed.

Is this correct?

So I can safely ignore the lightning warning with:

import warnings
warnings.filterwarnings("ignore", ".*does not have many workers.*")

as advised here?

[arthursw]

Ok I just found in the documentation:

num_workers refers to the number of workers used to load and transform the volumes. Multiprocessing is not needed to pop patches from the queue, so you should always use num_workers=0 for the DataLoader you instantiate to generate training batches.

[...]
patches_loader = DataLoader(
    patches_queue,
    batch_size=16,
    num_workers=0,  # this must be 0
)

So I have my answer, thanks!