DataLoader with num_workers > 1, and a Rand[Zoom/Rotate/Flip)d transforms

[hjmjohnson]

Describe the bug
When using a DataLoader with num_workers > 1, and a Rand[Zoom/Rotate/Flip)d transform, all the data in the multiple workers have the same random state.

To Reproduce

With train_ds having some random parameterized transforms.

    train_loader: DataLoader = DataLoader(
        train_ds,  # <-- This is a dataset of both the input raw data filenames + definition of transforms
        batch_size=1,
        shuffle=True,
        num_workers=88,
        collate_fn=list_data_collate,
    )

This is particularly disturbing when running on a machine with 40+ CPUs and huge numbers of images have the same parameter augmentation.

Expected behavior
Each transform should have it's own random parameters chosen, regardless of the number of workers chosen.

Screenshots
NOTE: The number of replicated rotation values is always equal to the num_workers specified.

Rotating by 19.367042973517755
Rotating by 19.367042973517755
Rotating by 19.367042973517755
Rotating by 19.367042973517755
Rotating by 4.039486469720721
Rotating by 4.039486469720721
Rotating by 4.039486469720721
Rotating by 4.039486469720721
Rotating by 13.13047017599905
Rotating by 13.13047017599905
Rotating by 13.13047017599905
Rotating by 13.13047017599905

[Nic-Ma]

Hi @hjmjohnson ,

Thanks for your bug report.
This is an known issue of "numpy + PyTorch multi-processing".
And you can easily fix it by adding below logic to your DataLoader initialization:

def worker_init_fn(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    worker_info.dataset.transform.set_random_state(worker_info.seed % (2 ** 32 - 1))

dataloader = torch.utils.data.DataLoader(... worker_init_fn=worker_init_fn)

Thanks.

[hjmjohnson]

FYI: I also came across documentation that indicates the real problem is out of the scope of MONAI.

https://pytorch.org/docs/stable/data.html

[hjmjohnson]

@Nic-Ma THANK YOU! Sorry for the invalid MONAI bug report. Your solution worked wonderfully!

[Nic-Ma]

You are welcome.
Thanks.

[tvercaut]

Should a note about this be put in our wiki or somewhere similar to start collating an FAQ?
Linking to other resources such https://pytorch.org/docs/stable/notes/faq.html#my-data-loader-workers-return-identical-random-numbers makes sense in such a FAQ of course.

[Nic-Ma]

Hi @tvercaut ,

Good idea, maybe @atbenmurray can help add this to our wiki page?
Ben spent much time to initialize our detailed wiki pages before.
Thanks.

[hjmjohnson]

@Nic-Ma It would be nice if the FAQ were indexed in a way that the Sphinx documentation could reference the FAQ content.

[Nic-Ma]

I will try to raise the topic to set up FAQ to the core team guys.
Maybe we can have a discussion this Friday.
Thanks.

[atbenmurray]

@Nic-Ma @tvercaut I can certainly help with the wiki stuff