ThreadDataLoader dataloading order is related to numworker, which makes it difficult to reproduce

[cvlearn913]

Describe the bug
During our experiments, we found that the ThreadDataLoader loading order made a difference when using the shuffling strategy related to numworkers

To Reproduce

import torch
from monai.utils import set_determinism
from monai.data import ThreadDataLoader
import monai
monai.config.print_debug_info()
set_determinism(0)
loader = ThreadDataLoader(torch.arange(10),batch_size=2,buffer_size=10,shuffle=True,num_workers=0)
for epoch in range(0,2):
    print("epoch:", epoch)
    for data in loader:
        print(data)
    print("------")
loader = ThreadDataLoader(torch.arange(10),batch_size=2,buffer_size=10,shuffle=True,num_workers=10)
for epoch in range(0,2):
    print("epoch:", epoch)
    for data in loader:
        print(data)
    print("------")

Expected behavior
produces the same output with different num_woker settings

Screenshots

Environment

MONAI version: 0.8.1
Numpy version: 1.21.4
Pytorch version: 1.8.1+cu111
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 71ff399a3ea07aef667b23653620a290364095b1

Optional dependencies:
Pytorch Ignite version: 0.4.8
Nibabel version: 3.2.1
scikit-image version: 0.18.3
Pillow version: 8.4.0
Tensorboard version: 2.8.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.9.1+cu111
tqdm version: 4.62.3
lmdb version: 1.3.0
psutil version: 5.9.0
pandas version: 1.3.4
einops version: 0.4.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.

Ensuring you use the relevant python executable, please paste the output of:

[wyli]

thanks for reporting, I did more tests, indeed it's an issue even for the default monai DataLoaders:

import torch

from monai.data import DataLoader
from monai.utils import set_determinism


def main():

    set_determinism(0)
    loader = DataLoader(torch.arange(10), batch_size=2, shuffle=True, num_workers=0)
    for epoch in range(0, 2):
        print("epoch:", epoch)
        for data in loader:
            print(data)
        print("------")

    set_determinism(0)
    loader = DataLoader(torch.arange(10), batch_size=2, shuffle=True, num_workers=10)
    for epoch in range(0, 2):
        print("epoch:", epoch)
        for data in loader:
            print(data)
        print("------")


if __name__ == "__main__":
    main()

the root cause is that the randomization implementation changed in the recent pytorch versions and monai's implementation is out of date.
A workaround would be using the generator=torch.Generator() parameter:

set_determinism(0)
loader = DataLoader(torch.arange(10), batch_size=2, shuffle=True, num_workers=0, generator=torch.Generator())

the output should be deterministic in the same way with different num_workers settings.
I'll try to improve monai's default implementation.