BrokenPipeError on Windows when setting the num_workers as 4 in DataLoader but 2 num_workers is fine

[songyuc]

🐛 Bug

BrokenPipeError on Windows when setting the num_workers as 4 in DataLoader but 2 num_workers is fine.

To Reproduce

Steps to reproduce the behavior:

I have submitted this issue in #31465, and the contributor commented as

See https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection.

I opened the link and found the relevant content Multiprocessing error “Broken pipe” as

Multiprocessing error “Broken pipe”
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
This issue happens when the child process ends before the parent process finishes sending data. There may be something wrong with your code. You can debug your code by reducing the num_worker of DataLoader to zero and see if the issue persists.

But my code ran normally if I set the num_workers as 2.
However, the 4 num_workers would still cause the BrokenPipeError.

Expected behavior

I want to set the num_workers as 4 instead of 2.

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

• PyTorch Version (e.g., 1.0): 1.3.1
• OS (e.g., Linux): Win10
• How you installed PyTorch (conda, pip, source): conda
• Build command you used (if compiling from source): by PyCharm
• Python version: 3.7.3
• CUDA/cuDNN version: CUDA 10.1
• GPU models and configuration: using CPU
• Any other relevant information:

Additional context

[albanD]

No need to reopen a new issue. You can continue the discussion in the original issue you already created. That way everyone has access to all the discussion context without reading multiple issues.
Thanks.

Closing as duplicate of #31465

[songyuc]

No need to reopen a new issue. You can continue the discussion in the original issue you already created. That way everyone has access to all the discussion context without reading multiple issues.
Thanks.
Closing as duplicate of #31465

But the original issue has been closed, I am not sure whether the developers would see the bugs?

[albanD]

If you don't agree with the fact that the original issue should be closed, you can re-open it to continue the discussion.

[songyuc]

If you don't agree with the fact that the original issue should be closed, you can re-open it to continue the discussion.

Thanks sincerely for your help!