WSL Crash for multi-gpu usage

[naruto0426]

Version

Microsoft Windows [Version 10.0.19044.1706]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.10.102.1

Distro Version

Ubuntu 20.04

Other Software

No response

Repro Steps

pytorch 1.10.0+ cuda 10.2(from wsl2 kernel)

Expected Behavior

Work normal for multi-gpu usage

Actual Behavior

When training with pytorch using multiple gpus, it will crash about 10-20 minutes.
I have try 2 cases.
First, train with multiple gpus with ddp in pytorch.
Second, train with one gpu in pytorch and open 2 program to train different model.
Both of them will crash in 20 minutes.

Diagnostic Logs

No response

[naruto0426]

WslLogs-2022-05-28_17-08-39.zip
WslLogs-2022-05-28_17-19-43.zip
WslLogs-2022-05-28_18-52-02.zip
WslLogs-2022-05-28_17-07-56.zip

[naruto0426]

I'm not sure if all of them have logged the problem, but at least two of them do.

[naruto0426]

Whem the problem occur, the wsl will shudown automatically.

[benhillis]

@spronovo - Is this something you and Iouri would want to look into?

[naruto0426]

By the way, when using DataParallel mode on multiple gpus training, it can sometimes run over half-of-day, but sometimes only run twenty minutes before crash.
When using DistributedDataParallel, it will train more fast, but crash sooner (often in twenty minutes).
When training with only single gpu, everything works like a charm, but very slow because my gpu is only 1080ti.
Maximum of gpu devices for testing this issue are three of 1080ti.
The test target is yolov5. (Link: https://docs.ultralytics.com/tutorials/multi-gpu-training/)

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!