GPU shared memory causing slow PyTorch training #10452

Windows Version

Microsoft Windows [Version 10.0.22621.2215]

WSL Version

WSL version: 1.2.5.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

5.15.90.1

Distro Version

Ubuntu 20.04

Other Software

PyTorch 2.0.1

Repro Steps

Run a PyTorch training job that repeatedly allocates close to the max physical VRAM available on an attached Nvidia GPU. The issue occurs both when the GPU is the primary display GPU and when it is not (this system has a 3080 and 3090 of which the 3080 drives the displays).

Expected Behavior

I expect similar it/s to before I installed some OS and driver updates and low usage of the GPU BUS and shared memory.

Actual Behavior

However, instead the GPU utilization is pinned to 100%, BUS usage spikes much higher than normal, GPU temps stay low, shared GPU memory spikes from it's normally near-zero value and training occurs at a snails pace. Reducing the batch size by one resolves this issue.

Example:

Diagnostic Logs

No response

Edit:

It appears some people on Reddit running into the same issue are blaming recent Nvidia driver updates: https://www.reddit.com/r/LocalLLaMA/comments/14fb9c0/last_nvidia_drivers_let_you_use_the_shared_memory/

Is this still the most relevant place to post this?