GPU_mem not correlated with task manager GPU memory usage

[aqmery]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

I've been using yolov8 for about a year now for this project, but for some reason something changed in the last week, where the GPU_mem on my training pc is not correlating anymore with the GPU memory usage in task manager.
I also keep getting message about pylace crashing, however, using some lighter settings I can train on my laptop with no issues and no obvious difference between GPU_mem and task manager GPU memory.
I don't really have a clue where to start debugging this, does anyone have any suggestions?

Additional

No response

[aqmery]

Also it doesn't look like a memory leak issue, as the memory spikes this high basically from the start of the training loop and never goes down or much higher. (and training does take a lot longer, which would make sense if it's using the shared GPU memory, just really confused about the numbers in the print-outs)

[aqmery]

I don't exactly know why, but restarting everything and making a new environment has somehow fixed this issue, no idea what the exact cause was, but it's fixed.

[glenn-jocher]

Great to hear that restarting and setting up a new environment resolved the issue! Sometimes, a fresh setup can clear up underlying conflicts or settings that are hard to pinpoint. If you encounter any other issues or have more questions in the future, feel free to reach out. Happy training with YOLOv8! 🚀