cuda_plot keeps GPU x open when only GPU y is selected

[mwpastore]

When I select e.g. CUDA device 1 with e.g. -g 1, I can see in nvtop that cuda_plot keeps a process open on CUDA device 0. I'm not sure what impact this has, if any, but I can see some undesirable RMA in numatop and this is the most likely explanation in my case. I think it would be better if cuda_plot completely disengaged the CUDA device(s) not selected with -g.

[mwpastore]

More info: a cuda_plot -g 0 only starts a single job on CUDA device 0, but a cuda_plot -g 1 starts jobs on CUDA devices 0 and 1.

[madMAx43v3r]

Yeah I've seen this with many CUDA applications before, the driver likes to use GPU 0 always for no apparent reason...

[madMAx43v3r]

I think it would be better if cuda_plot completely disengaged the CUDA device(s) not selected with -g.

I'm not using device 0 on purpose... it's the driver doing it internally.

[madMAx43v3r]

The first thing I do in the code is cudaSetDevice(device) for your selected -g

[mwpastore]

Maybe it's harmless? I was having trouble pinning down a solid benchmark result that showed any penalty from having that open, unused handle to the other GPU.

I wonder if you can do cudaSetValidDevices before anything else to exclude the non-selected GPU(s)?

[madMAx43v3r]

I think that function just remaps the integers, which I can use to simplify the code but that's it.

[mwpastore]

I was able to "solve" this "issue" by setting CUDA_VISIBLE_DEVICES in the environment before launching cuda_plot. I don't even need to use the -g flag anymore.

[madMAx43v3r]

good to know, thx