Add configuration to enable/disable NVML diagnostics

[pentschev]

• Closes Simple local cluster on laptop fails with deactivated GPU #4885

In some cases, users may want or need to explicitly disable NVML diagnostics, this PR adds support for that via Dask config.

[pentschev]

cc @quasiben @jrbourbeau @SteffenBauer

[pentschev]

Notice that I introduced a new section diagnostics for lack of a better place to add nvml, but feel free to suggest a more suitable section or name to replace diagnostics.

[quasiben]

What do you think about placing this under Dashboard section instead of creating a new section ?

cc @jacobtomlinson

[pentschev]

What do you think about placing this under Dashboard section instead of creating a new section ?

I thought about that, but I reach the conclusion it could be more confusing, since the NVML module is under distributed/diagnostics and there's another directory distributed/dashboard where the distributed.dashboard configs apply.

[quasiben]

That seems reasonable. I'm @jrbourbeau how do you feel about adding a new config section ?

[jrbourbeau]

IIUC this nvmlEnabled variable appears to be equivalent to the the distributed.diagnostics.nvml config value. If that's the case, I'd prefer to not introduce a new variable and instead just use the config value

[pentschev]

This is a good point, addressed in fed9e97 .

[jrbourbeau]

Since this is GPU specific, we might want to put this option in it's own section like rmm and ucx

distributed/distributed/distributed.yaml

Lines 228 to 237 in 0c15432

	rmm:
	pool-size: null
	ucx:
	cuda_copy: False # enable cuda-copy
	tcp: False # enable tcp
	nvlink: False # enable cuda_ipc
	infiniband: False # enable Infiniband
	rdmacm: False # enable RDMACM
	net-devices: null # define what interface to use for UCX comm
	reuse-endpoints: null # enable endpoint reuse

[pentschev]

I don't mind moving it there if you prefer, but it seems like the general feel about those is that they have been a mistake to be outside of the distributed schema. For example #4904 is attempting to move ucx into distributed to comply with the general config schema.

[jacobtomlinson]

I think putting this under diagnostics makes sense.

[jrbourbeau]

Thanks @pentschev!