Cluster info is incorrect in Client widget

[fredms]

I have successfully connect Client with my cluster, and the cluster:

I digged into the cluster.py and it looks like the value of "memory" is the sum of values of ['memory_limit'] for CPU and GPU cases. For my GPU case, the value of ['memory_limit'] is 0.

[jacobtomlinson]

Thanks @FredFLi!

As this issue is related to the upstream Cluster and Client objects I'm going to transfer this over to the distributed repo.

[jacobtomlinson]

This seems to be a difference between the logic in Client here

distributed/distributed/client.py

Line 858 in 04de4b2

memory = sum(w["memory_limit"] for w in workers)

to Cluster here

distributed/distributed/deploy/cluster.py

Line 217 in 04de4b2

memory = sum(v["memory_limit"] for v in self.scheduler_info["workers"].values())

and are reporting two different values for GPU use cases.

[fredms]

Thanks @jacobtomlinson.

[mrocklin]

and are reporting two different values for GPU use cases.

I would expect memory_limit in both cases to be host memory limit rather than GPU memory limit of workers. @jacobtomlinson you mentioned that you were confused about the objective of this value. Can you say a bit more about that?

[jacobtomlinson]

Ok sure. That was my expectation also.

My understanding of dask-cuda-worker is that it configures the worker with no host memory, which forces the scheduler to make different decisions around moving memory when using GPUs.

I expect this is what is happening here and is why the client is showing 0 bytes, but I'm curious what memory the Cluster object is showing and why they are different.

I also wonder if it would be beneficial to measure GPU memory when available and also show it on the widgets.

[fredms]

Thanks @mrocklin @jacobtomlinson .

yes, we didn't configure the memory-limit to the actual size of worker. That's because we use 0 to represent unlimited.

--memory-limit TEXT Maximum bytes of memory that this worker should use.
Use 0 for unlimited, or 'auto' for
TOTAL_MEMORY * min(1, nthreads / total_nthreads)

Since the memory is 0 as shown in client widget, it will always exceed 95% memory load and the worker's nanny process will terminate it.

[TomAugspurger]

What's the next step here?

[mrocklin]

I expect this is what is happening here and is why the client is showing 0 bytes

Should we be showing "unknown" or nothing in this case?

but I'm curious what memory the Cluster object is showing and why they are different.

I briefly looked at both the LocalCUDACluster and distributed.deploy.cluster.Cluster classes and didn't see why these might differ. I think that LocalCUDACluster treats memory_limit as host memory limit.

I also wonder if it would be beneficial to measure GPU memory when available and also show it on the widgets.

During the in-person developer meeting last month we took a poll and learned that people were ok with baking in a few assumptions around GPUs. Maybe this is one of them? One solution to this problem might be to add GPU memory to the workers and scheduler and widgets and such and handle things more explicitly.

[jacobtomlinson]

Thanks @mrocklin