idle dask-worker & dask-scheduler have elevated CPU utilization

[dkhokhlov]

I am running dask-worker & dask-scheduler colocated.
Idle dask-worker & dask-scheduler show ~3% for each worker/task, without any dask client connected.
log is clean.

10528 owner      20   0 13016  3168  2836 S  0.0  0.0  0:00.00 ├─ /bin/bash -c source /opt/miniconda3/etc/profile.d/conda.sh;conda activate owner;dask-worker --nprocs 10 localhos
10541 owner      20   0 1840M  113M 28772 S  6.0  0.1 15h26:52 │  └─ /home/owner/.conda/envs/owner/bin/python /home/owner/.conda/envs/owner/bin/dask-worker --nprocs 10 localhost:
10577 owner      20   0 1091M  323M 88340 S  2.6  0.3  8h25:52 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10573 owner      20   0 1088M  320M 88348 S  2.6  0.2  8h29:13 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10569 owner      20   0 1085M  317M 88568 S  2.6  0.2  8h21:54 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10566 owner      20   0 1082M  313M 88424 S  2.0  0.2  8h15:14 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10562 owner      20   0 1093M  325M 88476 S  2.0  0.3  8h33:52 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10559 owner      20   0 1089M  321M 88816 S  2.6  0.3  8h33:56 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10556 owner      20   0 1082M  313M 88600 S  2.6  0.2  8h25:37 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10554 owner      20   0 1085M  315M 88176 S  2.6  0.2  8h17:27 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10551 owner      20   0 1106M  339M 88212 S  2.6  0.3  8h32:54 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10547 owner      20   0 1091M  322M 87940 S  2.6  0.3  8h28:29 │     ├─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(track
10545 owner      20   0 28264 11332  6284 S  0.0  0.0  0:00.03 │     └─ /home/owner/.conda/envs/owner/bin/python -c from multiprocessing.resource_tracker import main;main(10)
10517 owner      20   0 13020  3268  2940 S  0.0  0.0  0:00.00 ├─ /bin/bash -c source /opt/miniconda3/etc/profile.d/conda.sh;conda activate owner;dask-scheduler
10540 owner      20   0 2282M 1839M 38752 S  3.3  1.4  9h17:18 │  └─ /home/owner/.conda/envs/owner/bin/python /home/owner/.conda/envs/owner/bin/dask-scheduler

Environment:

conda info

     active environment : owner
    active env location : /home/owner/.conda/envs/owner
            shell level : 2
       user config file : /home/owner/.condarc
 populated config files :
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __cuda=11.2=0
                          __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/miniconda3/pkgs
                          /home/owner/.conda/pkgs
       envs directories : /opt/miniconda3/envs
                          /home/owner/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Linux/5.4.0-62-generic ubuntu/18.04.5 glibc/2.27
                UID:GID : 0:0
             netrc file : None
           offline mode : False

dask                      2021.4.0           pyhd8ed1ab_0    conda-forge
dask-core                 2021.4.0           pyhd8ed1ab_0    conda-forge
distributed               2021.4.0         py38h578d9bd_0    conda-forge

[fjetter]

Even without clients connected or work submitted, the workers communicate with the scheduler, measure process metrics, etc.. There is always something happening.

Is the 2-3% utilisation a problem or where you simply surprised to see something going on?

[dkhokhlov]

Yes, 3% per worker can add up to significant number when you have many workers.
Any tunables that can be used to decrease the frequency of this heartbeat activity?

[fjetter]

I went back to check what we're running and what might take up some time. the heartbeats are currently not configurable but the interval length increases linearly with the number of workers.

distributed/distributed/scheduler.py

Lines 7740 to 7752 in e938259

	def heartbeat_interval(n):
	"""
	Interval in seconds that we desire heartbeats based on number of workers
	"""
	if n <= 10:
	return 0.5
	elif n < 50:
	return 1
	elif n < 200:
	return 2
	else:
	# no more than 200 hearbeats a second scaled by workers
	return n / 200 + 1

Other than this there are the following config options you might want to try

• distributed.admin.tick.interval + distributed.admin.tick.limit controls a periodic callback where we simply measure the time since the last iteration as a proxy to know if the event loop is frequently blocked. it shouldn't consume any CPU really but it is executed very frequently (20ms) by default. increasing this might help. you might want to adjust the limit as well to avoid getting many warning logs. There might be more "very cheap but very frequent" period callbacks. if this helps anything I can dig a bit further

• distributed.admin.system-monitor.interval controls the frequency of us collecting host metrics (CPU load, RSS, etc.). Default is 500ms which might be too much for a packed cluster. these collections are relatively cheap but not for free.

• Last but not least there is also a statistical profiling running by default. I'm not sure if we can fully disable it but increasing the interval to sufficiently large values should at least help for diagnosing the issue. we can put in a disable toggle without a problem. The relevant option is called distributed.worker.profile.interval. The profiling is also running on idle servers since we also profile the server code itself. all of this can be accessed on the individual bokeh dashboards (see also https://distributed.dask.org/en/latest/diagnosing-performance.html#statistical-profiling)

Please let us know if any of this helps. Feedback in this area is highly appreciated.

[dkhokhlov]

Managed to bring it down to 0.7% per worker. Profile interval made up most of the difference.

/etc/dask# cat dask.yaml
distributed:
   admin:
     tick:
       interval: 1s
     system-monitor:
       interval: 5s
   worker:
     profile:
       interval: 5s

[fjetter]

FWIW I checked with some much older versions and the problem exists there as well, i.e. it appears to not be a regression but rather a problem nobody noticed, yet.

[j3pic]

I've been having the same problem. I noticed that the CPU usage goes up the longer the cluster is running. This makes it impossible to use GCP's autoscaler to scale a Dask cluster based on the workers' CPU usage. I aggressively limited the amount of meta computation using intervals in the tens of seconds, and got child process CPU usage down to 0.3% (3.7% for the parent process). I'll see how this evolves over time. Maybe I'll only have to restart the dask-worker daemon every week.

[fjetter]

Maybe I'll only have to restart the dask-worker daemon every week.

I know this is not a fix but if you intend to restart workers periodically, we have functionality for this. There is the distributed.worker.lifetime where you can set duration (e.g. "1d"), a stagger and whether they should restart or if you do the restarting yourself. The intention of this feature was mostly for cost control but it may help you in this case as well.

[jacobtomlinson]

@j3pic

This makes it impossible to use GCP's autoscaler to scale a Dask cluster based on the workers' CPU usage.

Just a warning when using autoscaling tools outside of Dask Adaptive. When scaling down Dask chooses which workers to remove and actively consolidates memory onto other workers before removing them.

If you let something like the GCP autoscaler do this based on a metric like CPU or memory then workers which are holding futures can be removed during scale down. This triggers Dask to resubmit tasks to recalculate the lost memory, which could then cause a scale up again. Clusters can bounce around like this randomly without ever completing the workload.

We typically suggest not using any external autoscalers with Dask and instead use the native tooling like dask-cloudprovider.

Maybe I'll only have to restart the dask-worker daemon every week.

I am interested in learning more about your workload. Most folks I interact with that run Dask on GCP do so ephemerally, creating and destorying clusters as they are needed. It sounds like you have a long running Dask cluster which you reuse. Could you share more about what you are doing with it?

[mrocklin]

It might make sense for folks to take a look at how we use psutil. Maybe there are nicer/cheaper ways to get system usage information.

…

On Fri, Jul 16, 2021 at 5:38 AM Jacob Tomlinson ***@***.***> wrote: This makes it impossible to use GCP's autoscaler to scale a Dask cluster based on the workers' CPU usage. Just a warning when using autoscaling tools outside of Dask Adaptive. When scaling down Dask chooses which workers to remove and actively consolidates memory onto other workers before removing them. If you let something like the GCP autoscaler do this based on a metric like CPU or memory then workers which are holding futures can be removed during scale down. This triggers Dask to resubmit tasks to recalculate the lost memory, which could then cause a scale up again. Clusters can bounce around like this randomly without ever completing the workload. Maybe I'll only have to restart the dask-worker daemon every week. I am interested in learning more about your workload. Most folks I interact with that run Dask on GCP do so ephemerally, creating and destorying clusters as they are needed. It sounds like you have a long running Dask cluster which you reuse. Could you share more about what you are doing with it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5000 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGROI2D4HUB6TKZIFLTYADYTANCNFSM47ROKUDQ> .