Tune GC parameters

[pitrou]

According to Linux perf, 18% of the scheduler's runtime (and perhaps the workers as well, depending on the workload) can be spent in the Python GC. I suggest dask-worker and dask-scheduler (as well as Nanny perhaps) tune the GC parameters at process startup to lessen the number of garbage collection attempts.

Something vaguely like:

g0, g1, g2 = gc.get_threshold()
gc.set_threshold(g0 * 2, g1 * 2, g2 * 2)

[pitrou]

The following:

g0, g1, g2 = gc.get_threshold()
gc.set_threshold(g0 * 3, g1 * 3, g2 * 3)

seems to save around 8% on the scheduler benchmark.

[jonmorton]

8% seems like a lot! I am seeing similar gains.
I was getting a lot of "WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)" type messages until I increased the gc thresholds.

Any interest in incorporating this by default?

[mrocklin]

I would be fine with this. Probably we would want to do it in distributed/cli/dask_scheduler.py. Want to give it a try and submit a small PR @jonmorton ?

[mrocklin]

Done in #2624

…

On Wed, Apr 17, 2019 at 9:51 PM Jon Morton ***@***.***> wrote: 8% seems like a lot! I am seeing similar gains. I was getting a lot of "WARNING - full garbage collections took 13% CPU time recently (threshold: 10%)" type messages until I increased the gc thresholds. Any interest in incorporating this by default? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1653 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTH5O7NE6BNRLX2FDCTPQ7OZTANCNFSM4EJIOYJA> .