-
-
Notifications
You must be signed in to change notification settings - Fork 710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tune GC parameters #1653
Comments
The following:
seems to save around 8% on the scheduler benchmark. |
8% seems like a lot! I am seeing similar gains. Any interest in incorporating this by default? |
I would be fine with this. Probably we would want to do it in |
Done in #2624
…On Wed, Apr 17, 2019 at 9:51 PM Jon Morton ***@***.***> wrote:
8% seems like a lot! I am seeing similar gains.
I was getting a lot of "WARNING - full garbage collections took 13% CPU
time recently (threshold: 10%)" type messages until I increased the gc
thresholds.
Any interest in incorporating this by default?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1653 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACKZTH5O7NE6BNRLX2FDCTPQ7OZTANCNFSM4EJIOYJA>
.
|
According to Linux
perf
, 18% of the scheduler's runtime (and perhaps the workers as well, depending on the workload) can be spent in the Python GC. I suggestdask-worker
anddask-scheduler
(as well asNanny
perhaps) tune the GC parameters at process startup to lessen the number of garbage collection attempts.Something vaguely like:
The text was updated successfully, but these errors were encountered: