Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803
The default setting of
This can lead to large latency artifacts in programs, especially under peak load, or when saturating all processors during background GC phases.
The smaller the container / larger the machine = the worse this effect becomes: let's say you deploy a fleet of micro service workers, each container having a cpu quota of 4, on a fleet of 32 processor machines.
To understand why, you really have to understand the CFS quota mechanism; this blog post does well (with pictures); this kubernetes issue further explores the topic (especially as it relates to a recently resolved kernel cpu accounting bug). But to summarize it briefly for this issue:
Running an application workload at a reasonable level of cpu efficiency makes it quite likely that you'll be spiking up to your full quota and getting throttled.
Background waste workload, like concurrent GC, is especially likely to cause quota exhaustion.
I hesitate to even call this a "tail latency" problem; the artifacts are visible in the main body of and can shift the entire latency distribution.
If you care about latency, reliability, predictability (... insert more *ilities to taste), then the correct thing to do is to never exceed your cpu quota, by setting
Using this as a default for GOMAXPROCS makes the world safe again, which is why we use uber-go/automaxprocs in all of our microservices.
I really have to disagree with some of the latter suggestions in kubernetes/kubernetes#67577
Some background on uber-go/automaxprocs#13 (changing from
I'll reprise (copied with some edits) my description from that issue here for easy reading:
For comparative purposes, Oracle blog post about Java adding similar support ( especially for GC threads )
@jcorbin I'm certainly not opposed. After dissecting uber-go/automaxprocs it seems like it requires a bunch of string parsing to really get to the numbers.
This is possible to do from the runtime but also a bit complex. Note that you can't allocate and you need to use raw system calls to process files.
I previously did something similar to get the default huge page size but later found you could just read an integer hiding down in
I assume it's not quite so simple with cgroups (even though for bash on my machine that just ends up in
If it's possible to reach at these values in a simpler way (i.e. just a static path at which there's a file that just contains an integer) that would be preferred.