Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
runtime: make `GOMAXPROCS` cfs-aware on `GOOS=linux` #33803
The default setting of
This can lead to large latency artifacts in programs, especially under peak load, or when saturating all processors during background GC phases.
The smaller the container / larger the machine = the worse this effect becomes: let's say you deploy a fleet of micro service workers, each container having a cpu quota of 4, on a fleet of 32 processor machines.
To understand why, you really have to understand the CFS quota mechanism; this blog post does well (with pictures); this kubernetes issue further explores the topic (especially as it relates to a recently resolved kernel cpu accounting bug). But to summarize it briefly for this issue:
Running an application workload at a reasonable level of cpu efficiency makes it quite likely that you'll be spiking up to your full quota and getting throttled.
Background waste workload, like concurrent GC, is especially likely to cause quota exhaustion.
I hesitate to even call this a "tail latency" problem; the artifacts are visible in the main body of and can shift the entire latency distribution.
If you care about latency, reliability, predictability (... insert more *ilities to taste), then the correct thing to do is to never exceed your cpu quota, by setting
Using this as a default for GOMAXPROCS makes the world safe again, which is why we use uber-go/automaxprocs in all of our microservices.
I really have to disagree with some of the latter suggestions in kubernetes/kubernetes#67577
Some background on uber-go/automaxprocs#13 (changing from
I'll reprise (copied with some edits) my description from that issue here for easy reading:
@jcorbin I'm certainly not opposed. After dissecting uber-go/automaxprocs it seems like it requires a bunch of string parsing to really get to the numbers.
This is possible to do from the runtime but also a bit complex. Note that you can't allocate and you need to use raw system calls to process files.
I previously did something similar to get the default huge page size but later found you could just read an integer hiding down in
I assume it's not quite so simple with cgroups (even though for bash on my machine that just ends up in
If it's possible to reach at these values in a simpler way (i.e. just a static path at which there's a file that just contains an integer) that would be preferred.
GOMAXPROCS subsumes NumCPU for the purpose of sizing semaphores. If users set CPU affinity, then GOMAXPROCS will reflect that. If users only set GOMAXPROCS, then NumCPU would be inaccurate. Additionally, there are plans to make GOMAXPROCS aware of CPU quotas (golang/go#33803). Users are still advised to set CPU affinity instead of relying on GOMAXPROCS to limit CPU usage, because Staticcheck shells out to the underlying build system, which together with Staticcheck would be able to use more CPU than intended if limited by just GOMAXPROCS.