Repository navigation
Avoid setting CPU limits for Guaranteed pods #51135
Description
Activity
Can you explain a bit more about what "avoid setting CPU limits on Guaranteed pods by default" means? A user may set Limit, but depending what other pods are running on the node their pod gets scheduled to, they may or may not get more CPU than indicated by the Limit?
I guess I can see the argument for making this behavior configurable, but what granularity were you think of? Per (guaranteed) pod or per cluster? This might be something a cluster admin wants to set cluster-wide (though I guess they could use an admission controller to do that if it is specified per pod...)
BTW you mentioned that the current behavior thows users off. My naive assumption would be that someone sees the word "limit" and assumes that their pod will be "limited" to using that amount of CPU. So I think the current behavior is more intuitive than the alternative. That said I definitely see the argument for making it configurable.
Can you explain a bit more about what "avoid setting CPU limits on Guaranteed pods by default" means? A user may set Limit, but depending what other pods are running on the node their pod gets scheduled to, they may or may not get more CPU than indicated by the Limit?
There will be no CPU Quota limits (throttling) applied to Guaranteed pods. Essentially, their CPU limits will be ignored.
I guess I can see the argument for making this behavior configurable, but what granularity were you think of? Per (guaranteed) pod or per cluster? This might be something a cluster admin wants to set cluster-wide (though I guess they could use an admission controller to do that if it is specified per pod...)
I think having a config per-node (or groups of nodes) may be better.
So I think the current behavior is more intuitive than the alternative. That said I definitely see the argument for making it configurable.
What users expect is that when they set the limit to 1 CPU, they can continue to use 1 CPU continuously and not more than that. In reality though with CPU Quota applied, in a slot of CPU scheduling time, a pod can use more than 1 CPU then get unscheduled and not get access to CPU for the rest of the slot. This means the spikes are penalized very heavily which is not what users expect. CPU Quota may be useful for scavenging pods like Bu and BE pods.
Reacted by Sandor Szücs, Klaus @ LambdaClass, Oliver Hassett, Stephen Blum and ZakariaZermaniReacted by Nikita GuchakovSo I hit this one rather hard. I was setting CPU request == CPU limits, so that we would be in guaranteed class for critical system pods, so that those pods would be less likely to be evicted. However, I observed that critical pods were being CPU throttled and that this was causing really bad behaviour in our cluster.
Looking at the manifests in
cluster/, it seems that the recommended approach is:- Set CPU requests, don't set CPU limit (so we will be in burstable QoS)
- Still OK to set memory limit == request if we want to protect against a memory leak - we will still be in burstable QoS.
- Enable the ExperimentalCriticalPodAnnotation feature and set
scheduler.alpha.kubernetes.io/critical-podon the pods, along with a toleration - typically[{"key":"CriticalAddonsOnly", "operator":"Exists"}], but[{"operator":"Exists"}]for something that should run on every node e.g. a networking daemonset. - Ideally run the rescheduler, but this is not required because ExperimentalCriticalPodAnnotation should prevent eviction (despite being in burstable class)
Is that correct?
On the direct topic of this particular issue, if that's correct, I think it would be confusing to have the CPU limits not applied when limit == request. IMO the bigger problem is that a desire to get into guaranteed class nudged me to set CPU limits, and that a desire not to give up too much resources across the cluster caused me to set those limits too low.
But what is the duration of a CPU scheduling time unit? (And should we recommend making it smaller to avoid the feast-then-famine scenario?)
Reacted by Sandor Szücs, Adrien Chardon, Meet Shah, Dhawal Patel, realdimas, sriram komma, absolutarin, mudideng, Stephen Blum, pingvinton and 2 moreI think what you wrote is actually correct.
All of this should get a lot better once priority is fully implemented, both in the scheduler (alpha in 1.8) and on the kubelet (I assume in alpha in 1.9?). Then all of those steps should be reducible to
- Set CPU requests, don't set CPU limit
- Still OK to set memory limit == request if we want to protect against a memory leak - we will still be in burstable QoS.
- Set a high priority on your pod
- added 2 commits that reference this issue
on Nov 13, 2017 - added a commit that references this issue
on Nov 13, 2017 /remove-priority P2
/priority important-soon123 remaining items
- added a commit that references this issue
on Sep 2, 2018 There's a bug in CFS throttling affecting well behaved applications most, not just CFS period configuration tradeoff. In #67577 I linked upstream kernel bug, reproduction and the possible fix.
Hope that helps.
@bobrik, came across your gist regarding this issue in the cfs. Commented on the gist, but not sure if it sends out any notifications, so will post it here as well.
I'm also trying to understand this issue, and not sure how is it possible to run this cfs.go with cfs-quota 4000, and get Throttling in less than 50% of the iterations. If I understand it right cfs-quota 4000 is 4 ms, so basically trying to execute this 5ms burn in 4ms, it should always Throttle.... what am I missing here?
- added 2 commits that reference this issue
on Mar 19, 2019 I know this issue has been dormant for a while but I did ran into this issue and my research led me here. Having spent some time reading through all the comments I want to try and summarize my understanding and some gaps that I still have.
Using an example of a container with a limit of 10% CPU. This results in a cpu.quota of 10 when period is set to 100. If this application is scheduled, uses up all 10 ms of its allotment during that time it will then sit idle for the remaining 90ms of that time slot. For a processor bound application this probably isn't a big deal, but for an I/O bound this would introduce latency.
If on the other hand this application uses its allotment of 10ms evenly over the time period (1ms, sleep for 9ms - 1ms sleep 9ms this latency wouldn't be seen.
To me it really boils down to expressing things in terms of % of CPU. As we all know an application can't use a % of a CPU, it is either using the CPU or it isn't. (even when you specify 10% CPU at any point in time the application is actually using 1 CPU). In both the examples above the application used 10% of the CPU during the period.
This is kind of a blocker for production clusters, because you can not set CPU limits for all latency critical applications. When we dropped the CPU limits from our ingress controller we got 12-20x less latency for the p99. The ingress controller is consuming about 30m CPU on average (kubectl top pods), but even setting to 1500m did not dropped the p99 very much. We got only improvements of factor 2-3x.
Where I am confused is on this situation. If the application is truly only using 30m then setting to 1500m would mean it always gets CPU time during the time slot. How was that value obtained? I would think that the observed time period that that value was obtained from would have to match what the cpu.cfs_period_us is.
After reading through all of this I am leaning towards those that feel setting limits are not going to accomplish what most people think. And the better approach is to adequately set requests as in the end, during contention shares is what will control what gets CPU.
Reacted by Simon Elsbrock, Mario Vejlupek, Peter Kingswell and Chinmay G@cyrus-mc I think we could have been hit a kernel bug or node over provisioning changed scheduling time.
You can now use an alpha gate to set per node different cfs_period to make it smaller than 100ms, which could solve your problem. Use node pools to separate in case you need different cfs_period settings for different workloads.Is this problem fixed now? I saw that it was repaired by the kernel team. But I didn't find the relevant information.
Reacted by David Forsman, Nell Boulle, Subhas Dandapani (RDX), Robin Hermann, speedoops and gruberdevReacted by Andre, David Forsman, Bryan, Abdurrahman Alp Köken, Rajiv Chodisetti, Prashanth Banda, Benjamin Vujnovac, dot-dtorma, Daniil Osipov, Vedansh Patel and 31 moreI'm not sure CPU limits are in milliseconds in k8s, because in Linux kernel despite having "ms" naming they are in microseconds. I wrote a bunch of questions I have here, I hope someone will help me clarify it.
Yes cpu resource settings are in millicore
The effect of CPU throttling is non obvious to users and throws them off when they try out kubernetes. It also complicates CPU capacity planning for pods. Pods that are carefully placed in Guaranteed QoS class are typically of higher priority than other pods, and these pods are expected to get the best performance. On the other hand, Burstable pods may have to be limited in certain environments to avoid tail latencies for Guaranteed pods.
I propose we avoid setting CPU limits on Guaranteed pods by default and continue to impose CPU limits on Burstable pods. We can provide a config option to ease the rollout in existing clusters. Users may need time to handle this transition, and some of them may choose to use the existing policies.
Closely related is static CPU pinning policies in kubelet which will naturally limit Guaranteed pods to specific cores if they request integral number of cores. This feature may be combined with turning off CPU limiting for Guaranteed pods. That said, having an ability to configure each of these feature independently will still be helpful.
@derekwaynecarr we have discussed this in the past, but AFAIK, an issue hasn't been filed thus far. @dchen1107 thoughts?
If there is a general agreement that this is the intended direction longer term, I can post a patch to make CPU limit policies configurable.
@kubernetes/sig-node-feature-requests