New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider disabling CPU quota via Kubelet flag --cpu-cfs-quota=false #1026

Closed
hjacobs opened this Issue May 3, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@hjacobs
Copy link
Contributor

hjacobs commented May 3, 2018

To mitigate the CPU throttling problem (kubernetes/kubernetes#51135), we should consider disabling CPU quota/throttling all together. That's what @vishh from Google recommends.

@szuecs

This comment has been minimized.

Copy link
Member

szuecs commented May 8, 2018

@hjacobs we have no countermeasure for overload and therefore I have to honestly disagree with the provided solution.
The question is if we disable it, what do we measure to decide if it is a success or not?

@hjacobs

This comment has been minimized.

Copy link
Contributor

hjacobs commented May 15, 2018

Looks like we might get kubernetes/kubernetes#63437 merged upstream (thus changing this issue to "configure cfs quota period") 😄

hjacobs added a commit that referenced this issue May 15, 2018

@mikkeloscar

This comment has been minimized.

Copy link
Contributor

mikkeloscar commented May 15, 2018

FYI: I back ported the patch to Kubernetes v1.9.7 and built a hyperkube image for testing.

registry.opensource.zalan.do/teapot/hyperkube:v1.9.7-cpu.cfs_period-flag_custom.pr-1-21
@hjacobs

This comment has been minimized.

Copy link
Contributor

hjacobs commented May 15, 2018

@mikkeloscar thanks! I actually want to see the impact/behaviour of completely disabling CPU quota, see #1044 --- e.g. it would be interesting whether the cluster's average CPU utilization increases (even in cases where limits are not reached).

@szuecs

This comment has been minimized.

Copy link
Member

szuecs commented Jun 5, 2018

The problem with disabling limits is that pods that do not need much resources (requests a few) but use much will be equally treated as pods that requests resources as much as they need.

hjacobs added a commit that referenced this issue Jul 12, 2018

@hjacobs hjacobs assigned aermakov-zalando and unassigned szuecs Jul 19, 2018

aermakov-zalando added a commit that referenced this issue Aug 17, 2018

@hjacobs

This comment has been minimized.

Copy link
Contributor

hjacobs commented Sep 2, 2018

Option to disable CFS quota per cluster was rolled out in #1350

@hjacobs hjacobs closed this Sep 2, 2018

@hjacobs

This comment has been minimized.

Copy link
Contributor

hjacobs commented Nov 6, 2018

We disabled CFS quota for all Zalando Kubernetes clusters and have only positive results so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment