CFS quotas can lead to unnecessary throttling

[bobrik]

/kind bug

This is not a bug in Kubernets per se, it's more of a heads-up.

I've read this great blog post:

• https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/

From the blog post I learned that k8s is using cfs quotas to enforce CPU limits. Unfortunately, those can lead to unnecessary throttling, especially for well behaved tenants.

See this unresolved bug in Linux kernel I filed a while back:

• https://bugzilla.kernel.org/show_bug.cgi?id=198197

There's an open and stalled patch that addresses the issue (I've not verified if it works):

• https://lore.kernel.org/patchwork/cover/907448/

cc @ConnorDoyle @balajismaniam

[neolit123]

/sig node
/kind bug

[liggitt]

is this a duplicate of #51135?

[bobrik]

It's similar in spirit, but it seems to miss the fact that there's an actual bug in the kernel rather than just some configuration tradeoff in CFS quota period. I've liked #51135 back here to give people there more context.

[hjacobs]

As far as I understand this is another reason to either disable CFS quota (--cpu-cfs-quota=false) or make it configurable (#63437).

I also find this gist (linked from Kernel patch) very interesting to see (to gauge the impact): https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1

[vishh]

cc @adityakali

[juliantaylor]

Another issue with quotas is that the kubelet counts hyperthreads as "cpus". When a cluster becomes so loaded that two threads are scheduled on the same core and the process has a cpu quota quota one of them will only perform with a small fraction of the available processing power (it will only do something when something on the other thread stalls) but still consume quota as if it had a physical core for itself. So it consumes double the quota that it should without doing significantly more work.
This has the effect that on a fully loaded node node with hyperthreading enabled the performance will be half of what it would be with hyperthreading or quotas disabled.

Imo the kubelet should not consider hyperthreads as real cpus to avoid this situation.

[vishh]

@juliantaylor As I mentioned in #51135 turning off CPU quota might be the best approach for most k8s clusters running trusted workloads.