Use kernel 4.18 in EKS and ECS Amazon Linux AMIs to solve CFS throttling issues.

[willejs]

Apologies as this is kind of an Amazon linux 2 issue, but directly effects EKS and ECS.
Does anyone know when the Linux kernel version 4.18 will be shipped in Amazon Linux 2?
When it is, can the EKS and ECS images be rebuilt?

torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f
The patch above should fix the issue with CFS quotas leading to process throttling. This should enable us to use limits everywhere again without setting off prometheus alerts or degrading performance in latency sensitive components!
kubernetes/kubernetes#67577

[zflamig]

+1 this, would be great for FUSE too that shipped in 4.18.

[BrianChristie]

According to an excellent detailed response from AWS Support, the sched/fair: Fix bandwidth timer clock drift condition patch [0] is merged into the Linux kernel version 4.14.y branch and is available as part of 4.14.95 release [1].

The latest EKS Optimized Amazon Linux 2 AMI has a Linux kernel version which includes the patch.

# uname -a
Linux ip-192-168-213-62.eu-west-1.compute.internal 4.14.97-90.72.amzn2.x86_64 #1 SMP Tue Feb 5 20:46:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Reference:
[0] torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f

[1] Linux kernel patch 4.14.95
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.14.y&ofs=1000

I believe this issue can be closed.

[szymonpk]

I have switched all the nodes to 4.14.104-95.84.amzn2.x86_64, it didn't helped much. I still see throttled containers where CPU usage is minimal, mostly node-exporters.

[mifisignal]

It was subsequently reported that torvalds/linux@512ac99 introduced a regression. A patch to correct this can be found at https://lkml.org/lkml/2019/5/17/581 (not yet merged AFAICT).

[whereisaaron]

Not merged. The author, Dave Chiluk, is looking for support on the LKML to get attention to this patch, to get it reviewed and merged.

[willejs]

It looks like that patch is getting closer to being merged from that thread. I can imagine it's still probably months away from getting into the amazon linux build though.

[whereisaaron]

Reports of great improvements using 4.14.133 with 512ac99 and Dave Chiluk’s patch backported into it: kubernetes/kubernetes#67577 (comment) https://gist.github.com/PaulFurtado/ff6c67ec87416b66ba1c6fc70f7beec1

Hopefully test results like these will help it get merged soon.

[andrew-howden]

In our case we're looking for:

https://github.com/torvalds/linux/commit/79e9fed46038/

As we're exhausting the ephemeral TCP port range in containers.

Also in 4.18.

[hugoprudente]

@andrew-howden,

As per release-notes the Amazon Linux 2, base image for the EKS/ECS Optmized AMI's already have available the kernel 4.19.x[1]

I have upgraded my EKS Opmtized AMI with the 4.19.x kernel to confirm the version/patch available.

$ uname -r
4.19.72-25.58.amzn2.x86_64

With that I was able to confirm that the net-tcp: extend tcp_tw_reuse sysctl to enable loopback only optimization. patch that you required is available already on the 4.19 kernel provided by the AL2

Now is a question of time to the Amazon Linux 2 to start use the Kernel 4.19 as default, or you can use the https://github.com/awslabs/amazon-eks-ami, and build your own AMI with the AWS Supported Kernel as default.

Sadly the Dave Chiluk’s patch is not backported to neither versions of the kernel till now. Altought the version 512ac99 is available on both 4.19, 4.14

Reference
[1]: https://aws.amazon.com/amazon-linux-2/release-notes/