[EKS] : Reduction in EKS cluster creation time

[kirtichandak]

EKS cluster control plane provisioning time currently averages 15 minutes. We’ll use this issue to track the ongoing improvements we are making to reduce the creation time.

Which service(s) is this request for?
EKS

[heidemn]

Reducing the upgrade time would also be nice, especially since the upgrade today involves several manual steps, see #600 .
When I recently upgraded to 1.15 and 1.16, I remember something like 40, 45min per control plane upgrade until EKS reported that the upgrade had fully finished.

If upgrades could be faster, with the same reliability, this would be great. And it might be even more important than the creation time, assuming that clusters are upgraded several times in their lifetime.

[billinghamj]

Upgrading from 1.18 to 1.19 today took 47 mins according to Terraform logs for the control plane, then 34 mins to upgrade a single very small nodegroup (7 nodes), then you have to manually update core-dns, kube-proxy and aws-cni.

So overall you're talking an absolute minimum of 1.5 hours if you're watching the thing like a hawk and not wasting any time with gaps. This does seem a little crazy and unsustainable :\

I do hope it doesn't keep getting worse with future versions too...

[mreferre]

@billinghamj thanks for the feedback. You are being heard. On a tangent (and not as a mean to respond to your specific question/need) I am wondering if you have considered using Fargate in your EKS deployments. Among other advantages, one thing that in the context of this thread would be interesting is the fact there are no nodes to upgrade and also that AWS embeds, as part of the Fargate, service components you don't need to care about (kubelet/proxy, cni, log routers, etc). In your extremely specific example you would "just" had to upgrade the control plane and core-dns. Just curious if you took Fargate into account and, if you did, what made you stick to "regular" EC2 worker nodes.

[heidemn]

@mreferre in my opinion, running Fargate in EKS is not cost-effective:

• EKS adds overhead cost for the control plane,
• Fargate makes compute power more expensive, compared to EC2.

I don't think there's much of a use case to run the main EKS workloads in Fargate.
Maybe it can be used for small tools (e.g. cron jobs), but not for apps that need a lot of CPU.

Side note: Price reductions of any kind are always welcome :-)

[mreferre]

Thanks for the feedback @heidemn. The raw compute costs of Fargate are (on average) only roughly 20% more expensive than standard EC2 prices (after a consistent price reduction we announced a while back).

Ironically, I think that for very tiny/small workloads, Fargate isn't very cost effective given that the smallest pod size is 0.25vCPU/512MB of memory (and in most cases it would be more convenient to consolidate many tiny workloads on EC2 instances). However for larger workloads, assuming your pod utilization is high, Fargate may become cost effective pretty quickly given there are no worker nodes waste (most K8s clusters are utilized only at a fraction of their full capacity, which you are paying for). If you have a real life example of where you concluded that Fargate was more expensive could you share it please (even offline, I can be reached at mreferre @ amazon dot com).

Also, I did not mean this to become a distraction from the original question in this thread.

Thanks!

PS here a few more considerations around EKS/Fargate economics. I'd like to understand where these assumptions aren't correct (we want to learn more about practical cases where these assumptions are not applicable).

[heidemn]

Thanks, that blog post is definitely helpful. I will give Fargate on EKS a try soon.

To close this (off-)topic:
What I could imagine is that starting a Pod might be slower on Fargate than on EC2 (if the instance is already running).
But the benefits of better isolation and not having to maintain servers are definitely not bad.

[billinghamj]

Our services tend to use around 30MB RAM, are IO bound, and we run hundreds/thousands of them on our non-prod cluster.

Even when using exclusively ARM instance types on spot, due to the pods-per-node limits, EKS is doing pretty poorly for us right now cost-wise (with Rancher, we ran the entire thing on a single instance with no performance issues). Obviously Fargate would exacerbate this massively

Aside from that, we generally are happier having a bit more control, and want as close as possible to a "plain vanilla" K8s setup. History has told us not to trust AWS too much when it comes to behind-the-scenes magic. Before managed spot instances were available, we were quite happy with self-managed nodes too

On principle, lack of ARM support is also a complete non-starter for us. We want to push for that future hard, so we're voting with our feet

[mreferre]

@billinghamj this makes a lot of sense. Thanks for the feedback.

[kirtichandak]

We have now introduced a change for 1.19 clusters that reduces control plane creation time by 40%, enabling you to create an EKS control plane in 9 minutes on an average. The improvement is also coming to other supported Kubernetes versions in a few weeks.

[mveitas]

@kirtichandak Does this just apply to cluster creation or are some of these improvements going to be seen in upgrading a cluster version?

[kirtichandak]

The change to reduce control plane creation time by 40% is now available for all EKS supported versions. This enables you to create an EKS control plane in 9 minutes on an average.

We are currently working on reducing this time further and we'll keep using this issue to track upcoming improvements.

[kkapoor1987]

@kirtichandak Is this improvement specific to certain regions ?