[EKS] [request]: Provide platform version upgrade feature

[dogzzdogzz]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Provide platform version upgrade feature

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We just want to upgrade the EKS platform version for security patch instead of major version upgrade.
From doc it says
Automatic upgrades of existing Amazon EKS platform versions are rolled out incrementally. The roll-out process might take some time. If you need the latest Amazon EKS platform version features immediately, you should create a new Amazon EKS cluster.

I don’t know when the automatic update will be actually performed. For example, the latest platform ver for 1.15 is already eks.4 , however all our 1.15 clusters still stay at eks.2 , there is no automatic upgrade performed.

Even if automatic upgrade is performed, i still prefer manual upgrade, It would be better to let user decide when to upgrade the control plane. Automatic upgrade is kind of risky.

[booleanbetrayal]

A manual upgrade path seems pretty important for environmental consistency, and I assume even more so for organizations with clusters spanning multiple AWS accounts.

[Gowiem]

For one of my clients, I have 5 EKS clusters and I implemented logging utilizing Fargate Logging which was just released in 1.18.9 eks.3. This worked in all clusters except when we finally went to roll this out to our production cluster, we found out that cluster was still on 1.18 eks.2. That threw a wrench into the works and removed our ability to enable logging in our production cluster where logging is likely the most important. I contacted AWS Business Support and they informed me that we have no option to deal with this as the engineering team can't upgrade clusters on request. We then waited over a month to randomly get the upgrade and now it's finally working.

This is a sub-par experience and definitely could be improved. I'd love to see this feature land so I don't get shot in the foot by this in the future. 👍

[goyertp]

Would really appreciate the feature automatic upgrades of existing Amazon EKS platform versions.

[alex-berger]

Zero control over platform version upgrade process poses a substantial operational risk to cluster owners. Today, I realized that our quality assurance (testing) clusters are still running an older platform version than our production clusters, which makes me really nervous. So, far I had not problems, but we all know Murphy's law and one day the platform version might make the difference.

Are there any news on this from AWS?

[jalaziz]

Recently ran into this. Both of our clusters are multiple platform versions behind. As far as I can tell, automatic upgrading doesn't seem to be occurring at any reasonable interval.

[youwalther65]

Faced the same issue. My cluster build in January was still on 1.19 platform v1. Now that I wanted to use the new managed add-on for CoreDNS and kube-rpoxy which requires eks.4 version I got stuck. Having test clusters and pipelines which detailed testing does not help in this case because you can't spinup a test cluster with exactly the same platform version as the production cluster :-(

[alex-berger]

This is really annoying and I raised this with AWS support, with the usual useless (and to be expected) generic no ETA, please up-vote response.
If you have clusters in AWS accounts which are not covered by an AWS Support Plan (e.g. test and development environments) then you are stuck and your only option is to recreate (delete & create) your clusters from scratch, which results in service downtime and wastes your time.

As this issue is still in the early stage (proposed label), all we can do is try to organize as many up-votes as possible.

[jalaziz]

This is really annoying and I raised this with AWS support, with the usual useless (and to be expected) generic no ETA, please up-vote response.
If you have clusters in AWS accounts which are not covered by an AWS Support Plan (e.g. test and development environments) then you are stuck and your only option is to recreate (delete & create) your clusters from scratch, which results in service downtime and wastes your time.

As this issue is still in the early stage (proposed label), all we can do is try to organize as many up-votes as possible.

Even with an AWS Support Plan they don't seem to want (or be able) to do anything about it.

[alex-berger]

@jalaziz Yes, that's what I just learned. That's really ridiculous and I will try to escalate this in order to figure out whether there is really nothing that can be done.

[zhelyan]

Beanstalk has managed updates. You can also update environments manually. You can even rollback platform upgrades on demand.
I'd be happy to see this feature added to EKS

[visit1985]

We would love to test a new Fargate feature, but we are stuck on 1.21 eks.2 till this issue is fixed or 1.22 is released. 😞

[Eslam-mohammed-anwar]

we need to make use of Enabling Windows support but we stuck on 1.21 eks.2

[mumbley]

Just came across this problem while evaluating the EBS snapshot CDR. this is unavailable in 1.21.2 but is available in 1.21.5. So it is a feature we cannot now adopt until we upgrade to 1.22, which is a big change in Kubernetes.

What is even more annoying is that we only stumbled upon this by accident. Having tried to install on a 1.21 build and it failing to find a supported CDR, and given it was a test system, now in an intermediate state, (as all of the pre-upgrade configuration steps had worked fine (and no checklist discussing patch versions)) I blew away the cluster and rebuilt using the exact same terraform only to find the CDR magically appear as a valid configuration. If I had tested this on a fresh deployment and then tried to roll out into production systems, having run all the architecture sign-off, regression testing, change control, development work to integrate the snapshots etc only to find we could not use the feature in production, I may well be recommending next steps to be switching cloud vendors

[ali-parsfar]

I have 2 questions: Is there any possible configuration change in the cluster , that can trigger , the Control-Plane , instance refresh ? Is an instance refresh in the cluster can force the platform update, in the control-plane ?

[visit1985]

Looks like my cluster recently got upgraded from 1.21 eks.2 to 1.21 eks.4 automatically.

The docs state, that this should be the case:

Amazon EKS automatically upgrades all existing clusters to the latest Amazon EKS platform version for their corresponding Kubernetes minor version.

https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html

Interestingly, they also state that eks.4 comes with K8s 1.21.5, but the nodes created on Fargate are still running kubelet version 1.21.2. So maybe that's a typo. 😁

Did anyone come across an announcement for that?

[adelwin]

I have an issue introduced by the platformVersion as well. Specifically the IRSA usage of regional-endpoint.

Same chronological as alot of the guys up there, our production cluster is on a lower platformVersion, so we need to set an environment variable to force the STS to use regional-endpoints.

I can understand that the platformVersion cannot be set during creation, but at least can we get either:

• Clarity of timeline for the automatic upgrade (e.g. 1-4 months after new platformVersion is released)
• Or capability to trigger platformVersion upgrade, even if it's just an imperative commands.

It also doesnt help that the documentation just says:

If you need the latest Amazon EKS platform version features immediately, you should create a new Amazon EKS cluster.

[adelwin]

I have 2 questions: Is there any possible configuration change in the cluster , that can trigger , the Control-Plane , instance refresh ? Is an instance refresh in the cluster can force the platform update, in the control-plane ?

I've tried instance-refresh, didn't help

I've tried eksctl update cluster it just says "no update required"

I've tried update from the UI, also no help.

Cheers.