chore: Use policy/v1 instead of policy/v1beta1

[olemarkus]

Description

Switch from policy/v1beta to policy/v1 to support Kubernetes 1.25+.
Kubernetes 1.25 does no longer serve the v1beta1 API and breaks consolidation.

Also bumps Kubernetes libs to 0.22 as 0.21 does not have the types for the entire v1 API. This also brings in upgrade of controller-runtime and knative.

Since policy/v1 requires kubernetes 1.21, users will have to upgrade Kubernetes before upgrading Karpenter.

How was this change tested?

• Unit tests/ginkgo tests
• Manual on a 1.25 cluster

Does this change impact docs?

• Yes, PR includes docs updates

Release Note

action required

Switch usage of policy/v1beta1 to policy/v1
Policy/v1 was introduced in Kubernetes 1.21 and users have to upgrade to a supported version.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	ea5fc35
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/63208842488ea20009716c38

[bwagner5]

we can remove 1.19 from our testing since it is out of the EKS support timeframe. 1.20 is still technically supported until November 2022. https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#:~:text=.-,Amazon%20EKS%20Kubernetes%20release%20calendar,-Note

[olemarkus]

Isn't Karpenter independent from EKS? Being held back to EKS versions is not ideal for other providers and e.g Load Balancer Controller forked in order to support newer versions than EKS does.

AWS CSI drivers, Cloud Controller Manager etc all keep up with the latest Kubernetes versions.

How do we approach this?

[bwagner5]

We're not specifically tied to EKS, but a lot of users on Karpenter today are using EKS so we need to be deliberate on trying not to break current users.

[olemarkus]

I appreciate that. But if they stay on current Karpenter version that will continue to work. And that software keeps requiring newish versions of Kubernetes is nothing new. And as mention, you have the option to fork like Load Balancer Controller did.

But kOps is looking to release 1.25 in the coming weeks, ideally without Karpenter behind a feature flag. And that required K8s 1.25 support. So how can we achieve this?

[ellistarn]

I appreciate that. But if they stay on current Karpenter version that will continue to work. And that software keeps requiring newish versions of Kubernetes is nothing new. And as mention, you have the option to fork like Load Balancer Controller did.
But kOps is looking to release 1.25 in the coming weeks, ideally without Karpenter behind a feature flag. And that required K8s 1.25 support. So how can we achieve this?

If Kubernetes itself does not allow these API versions to coexist, I think likely we need to stay up to date with the latest Kubernetes version on principles of neutrality -- we can't allow EKS to block Kops adoption of Karpenter. Agree with @olemarkus that customers can still run older versions of Karpenter with older versions of Kubernetes. Essentially, we already have an implicit version matrix, and this is the first time we've run into it.

[olemarkus]

Thanks. Yeah policy/v1beta1 is no longer served in 1.25 and no longer exists as part of the k8s.io/api package. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/

Do you want me to do more regarding e.g documentation in this PR?

[olemarkus]

kOps 1.25 is entering beta phase now and this is really blocking. How can we progress this PR?

[jonathan-innis]

Essentially, we already have an implicit version matrix, and this is the first time we've run into it.

We should document this compatibility matrix as part of this PR merge somewhere in the docs similar to how other projects are doing this (Nginx, Istio, etc.)

[olemarkus]

Essentially, we already have an implicit version matrix, and this is the first time we've run into it.

We should document this compatibility matrix as part of this PR merge somewhere in the docs similar to how other projects are doing this (Nginx, Istio, etc.)

That's a good idea. I can add that matrix somewhere after this has merged and we know which specific karpenter version this goes into.

[olemarkus]

How can we progress on this?

[ellistarn]

Hey @olemarkus, sorry for the slow progress here. We've been having some internal discussions about this problem generally and are working on some longer term solutions to avoid this problem in the future. For the short term, we're decided to let this problem solve itself with time given the short timeframe, though we recognize that the underlying issue of version compatibility is unresolved and this may recur.

On November 1 (https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html), this version of Kubernetes is deprecated. For customers scrambling to upgrade their 1.20 clusters, we want to minimize potential impact of an incompatible Karpenter version hindering the upgrade process and ultimately slowing this deprecation. Let's hold this PR until that deprecation occurs, and get it in as soon as possible afterwards.

[Noksa]

Any update? :)

[jonathan-innis]

Any update? :)

We're looking at taking this relatively soon but most likely won't get to the 1.25 upgrade until after the holidays since so many people are out of the office. This is definitely a priority when the majority of the team is back online in the new year now that 1.20 is formally deprecated by EKS

[ellistarn]

Rebased over here: kubernetes-sigs/karpenter#150. @olemarkus, wanna toss a commit on so you get credit?

[olemarkus]

@ellistarn too late I guess.