docs: [RFC] Add v1beta1 API RFC

[jonathan-innis]

Fixes #N/A

Description

This PR proposes the v1beta1 APIs and full Change List into the design directory

How was this change tested?

Does this change impact docs?

• [] Yes, PR includes docs updates
• [] Yes, issue opened: #
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	8184ef1
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/65040439bc3ebd00082ced2f

[ellistarn]

Awesome work on this!!!

[garvinp-stripe]

Great work! I had some time to think, this isn't necessarily a strong opinion but I think there is some value in considering. I don't know what the timeline looks like and maybe what I am thinking doesn't make sense at v1beta1 however I think there is an opportunity to drive some usability improvement since there is going to be significant work in changing the custom resources.

I wanted to see if we can take some inspiration from https://gateway-api.sigs.k8s.io/ which is somewhat new and have been through quite a bit of changes in its overall design. Specifically the way it carved out custom resources that is able to abstract discrete configuration depending on personas. I think this way of thinking can yield quite a bit of insight into how to structure custom resources so that cluster operators can easily provide application developers the appropriate level of control.

Starting with NodePool. I think it has a lot of abstraction leaks between App dev vs Cluster Operator. Ideally app devs can specify their pods -> node, maybe some taints and label for their nodes (pretty much all the parameters in spec). However app dev shouldn't be able to specify kubelet configurations since this is usually managed by cluster operator. I think if possible this can be carved out to something like NodeProviders which are associated to NodePool. Similar to HTTPRoutes to Gateway in GatewayAPI. I think this would significantly improve the flexibility of how cluster operators manage Provisioners/ NodePool since often times there are much more app dev teams to cluster operator teams and most cluster operators are currently creating provisioners per app team.

I think something nice would be "version" similar to LaunchTemplate version. This allows Cluster Operators to rollout and rollback changes to nodes safely and have the necessary controls when managing many NodeClasses. Ideally each NodeClasses version should be immutable. Right now it is difficult to expand on NodeClass for nodes outside of AWS. It may be valuable to split NodeClasses, I don't have a good name, such that we have something that is at the cloud provider level and one that just defines nodes specification specific to a cloud provider. (Similar to gatewayclass and gateway in Gateway API). The Cloud provider layer resource defines the controller for creating nodes for that cloud provider and the nodeclass themselves work in the context of each cloud provider.

Thanks

[jonathan-innis]

I think something nice would be "version" similar to LaunchTemplate version. This allows Cluster Operators to rollout and rollback changes to nodes safely and have the necessary controls when managing many NodeClasses. Ideally each NodeClasses version should be immutable. Right now it is difficult to expand on NodeClass for nodes outside of AWS. It may be valuable to split NodeClasses, I don't have a good name, such that we have something that is at the cloud provider level and one that just defines nodes specification specific to a cloud provider. (Similar to gatewayclass and gateway in Gateway API). The Cloud provider layer resource defines the controller for creating nodes for that cloud provider and the nodeclass themselves work in the context of each cloud provider.

I'm still thinking on the other comment that you made around the Gateway API, but, for this one, the idea was that the NodeClass is the cloudprovider extensibility mechanism; it's effectively the interface to define how Karpenter can actually bring up a Node in a given Cloudprovider. If another Cloudprovider came along and wanted to support Karpenter, I would expect that they would create their own NodeClass and this NodeClass would define its own interface with how to launch Nodes in that Cloudprovider.

Generally, we wanted to avoid creating too many disparate concepts that have to be individually reasoned about (i.e. more API kinds). We effectively have taken the stance that if something is common enough that it should be useful to all cloudproviders, then it should be hoisted up into the NodePool; otherwise, it belongs within the cloudprovider-specific NodeClass.

Is there a specific use-case that you have in mind where you think splitting out the details of the NodeClass could be useful beyond hoisting common cross-cloudprovider values into the NodePool?

[tallaxes]

Same question here, wondering what prompted the reversal ...

[tallaxes]

@jonathan-innis, I am curios what prompted the reversal of the original plan?