From 77add737e8703f773b3082e878a904250083ca1d Mon Sep 17 00:00:00 2001 From: Chris Negus Date: Wed, 18 Oct 2023 03:56:48 -0400 Subject: [PATCH] docs: Upgrade docs for v1beta1 (#4749) Co-authored-by: Jonathan Innis --- website/content/en/preview/concepts/_index.md | 2 +- .../content/en/preview/contributing/_index.md | 2 +- website/content/en/preview/faq.md | 6 +- .../en/preview/getting-started/_index.md | 4 +- .../content/en/preview/reference/_index.md | 2 +- website/content/en/preview/troubleshooting.md | 2 +- .../content/en/preview/upgrading/_index.md | 10 + .../en/preview/upgrading/compatibility.md | 87 +++ .../preview/{ => upgrading}/upgrade-guide.md | 255 ++++--- .../upgrading/v1beta1-controller-policy.json | 219 ++++++ .../en/preview/upgrading/v1beta1-reference.md | 661 ++++++++++++++++++ website/static/_redirects | 5 + 12 files changed, 1156 insertions(+), 99 deletions(-) create mode 100644 website/content/en/preview/upgrading/_index.md create mode 100644 website/content/en/preview/upgrading/compatibility.md rename website/content/en/preview/{ => upgrading}/upgrade-guide.md (65%) create mode 100644 website/content/en/preview/upgrading/v1beta1-controller-policy.json create mode 100644 website/content/en/preview/upgrading/v1beta1-reference.md diff --git a/website/content/en/preview/concepts/_index.md b/website/content/en/preview/concepts/_index.md index f6bc33096257..67d573135369 100755 --- a/website/content/en/preview/concepts/_index.md +++ b/website/content/en/preview/concepts/_index.md @@ -1,7 +1,7 @@ --- title: "Concepts" linkTitle: "Concepts" -weight: 10 +weight: 20 description: > Understand key concepts of Karpenter --- diff --git a/website/content/en/preview/contributing/_index.md b/website/content/en/preview/contributing/_index.md index 10bb749d39dc..6ec2c3df504e 100644 --- a/website/content/en/preview/contributing/_index.md +++ b/website/content/en/preview/contributing/_index.md @@ -1,7 +1,7 @@ --- title: "Contributing" linkTitle: "Contributing" -weight: 100 +weight: 40 description: > Learn about how to contribute to Karpenter --- diff --git a/website/content/en/preview/faq.md b/website/content/en/preview/faq.md index 09d7d051ee8a..7f91d19a341a 100644 --- a/website/content/en/preview/faq.md +++ b/website/content/en/preview/faq.md @@ -1,7 +1,7 @@ --- title: "FAQs" linkTitle: "FAQs" -weight: 90 +weight: 60 description: > Review Karpenter Frequently Asked Questions --- @@ -34,7 +34,7 @@ Yes, as long as the controller has network and IAM/RBAC access to the Kubernetes ## Compatibility ### Which versions of Kubernetes does Karpenter support? -See the [Compatibility Matrix in the Upgrade Guide]({{< ref "./upgrade-guide#compatibility-matrix" >}}) to view the supported Kubernetes versions per Karpenter released version. +See the [Compatibility Matrix in the Upgrade Section]({{< ref "./upgrading/compatibility#compatibility-matrix" >}}) to view the supported Kubernetes versions per Karpenter released version. ### What Kubernetes distributions are supported? Karpenter documents integration with a fresh or existing installation of the latest AWS Elastic Kubernetes Service (EKS). Other Kubernetes distributions (KOPs, etc.) can be used, but setting up cloud provider permissions for those distributions has not been documented. @@ -197,7 +197,7 @@ To upgrade Karpenter to version `$VERSION`, make sure that the `KarpenterNode IA Next, locate `KarpenterController IAM Role` ARN (i.e., ARN of the resource created in [Create the KarpenterController IAM Role](../getting-started/getting-started-with-karpenter/#create-the-karpentercontroller-iam-role)) and pass them to the helm upgrade command. {{% script file="./content/en/{VERSION}/getting-started/getting-started-with-karpenter/scripts/step08-apply-helm-chart.sh" language="bash"%}} -For information on upgrading Karpenter, see the [Upgrade Guide]({{< ref "./upgrade-guide/" >}}). +For information on upgrading Karpenter, see the [Upgrade Guide]({{< ref "./upgrading/upgrade-guide/" >}}). ## Upgrading Kubernetes Cluster diff --git a/website/content/en/preview/getting-started/_index.md b/website/content/en/preview/getting-started/_index.md index 89a2ba78422c..12fd2a9f9fd2 100644 --- a/website/content/en/preview/getting-started/_index.md +++ b/website/content/en/preview/getting-started/_index.md @@ -1,11 +1,9 @@ --- title: "Getting Started" linkTitle: "Getting Started" -weight: 1 +weight: 10 description: > Choose from different methods to get started with Karpenter -cascade: - type: docs --- diff --git a/website/content/en/preview/reference/_index.md b/website/content/en/preview/reference/_index.md index 92d4249d1678..8fb2918366d8 100644 --- a/website/content/en/preview/reference/_index.md +++ b/website/content/en/preview/reference/_index.md @@ -1,7 +1,7 @@ --- title: "Reference" linkTitle: "Reference" -weight: 100 +weight: 50 description: > Reference documentation for Karpenter --- \ No newline at end of file diff --git a/website/content/en/preview/troubleshooting.md b/website/content/en/preview/troubleshooting.md index 430b2669d5f1..bc50aa8693cc 100644 --- a/website/content/en/preview/troubleshooting.md +++ b/website/content/en/preview/troubleshooting.md @@ -1,7 +1,7 @@ --- title: "Troubleshooting" linkTitle: "Troubleshooting" -weight: 90 +weight: 70 description: > Troubleshoot Karpenter problems --- diff --git a/website/content/en/preview/upgrading/_index.md b/website/content/en/preview/upgrading/_index.md new file mode 100644 index 000000000000..ef1368b22a75 --- /dev/null +++ b/website/content/en/preview/upgrading/_index.md @@ -0,0 +1,10 @@ +--- +title: "Upgrading" +linkTitle: "Upgrading" +weight: 30 +description: > + Upgrading Karpenter guide and reference +cascade: + type: docs +--- + diff --git a/website/content/en/preview/upgrading/compatibility.md b/website/content/en/preview/upgrading/compatibility.md new file mode 100644 index 000000000000..85ce9379e94a --- /dev/null +++ b/website/content/en/preview/upgrading/compatibility.md @@ -0,0 +1,87 @@ +--- +title: "Compatibility" +linkTitle: "Compatibility" +weight: 20 +description: > + Compatibility issues for Karpenter +--- + +# Compatibility + +To make upgrading easier we aim to minimize the introduction of breaking changes. +Before you begin upgrading Karpenter, consider Karpenter compatibility issues related to Kubernetes and the NodePool API (previously Provisioner). + +## Compatibility Matrix + +[comment]: <> (the content below is generated from hack/docs/compatibilitymetrix_gen_docs.go) + +| KUBERNETES | 1.23 | 1.24 | 1.25 | 1.26 | 1.27 | 1.28 | +|------------|---------|---------|---------|---------|---------|---------| +| karpenter | 0.21.x+ | 0.21.x+ | 0.25.x+ | 0.28.x+ | 0.28.x+ | 0.28.x+ | + +[comment]: <> (end docs generated content from hack/docs/compatibilitymetrix_gen_docs.go) + +{{% alert title="Note" color="warning" %}} +Karpenter currently does not support the following [new `topologySpreadConstraints` keys](https://kubernetes.io/blog/2023/04/17/fine-grained-pod-topology-spread-features-beta/), promoted to beta in Kubernetes 1.27: +- `matchLabelKeys` +- `nodeAffinityPolicy` +- `nodeTaintsPolicy` + +For more information on Karpenter's support for these keys, view [this tracking issue](https://github.com/aws/karpenter-core/issues/430). +{{% /alert %}} + +## Compatibility issues + +When we introduce a breaking change, we do so only as described in this document. + +Karpenter follows [Semantic Versioning 2.0.0](https://semver.org/) in its stable release versions, while in +major version zero (v0.y.z) [anything may change at any time](https://semver.org/#spec-item-4). +However, to further protect users during this phase we will only introduce breaking changes in minor releases (releases that increment y in x.y.z). +Note this does not mean every minor upgrade has a breaking change as we will also increment the +minor version when we release a new feature. + +Users should therefore check to see if there is a breaking change every time they are upgrading to a new minor version. + +### How Do We Break Incompatibility? + +When there is a breaking change we will: + +* Increment the minor version when in major version 0 +* Add a permanent separate section named `upgrading to vx.y.z+` under [release upgrade notes](#release-upgrade-notes) + clearly explaining the breaking change and what needs to be done on the user side to ensure a safe upgrade +* Add the sentence “This is a breaking change, please refer to the above link for upgrade instructions” to the top of the release notes and in all our announcements + +### How Do We Find Incompatibilities? + +Besides the peer review process for all changes to the code base we also do the followings in order to find +incompatibilities: +* (To be implemented) To check the compatibility of the application, we will automate tests for installing, uninstalling, upgrading from an older version, and downgrading to an older version +* (To be implemented) To check the compatibility of the documentation with the application, we will turn the commands in our documentation into scripts that we can automatically run + +### Security Patches + +While we are in major version 0 we will not release security patches to older versions. +Rather we provide the patches in the latest versions. +When at major version 1 we will have an EOL (end of life) policy where we provide security patches +for a subset of older versions and deprecate the others. + +## Release Types + +Karpenter offers three types of releases. This section explains the purpose of each release type and how the images for each release type are tagged in our [public image repository](https://gallery.ecr.aws/karpenter). + +### Stable Releases + +Stable releases are the most reliable releases that are released with weekly cadence. Stable releases are our only recommended versions for production environments. +Sometimes we skip a stable release because we find instability or problems that need to be fixed before having a stable release. +Stable releases are tagged with Semantic Versioning. For example `v0.13.0`. + +### Release Candidates + +We consider having release candidates for major and important minor versions. Our release candidates are tagged like `vx.y.z-rc.0`, `vx.y.z-rc.1`. The release candidate will then graduate to `vx.y.z` as a normal stable release. +By adopting this practice we allow our users who are early adopters to test out new releases before they are available to the wider community, thereby providing us with early feedback resulting in more stable releases. + +### Snapshot Releases + +We release a snapshot release for every commit that gets merged into the main repository. This enables our users to immediately try a new feature or fix right after it's merged rather than waiting days or weeks for release. +Snapshot releases are suitable for testing, and troubleshooting but users should exercise great care if they decide to use them in production environments. +Snapshot releases are tagged with the git commit hash prefixed by the Karpenter major version. For example `v0-fc17bfc89ebb30a3b102a86012b3e3992ec08adf`. For more detailed examples on how to use snapshot releases look under "Usage" in [Karpenter Helm Chart](https://gallery.ecr.aws/karpenter/karpenter). diff --git a/website/content/en/preview/upgrade-guide.md b/website/content/en/preview/upgrading/upgrade-guide.md similarity index 65% rename from website/content/en/preview/upgrade-guide.md rename to website/content/en/preview/upgrading/upgrade-guide.md index 487cb1677571..dcb24ee1f0e2 100644 --- a/website/content/en/preview/upgrade-guide.md +++ b/website/content/en/preview/upgrading/upgrade-guide.md @@ -8,121 +8,198 @@ description: > Karpenter is a controller that runs in your cluster, but it is not tied to a specific Kubernetes version, as the Cluster Autoscaler is. Use your existing upgrade mechanisms to upgrade your core add-ons in Kubernetes and keep Karpenter up to date on bug fixes and new features. +This guide contains information needed to upgrade to the latest release of Karpenter, along with compatibility issues you need to be aware of when upgrading from earlier Karpenter versions. -To make upgrading easier we aim to minimize introduction of breaking changes with the following: +### CRD Upgrades -## Compatibility Matrix +Karpenter ships with a few Custom Resource Definitions (CRDs). These CRDs are published: +* As an independent helm chart [karpenter-crd](https://gallery.ecr.aws/karpenter/karpenter-crd) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter-crd) that can be used by Helm to manage the lifecycle of these CRDs. + * To upgrade or install `karpenter-crd` run: + ``` + helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version vx.y.z --namespace karpenter --create-namespace + ``` -[comment]: <> (the content below is generated from hack/docs/compataiblitymetrix_gen_docs.go) +{{% alert title="Note" color="warning" %}} +If you get the error `invalid ownership metadata; label validation error:` while installing the `karpenter-crd` chart from an older version of Karpenter, follow the [Troubleshooting Guide]({{}}) for details on how to resolve these errors. +{{% /alert %}} -| KUBERNETES | 1.23 | 1.24 | 1.25 | 1.26 | 1.27 | 1.28 | -|------------|---------|---------|---------|---------|---------|--------| -| karpenter | 0.21.x+ | 0.21.x+ | 0.25.x+ | 0.28.x+ | 0.28.x+ | 0.31.x | +* As part of the helm chart [karpenter](https://gallery.ecr.aws/karpenter/karpenter) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter/crds). Helm [does not manage the lifecycle of CRDs using this method](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/), the tool will only install the CRD during the first installation of the helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed. When CRDs are changed, we will make a note in the version's upgrade guide. -[comment]: <> (end docs generated content from hack/docs/compataiblitymetrix_gen_docs.go) +In general, you can reapply the CRDs in the `crds` directory of the Karpenter helm chart: -{{% alert title="Note" color="warning" %}} -Karpenter currently does not support the following [new `topologySpreadConstraints` keys](https://kubernetes.io/blog/2023/04/17/fine-grained-pod-topology-spread-features-beta/), promoted to beta in Kubernetes 1.27: -- `matchLabelKeys` -- `nodeAffinityPolicy` -- `nodeTaintsPolicy` +```shell +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodepols.yaml +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodeclaims.yaml +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml +``` +g +### Upgrading to v0.32.0+ -For more information on Karpenter's support for these keys, view [this tracking issue](https://github.com/aws/karpenter-core/issues/430). -{{% /alert %}} +#### v1beta1 Migration -## Compatibility issues +Here is some information you should know about upgrading the Karpenter controller to v0.32.x: -To make upgrading easier, we aim to minimize the introduction of breaking changes with the following components: +* **Towards a v1 release**: The latest version of Karpenter sets the stage for Karpenter v1. Karpenter v0.32.x implements the Karpenter v1beta1 API spec. The intention is to have v1beta1 be used as the v1 spec, with only minimal changes needed. +* **Path to upgrading**: This procedure assumes that you are upgrading from Karpenter v0.31.x to v0.32.x. If you are on an earlier version of Karpenter, review the [Release Upgrade Notes]({{< relref "#release-upgrade-notes" >}}) for earlier versions' breaking changes. +* **Enhancing and renaming components**: For v1beta1, APIs have been enhanced to improve and solidify Karpenter APIs. Part of these enhancements includes renaming the Kinds for all Karpenter CustomResources. The following name changes have been made: + * Provisioner -> NodePool + * Machine -> NodeClaim + * AWSNodeTemplate -> EC2NodeClass +* **Running v1alpha1 alongside v1beta1**: Having different Kind names for v1alpha5 and v1beta1 allows them to coexist for the same Karpenter controller for v0.32.x. This gives you time to transition to the new v1beta1 APIs while existing Provisioners and other objects stay in place. Keep in mind that there is no guarantee that the two versions will be able to coexist in future Karpenter versions. -* NodePool API -* EC2NodeClass API -* Helm Chart +Some things that will help you with this upgrade include: -We try to maintain compatibility with: +* **[v1beta1 Upgrade Reference]({{< relref "v1beta1-reference" >}})**: Provides a complete reference to help you transition your Provisioner, Machine, and AWSNodeTemplate manifests, as well as other components, to be able to work with the new v1beta1 names, labels, and other elements. +* **[Karpenter conversion tool](https://github.com/aws/karpenter/tree/main/tools/karpenter-convert)**: Simplifies the creation of NodePool and EC2NodeClass manifests. -* The application itself -* The documentation of the application +##### Procedure -When we introduce a breaking change, we do so only as described in this document. +This procedure assumes you are running the Karpenter controller on cluster and want to upgrade that cluster to v0.32.x. -Karpenter follows [Semantic Versioning 2.0.0](https://semver.org/) in its stable release versions, while in -major version zero (v0.y.z) [anything may change at any time](https://semver.org/#spec-item-4). -However, to further protect users during this phase we will only introduce breaking changes in minor releases (releases that increment y in x.y.z). -Note this does not mean every minor upgrade has a breaking change as we will also increment the -minor version when we release a new feature. +**NOTE**: Please read through the entire procedure before beginning the upgrade. There are major changes in this upgrade, so you should carefully evaluate your cluster and workloads before proceeding. -Users should therefore check to see if there is a breaking change every time they are upgrading to a new minor version. -### Custom Resource Definition (CRD) Upgrades +#### Prerequisites -Karpenter ships with a few Custom Resource Definitions (CRDs). These CRDs are published: -* As an independent helm chart [karpenter-crd](https://gallery.ecr.aws/karpenter/karpenter-crd) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter-crd) that can be used by Helm to manage the lifecycle of these CRDs. - * To upgrade or install `karpenter-crd` run: +To upgrade your provisioner and AWSNodeTemplate YAML files to be compatible with v1beta1, you can either update them manually or use the [karpenter-convert](https://github.com/aws/karpenter/tree/main/tools/karpenter-convert) CLI tool. To install that tool: + +``` +go install github.com/aws/karpenter/tools/karpenter-convert/cmd/karpenter-convert@latest +``` +Add `~/go/bin` to your $PATH, if you have not already done so. + +1. Determine the current cluster version: Run the following to make sure that your Karpenter version is v0.31.x: + ``` + kubectl get pod -A | grep karpenter + kubectl describe pod -n karpenter karpenter-xxxxxxxxxx-xxxxx | grep Image: | grep v0..... + ``` + Sample output: + ``` + Image: public.ecr.aws/karpenter/controller:v0.31.0@sha256:d29767fa9c5c0511a3812397c932f5735234f03a7a875575422b712d15e54a77 + ``` + + {{% alert title="Note" color="primary" %}} + v0.31.2 introduces minor changes to Karpenter so that rollback from v0.32.0 is supported. If you are coming from some other patch version of minor version v0.31.x, note that v0.31.2 is the _only_ patch version that supports rollback. + {{% /alert %}} + +2. Review for breaking changes: If you are already running Karpenter v0.31.x, you can skip this step. If you are running an earlier Karpenter version, you need to review the upgrade notes for each minor release. + +3. Set environment variables for your cluster: + + ```bash + export KARPENTER_VERSION=v0.32.0 + export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov + export CLUSTER_NAME="${USER}-karpenter-demo" + export AWS_REGION="us-west-2" + export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" + export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter" + export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)" + ``` + +4. Apply the new Karpenter policy and assign it to the existing Karpenter role: + + ```bash + TEMPOUT=$(mktemp) + curl -fsSL https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}website/content/en/preview/upgrade/v1beta1-controller-policy.json > ${TEMPOUT} + + POLICY_DOCUMENT=$(envsubst < ${TEMPOUT}) + POLICY_NAME="KarpenterControllerPolicy-${CLUSTER_NAME}-v1beta1" + ROLE_NAME="${CLUSTER_NAME}-karpenter" + + POLICY_ARN=$(aws iam create-policy --policy-name "${POLICY_NAME}" --policy-document "${POLICY_DOCUMENT}" | jq -r .Policy.Arn) + aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}" + ``` + +5. Apply the v0.32.0 Custom Resource Definitions (CRDs) in the crds directory of the Karpenter helm chart. Here are the ways you can do this: + + * As an independent helm chart [karpenter-crd](https://gallery.ecr.aws/karpenter/karpenter-crd) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter-crd) that can be used by Helm to manage the lifecycle of these CRDs. To upgrade or install `karpenter-crd` run: ``` helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version vx.y.z --namespace karpenter --create-namespace ``` -{{% alert title="Note" color="warning" %}} -If you get the error `invalid ownership metadata; label validation error:` while installing the `karpenter-crd` chart from an older version of Karpenter, follow the [Troubleshooting Guide]({{}}) for details on how to resolve these errors. -{{% /alert %}} + {{% alert title="Note" color="warning" %}} + < If you get the error `invalid ownership metadata; label validation error:` while installing the `karpenter-crd` chart from an older version of Karpenter, follow the [Troubleshooting Guide]({{}}) for details on how to resolve these errors. + {{% /alert %}} -* As part of the helm chart [karpenter](https://gallery.ecr.aws/karpenter/karpenter) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter/crds). Helm [does not manage the lifecycle of CRDs using this method](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/), the tool will only install the CRD during the first installation of the helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed. When CRDs are changed, we will make a note in the version's upgrade guide. + * As part of the helm chart [karpenter](https://gallery.ecr.aws/karpenter/karpenter) - [source](https://github.com/aws/karpenter/blob/main/charts/karpenter/crds). Helm [does not manage the lifecycle of CRDs using this method](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/), the tool will only install the CRD during the first installation of the helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed. When CRDs are changed, we will make a note in the version's upgrade guide. In general, ou can reapply the CRDs in the `crds` directory of the Karpenter helm chart: -In general, you can reapply the CRDs in the `crds` directory of the Karpenter helm chart: + ```bash + kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodepools.yaml + kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodeclaims.yaml + kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml + ``` +6. Upgrade Karpenter to the new version: + + ```bash + helm registry logout public.ecr.aws + + helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \ + --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ + --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \ + --set settings.aws.clusterName=${CLUSTER_NAME} \ + --set settings.aws.interruptionQueueName=${CLUSTER_NAME} \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + --wait + ``` -```shell -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodepols.yaml -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.sh_nodeclaims.yaml -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml -``` +7. Convert each AWSNodeTemplate to an EC2NodeClass. To convert your v1alpha Karpenter manifests to v1beta1, you can either manually apply changes to API components or use the [Karpenter conversion tool](https://github.com/aws/karpenter/tree/main/tools/karpenter-convert). + See the [AWSNodeTemplate to EC2NodeClass]({{< relref "v1beta1-reference#awsnodetemplate-to-ec2nodeclass" >}}) section of the Karpenter Upgrade Reference for details on how to update to Karpenter AWSNodeTemplate objects. Here is an example of how to use the `karpenter-convert` CLI to convert an AWSNodeTemplate file to a EC2NodeClass file: -### How Do We Break Incompatibility? + ```bash + karpenter-convert -f awsnodetemplate.yaml > ec2nodeclass.yaml + ``` -When there is a breaking change we will: +8. Edit the converted EC2NodeClass file manually: -* Increment the minor version when in major version 0 -* Add a permanent separate section named `upgrading to vx.y.z+` under [released upgrade notes](#released-upgrade-notes) - clearly explaining the breaking change and what needs to be done on the user side to ensure a safe upgrade -* Add the sentence “This is a breaking change, please refer to the above link for upgrade instructions” to the top of the release notes and in all our announcements + * Specify your AWS role where there is a `$KARPENTER_NODE_ROLE` placeholder. For example, if you created your cluster using the [Getting Started with Karpenter](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/) guide, you would use the name `KarpenterNodeRole-$CLUSTER_NAME`, substituting your cluster name for `$CLUSTER_NAME`. + * Otherwise, check the file for accuracy. -### How Do We Find Incompatibilities? +9. When you are satisfied with your EC2NodeClass file, apply it as follows: -Besides the peer review process for all changes to the code base we also do the followings in order to find -incompatibilities: -* (To be implemented) To check the compatibility of the application, we will automate tests for installing, uninstalling, upgrading from an older version, and downgrading to an older version -* (To be implemented) To check the compatibility of the documentation with the application, we will turn the commands in our documentation into scripts that we can automatically run + ```bash + kubectl apply -f ec2nodeclass.yaml + ``` -### Security Patches +10. Convert each Provisioner to a NodePool. Again, either manually update your Provisioner manifests or use the karpenter-convert CLI tool: -While we are in major version 0 we will not release security patches to older versions. -Rather we provide the patches in the latest versions. -When at major version 1 we will have an EOL (end of life) policy where we provide security patches -for a subset of older versions and deprecate the others. + ```bash + karpenter-convert -f provisioner.yaml > nodepool.yaml + ``` -## Release Types +11. When you are satisfied with your NodePool file, apply it as follows: -Karpenter offers three types of releases. This section explains the purpose of each release type and how the images for each release type are tagged in our [public image repository](https://gallery.ecr.aws/karpenter). + ```bash + kubectl apply -f nodepool.yaml + ``` -### Stable Releases +12. Roll over nodes: With the new NodePool yaml in hand, there are several ways you can begin to roll over your nodes to use the new NodePool: -Stable releases are the most reliable releases that are released with weekly cadence. Stable releases are our only recommended versions for production environments. -Sometimes we skip a stable release because we find instability or problems that need to be fixed before having a stable release. -Stable releases are tagged with Semantic Versioning. For example `v0.13.0`. + * Periodic Rolling with [Drift]({{< relref "../concepts/disruption#drift" >}}): Enable [drift]({{< relref "../concepts/disruption#drift" >}}) in your NodePool file, then do the following: + - Add the following taint to the old Provisioner: `karpenter.sh/legacy=true:NoSchedule` + - Wait as Karpenter marks all machines owned by that Provisioner as having drifted. + - Watch as replacement nodes are launched from the new NodePool resource. -### Release Candidates + Because Karpenter will only roll of one node at a time, it may take some time for Karpenter to completely roll all nodes under a Provisioner. -We consider having release candidates for major and important minor versions. Our release candidates are tagged like `vx.y.z-rc.0`, `vx.y.z-rc.1`. The release candidate will then graduate to `vx.y.z` as a normal stable release. -By adopting this practice we allow our users who are early adopters to test out new releases before they are available to the wider community, thereby providing us with early feedback resulting in more stable releases. + * Forced Deletion: For each Provisioner in your cluster: -### Snapshot Releases + - Delete the old Provisioner with: `kubectl delete provisioner --cascade=foreground` + - Wait as Karpenter deletes all the Provisioner's nodes. All nodes will drain simultaneously. New nodes are launched after the old ones have been drained. -We release a snapshot release for every commit that gets merged into the main repository. This enables our users to immediately try a new feature or fix right after it's merged rather than waiting days or weeks for release. -Snapshot releases are suitable for testing, and troubleshooting but users should exercise great care if they decide to use them in production environments. -Snapshot releases are tagged with the git commit hash prefixed by the Karpenter major version. For example `v0-fc17bfc89ebb30a3b102a86012b3e3992ec08adf`. For more detailed examples on how to use snapshot releases look under "Usage" in [Karpenter Helm Chart](https://gallery.ecr.aws/karpenter/karpenter). + * Manual Rolling: For each Provisioner in your cluster: + - Add the following taint to the old Provisioner: `karpenter.sh/legacy=true:NoSchedule` + - For all the nodes owned by the Provisioner, delete one at a time as follows: `kubectl delete node ` -## Released Upgrade Notes +13. Update workload labels: Old v1alpha labels (`karpenter.sh/do-not-consolidate` and `karpenter.sh/do-not-evict`) are deprecated, but will not be dropped until Karpenter v1. However, you can begin updating those labels at any time with `karpenter.sh/do-not-disrupt`. You should check that there are no more Provisioner, AWSNodeTemplate, or Machine resources on your cluster. at which time you can delete the old CRDs. To validate that there are no more machines, type: -### Upgrading to v0.32.0+ + ```bash + kubectl get machines + ``` + +#### Additional Release Notes * Karpenter now serves the webhook prometheus metrics server on port `8001`. If this port is already in-use on the pod or you are running in `hostNetworking` mode, you may need to change this port value. You can configure this port value through the `WEBHOOK_METRICS_PORT` environment variable or the `webhook.metrics.port` value if installing via Helm. * Karpenter now exposes the ability to disable webhooks through the `webhook.enabled=false` value. This value will disable the webhook server and will prevent any permissions, mutating or validating webhook configurations from being deployed to the cluster. @@ -133,7 +210,7 @@ Snapshot releases are tagged with the git commit hash prefixed by the Karpenter ### Upgrading to v0.30.0+ -* Karpenter will now [statically drift]({{}}) on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the `karpenter.sh/provisioner-hash` annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the `karpenter.k8s.aws/nodetemplate-hash` annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added. +* Karpenter will now [statically drift]({{}}) on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the `karpenter.sh/provisioner-hash` annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the `karpenter.k8s.aws/nodetemplate-hash` annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added. * Karpenter will now fail validation on AWSNodeTemplates and Provisioner `spec.provider` that have `amiSelectors`, `subnetSelectors`, or `securityGroupSelectors` set with a combination of id selectors (`aws-ids`, `aws::ids`) and other selectors. * Karpenter now statically sets the `securityContext` at both the pod and container-levels and doesn't allow override values to be passed through the helm chart. This change was made to adhere to [Restricted Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted), which follows pod hardening best practices. @@ -150,7 +227,7 @@ Karpenter `v0.29.1` contains a [file descriptor and memory leak bug](https://git * Karpenter has changed the default metrics service port from 8080 to 8000 and the default webhook service port from 443 to 8443. In `v0.28.0`, the Karpenter pod port was changed to 8000, but referenced the service by name, allowing users to scrape the service at port 8080 for metrics. `v0.29.0` aligns the two ports so that service and pod metrics ports are the same. These ports are set by the `controller.metrics.port` and `webhook.port` helm chart values, so if you have previously set these to non-default values, you may need to update your Prometheus scraper to match these new values. * Karpenter will now reconcile nodes that are drifted due to their Security Groups or their Subnets. If your AWSNodeTemplate's Security Groups differ from the Security Groups used for an instance, Karpenter will consider it drifted. If the Subnet used by an instance is not contained in the allowed list of Subnets for an AWSNodeTemplate, Karpenter will also consider it drifted. - * Since Karpenter uses tags for discovery of Subnets and SecurityGroups, check the [Threat Model]({{}}) to see how to manage this IAM Permission. + * Since Karpenter uses tags for discovery of Subnets and SecurityGroups, check the [Threat Model]({{}}) to see how to manage this IAM Permission. ### Upgrading to v0.28.0+ @@ -159,8 +236,8 @@ Karpenter `v0.28.0` is incompatible with Kubernetes version 1.26+, which can res {{% /alert %}} * The `extraObjects` value is now removed from the Helm chart. Having this value in the chart proved to not work in the majority of Karpenter installs and often led to anti-patterns, where the Karpenter resources installed to manage Karpenter's capacity were directly tied to the install of the Karpenter controller deployments. The Karpenter team recommends that, if you want to install Karpenter manifests alongside the Karpenter helm chart, to do so by creating a separate chart for the manifests, creating a dependency on the controller chart. -* The `aws.nodeNameConvention` setting is now removed from the [`karpenter-global-settings`]({{}}) ConfigMap. Because Karpenter is now driving its orchestration of capacity through Machines, it no longer needs to know the node name, making this setting obsolete. Karpenter ignores configuration that it doesn't recognize in the [`karpenter-global-settings`]({{}}) ConfigMap, so leaving the `aws.nodeNameConvention` in the ConfigMap will simply cause this setting to be ignored. -* Karpenter now defines a set of "restricted tags" which can't be overridden with custom tagging in the AWSNodeTemplate or in the [`karpenter-global-settings`]({{}}) ConfigMap. If you are currently using any of these tag overrides when tagging your instances, webhook validation will now fail. These tags include: +* The `aws.nodeNameConvention` setting is now removed from the [`karpenter-global-settings`]({{}}) ConfigMap. Because Karpenter is now driving its orchestration of capacity through Machines, it no longer needs to know the node name, making this setting obsolete. Karpenter ignores configuration that it doesn't recognize in the [`karpenter-global-settings`]({{}}) ConfigMap, so leaving the `aws.nodeNameConvention` in the ConfigMap will simply cause this setting to be ignored. +* Karpenter now defines a set of "restricted tags" which can't be overridden with custom tagging in the AWSNodeTemplate or in the [`karpenter-global-settings`]({{}}) ConfigMap. If you are currently using any of these tag overrides when tagging your instances, webhook validation will now fail. These tags include: * `karpenter.sh/managed-by` * `karpenter.sh/provisioner-name` @@ -182,7 +259,7 @@ Karpenter creates a mapping between CloudProvider machines and CustomResources i * `karpenter.sh/provisioner-name` * `kubernetes.io/cluster/${CLUSTER_NAME}` -Because Karpenter takes this dependency, any user that has the ability to Create/Delete these tags on CloudProvider machines will have the ability to orchestrate Karpenter to Create/Delete CloudProvider machines as a side effect. Check the [Threat Model]({{}}) to see how this might affect you, and ways to mitigate this. +Because Karpenter takes this dependency, any user that has the ability to Create/Delete these tags on CloudProvider machines will have the ability to orchestrate Karpenter to Create/Delete CloudProvider machines as a side effect. Check the [Threat Model]({{}}) to see how this might affect you, and ways to mitigate this. {{% /alert %}} {{% alert title="Rolling Back" color="warning" %}} @@ -217,7 +294,7 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh * The `karpenter_allocation_controller_scheduling_duration_seconds` metric name changed to `karpenter_provisioner_scheduling_duration_seconds` ### Upgrading to v0.26.0+ -* The `karpenter.sh/do-not-evict` annotation no longer blocks node termination when running `kubectl delete node`. This annotation on pods will only block automatic deprovisioning that is considered "voluntary," that is, disruptions that can be avoided. Disruptions that Karpenter deems as "involuntary" and will ignore the `karpenter.sh/do-not-evict` annotation include spot interruption and manual deletion of the node. See [Disabling Deprovisioning]({{}}) for more details. +* The `karpenter.sh/do-not-evict` annotation no longer blocks node termination when running `kubectl delete node`. This annotation on pods will only block automatic deprovisioning that is considered "voluntary," that is, disruptions that can be avoided. Disruptions that Karpenter deems as "involuntary" and will ignore the `karpenter.sh/do-not-evict` annotation include spot interruption and manual deletion of the node. See [Disabling Deprovisioning]({{}}) for more details. * Default resources `requests` and `limits` are removed from the Karpenter's controller deployment through the Helm chart. If you have not set custom resource `requests` or `limits` in your helm values and are using Karpenter's defaults, you will now need to set these values in your helm chart deployment. * The `controller.image` value in the helm chart has been broken out to a map consisting of `controller.image.repository`, `controller.image.tag`, and `controller.image.digest`. If manually overriding the `controller.image`, you will need to update your values to the new design. @@ -225,9 +302,9 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh * Cluster Endpoint can now be automatically discovered. If you are using Amazon Elastic Kubernetes Service (EKS), you can now omit the `clusterEndpoint` field in your configuration. In order to allow the resolving, you have to add the permission `eks:DescribeCluster` to the Karpenter Controller IAM role. ### Upgrading to v0.24.0+ -* Settings are no longer updated dynamically while Karpenter is running. If you manually make a change to the [`karpenter-global-settings`]({{}}) ConfigMap, you will need to reload the containers by restarting the deployment with `kubectl rollout restart -n karpenter deploy/karpenter` -* Karpenter no longer filters out instance types internally. Previously, `g2` (not supported by the NVIDIA device plugin) and FPGA instance types were filtered. The only way to filter instance types now is to set requirements on your provisioner or pods using well-known node labels described [here]({{}}). If you are currently using overly broad requirements that allows all of the `g` instance-category, you will want to tighten the requirement, or add an instance-generation requirement. -* `aws.tags` in [`karpenter-global-settings`]({{}}) ConfigMap is now a top-level field and expects the value associated with this key to be a JSON object of string to string. This is change from previous versions where keys were given implicitly by providing the key-value pair `aws.tags.: value` in the ConfigMap. +* Settings are no longer updated dynamically while Karpenter is running. If you manually make a change to the [`karpenter-global-settings`]({{}}) ConfigMap, you will need to reload the containers by restarting the deployment with `kubectl rollout restart -n karpenter deploy/karpenter` +* Karpenter no longer filters out instance types internally. Previously, `g2` (not supported by the NVIDIA device plugin) and FPGA instance types were filtered. The only way to filter instance types now is to set requirements on your provisioner or pods using well-known node labels described [here]({{}}). If you are currently using overly broad requirements that allows all of the `g` instance-category, you will want to tighten the requirement, or add an instance-generation requirement. +* `aws.tags` in [`karpenter-global-settings`]({{}}) ConfigMap is now a top-level field and expects the value associated with this key to be a JSON object of string to string. This is change from previous versions where keys were given implicitly by providing the key-value pair `aws.tags.: value` in the ConfigMap. ### Upgrading to v0.22.0+ * Do not upgrade to this version unless you are on Kubernetes >= v1.21. Karpenter no longer supports Kubernetes v1.20, but now supports Kubernetes v1.25. This change is due to the v1 PDB API, which was introduced in K8s v1.20 and subsequent removal of the v1beta1 API in K8s v1.25. @@ -237,11 +314,11 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh ### Upgrading to v0.19.0+ * The karpenter webhook and controller containers are combined into a single binary, which requires changes to the helm chart. If your Karpenter installation (helm or otherwise) currently customizes the karpenter webhook, your deployment tooling may require minor changes. -* Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling `aws.interruptionQueueName`. For more details on Karpenter's interruption handling, see the [Interruption Handling Docs]({{< ref "./concepts/disruption/#interruption" >}}). For common questions on the migration process, see the [FAQ]({{< ref "./faq/#interruption-handling" >}}) +* Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling `aws.interruptionQueueName`. For more details on Karpenter's interruption handling, see the [Interruption Handling Docs]({{< ref "../concepts/disruption/#interruption" >}}). * Instance category defaults are now explicitly persisted in the Provisioner, rather than handled implicitly in memory. By default, Provisioners will limit instance category to c,m,r. If any instance type constraints are applied, it will override this default. If you have created Provisioners in the past with unconstrained instance type, family, or category, Karpenter will now more flexibly use instance types than before. If you would like to apply these constraints, they must be included in the Provisioner CRD. * Karpenter CRD raw YAML URLs have migrated from `https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}charts/karpenter/crds/...` to `https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/...`. If you reference static Karpenter CRDs or rely on `kubectl replace -f` to apply these CRDs from their remote location, you will need to migrate to the new location. * Pods without an ownerRef (also called "controllerless" or "naked" pods) will now be evicted by default during node termination and consolidation. Users can prevent controllerless pods from being voluntarily disrupted by applying the `karpenter.sh/do-not-evict: "true"` annotation to the pods in question. -* The following CLI options/environment variables are now removed and replaced in favor of pulling settings dynamically from the [`karpenter-global-settings`]({{}}) ConfigMap. See the [Settings docs]({{}}) for more details on configuring the new values in the ConfigMap. +* The following CLI options/environment variables are now removed and replaced in favor of pulling settings dynamically from the [`karpenter-global-settings`]({{}}) ConfigMap. See the [Settings docs]({{}}) for more details on configuring the new values in the ConfigMap. * `CLUSTER_NAME` -> `settings.aws.clusterName` * `CLUSTER_ENDPOINT` -> `settings.aws.clusterEndpoint` @@ -253,11 +330,11 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh * `VM_MEMORY_OVERHEAD` -> `settings.aws.vmMemoryOverheadPercent` ### Upgrading to v0.18.0+ -* v0.18.0 removes the `karpenter_consolidation_nodes_created` and `karpenter_consolidation_nodes_terminated` prometheus metrics in favor of the more generic `karpenter_nodes_created` and `karpenter_nodes_terminated` metrics. You can still see nodes created and terminated by consolidation by checking the `reason` label on the metrics. Check out all the metrics published by Karpenter [here]({{}}). +* v0.18.0 removes the `karpenter_consolidation_nodes_created` and `karpenter_consolidation_nodes_terminated` prometheus metrics in favor of the more generic `karpenter_nodes_created` and `karpenter_nodes_terminated` metrics. You can still see nodes created and terminated by consolidation by checking the `reason` label on the metrics. Check out all the metrics published by Karpenter [here]({{}}). ### Upgrading to v0.17.0+ Karpenter's Helm chart package is now stored in [Karpenter's OCI (Open Container Initiative) registry](https://gallery.ecr.aws/karpenter/karpenter). The Helm CLI supports the new format since [v3.8.0+](https://helm.sh/docs/topics/registries/). -With this change [charts.karpenter.sh](https://charts.karpenter.sh/) is no longer updated but preserved to allow using older Karpenter versions. For examples on working with the Karpenter helm charts look at [Install Karpenter Helm Chart]({{< ref "./getting-started/getting-started-with-karpenter/#install-karpenter-helm-chart" >}}). +With this change [charts.karpenter.sh](https://charts.karpenter.sh/) is no longer updated but preserved to allow using older Karpenter versions. For examples on working with the Karpenter helm charts look at [Install Karpenter Helm Chart]({{< ref "../getting-started/getting-started-with-karpenter/#install-karpenter-helm-chart" >}}). Users who have scripted the installation or upgrading of Karpenter need to adjust their scripts with the following changes: 1. There is no longer a need to add the Karpenter helm repo to helm @@ -307,13 +384,13 @@ aws ec2 delete-launch-template --launch-template-id operator: Exists ``` -* v0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the `ec2:DescribeImages` permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the [Custom AMI documentation here]({{}}) to get started +* v0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the `ec2:DescribeImages` permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the [Custom AMI documentation here]({{}}) to get started * v0.14.0 adds an an additional default toleration (CriticalAddonOnly=Exists) to the Karpenter helm chart. This may cause Karpenter to run on nodes with that use this Taint which previously would not have been schedulable. This can be overridden by using `--set tolerations[0]=null`. * v0.14.0 deprecates the `AWS_ENI_LIMITED_POD_DENSITY` environment variable in-favor of specifying `spec.kubeletConfiguration.maxPods` on the Provisioner. `AWS_ENI_LIMITED_POD_DENSITY` will continue to work when `maxPods` is not set on the Provisioner. If `maxPods` is set, it will override `AWS_ENI_LIMITED_POD_DENSITY` on that specific Provisioner. ### Upgrading to v0.13.0+ -* v0.13.0 introduces a new CRD named `AWSNodeTemplate` which can be used to specify AWS Cloud Provider parameters. Everything that was previously specified under `spec.provider` in the Provisioner resource, can now be specified in the spec of the new resource. The use of `spec.provider` is deprecated but will continue to function to maintain backwards compatibility for the current API version (v1alpha5) of the Provisioner resource. v0.13.0 also introduces support for custom user data that doesn't require the use of a custom launch template. The user data can be specified in-line in the AWSNodeTemplate resource. Read the [UserData documentation here](../aws/operating-systems) to get started. +* v0.13.0 introduces a new CRD named `AWSNodeTemplate` which can be used to specify AWS Cloud Provider parameters. Everything that was previously specified under `spec.provider` in the Provisioner resource, can now be specified in the spec of the new resource. The use of `spec.provider` is deprecated but will continue to function to maintain backwards compatibility for the current API version (v1alpha5) of the Provisioner resource. v0.13.0 also introduces support for custom user data that doesn't require the use of a custom launch template. The user data can be specified in-line in the AWSNodeTemplate resource. If you are upgrading from v0.10.1 - v0.11.1, a new CRD `awsnodetemplate` was added. In v0.12.0, this crd was renamed to `awsnodetemplates`. Since helm does not manage the lifecycle of CRDs, you will need to perform a few manual steps for this CRD upgrade: 1. Make sure any `awsnodetemplate` manifests are saved somewhere so that they can be reapplied to the cluster. diff --git a/website/content/en/preview/upgrading/v1beta1-controller-policy.json b/website/content/en/preview/upgrading/v1beta1-controller-policy.json new file mode 100644 index 000000000000..e6923be897d2 --- /dev/null +++ b/website/content/en/preview/upgrading/v1beta1-controller-policy.json @@ -0,0 +1,219 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowScopedEC2InstanceActions", + "Effect": "Allow", + "Resource": [ + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}::image/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}::snapshot/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:spot-instances-request/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:security-group/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:subnet/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*" + ], + "Action": [ + "ec2:RunInstances", + "ec2:CreateFleet" + ] + }, + { + "Sid": "AllowScopedEC2InstanceActionsWithTags", + "Effect": "Allow", + "Resource": [ + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:fleet/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:volume/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:network-interface/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*" + ], + "Action": [ + "ec2:RunInstances", + "ec2:CreateFleet", + "ec2:CreateLaunchTemplate" + ], + "Condition": { + "StringEquals": { + "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned" + }, + "StringLike": { + "aws:RequestTag/karpenter.sh/nodepool": "*" + } + } + }, + { + "Sid": "AllowScopedResourceCreationTagging", + "Effect": "Allow", + "Resource": [ + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:fleet/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:volume/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:network-interface/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*" + ], + "Action": "ec2:CreateTags", + "Condition": { + "StringEquals": { + "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", + "ec2:CreateAction": [ + "RunInstances", + "CreateFleet", + "CreateLaunchTemplate" + ] + }, + "StringLike": { + "aws:RequestTag/karpenter.sh/nodepool": "*" + } + } + }, + { + "Sid": "AllowScopedResourceTagging", + "Effect": "Allow", + "Resource": "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*", + "Action": "ec2:CreateTags", + "Condition": { + "StringEquals": { + "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned" + }, + "StringLike": { + "aws:ResourceTag/karpenter.sh/nodepool": "*" + }, + "ForAllValues:StringEquals": { + "aws:TagKeys": [ + "karpenter.sh/nodeclaim", + "Name" + ] + } + } + }, + { + "Sid": "AllowScopedDeletion", + "Effect": "Allow", + "Resource": [ + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*", + "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*" + ], + "Action": [ + "ec2:TerminateInstances", + "ec2:DeleteLaunchTemplate" + ], + "Condition": { + "StringEquals": { + "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned" + }, + "StringLike": { + "aws:ResourceTag/karpenter.sh/nodepool": "*" + } + } + }, + { + "Sid": "AllowRegionalReadActions", + "Effect": "Allow", + "Resource": "*", + "Action": [ + "ec2:DescribeAvailabilityZones", + "ec2:DescribeImages", + "ec2:DescribeInstances", + "ec2:DescribeInstanceTypeOfferings", + "ec2:DescribeInstanceTypes", + "ec2:DescribeLaunchTemplates", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSpotPriceHistory", + "ec2:DescribeSubnets" + ], + "Condition": { + "StringEquals": { + "aws:RequestedRegion": "${AWS_REGION}" + } + } + }, + { + "Sid": "AllowSSMReadActions", + "Effect": "Allow", + "Resource": "arn:${AWS_PARTITION}:ssm:${AWS_REGION}::parameter/aws/service/*", + "Action": "ssm:GetParameter" + }, + { + "Sid": "AllowPricingReadActions", + "Effect": "Allow", + "Resource": "*", + "Action": "pricing:GetProducts" + }, + { + "Sid": "AllowInterruptionQueueActions", + "Effect": "Allow", + "Resource": "arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}", + "Action": [ + "sqs:DeleteMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ReceiveMessage" + ] + }, + { + "Sid": "AllowPassingInstanceRole", + "Effect": "Allow", + "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}", + "Action": "iam:PassRole", + "Condition": { + "StringEquals": { + "iam:PassedToService": "ec2.amazonaws.com" + } + } + }, + { + "Sid": "AllowScopedInstanceProfileCreationActions", + "Effect": "Allow", + "Resource": "*", + "Action": "iam:CreateInstanceProfile", + "Condition": { + "StringEquals": { + "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", + "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" + } + } + }, + { + "Sid": "AllowScopedInstanceProfileTagActions", + "Effect": "Allow", + "Resource": "*", + "Action": "iam:TagInstanceProfile", + "Condition": { + "StringEquals": { + "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", + "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}", + "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", + "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" + } + } + }, + { + "Sid": "AllowScopedInstanceProfileActions", + "Effect": "Allow", + "Resource": "*", + "Action": [ + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:DeleteInstanceProfile" + ], + "Condition": { + "StringEquals": { + "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", + "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}" + } + } + }, + { + "Sid": "AllowInstanceProfileReadActions", + "Effect": "Allow", + "Resource": "*", + "Action": "iam:GetInstanceProfile" + }, + { + "Sid": "AllowAPIServerEndpointDiscovery", + "Effect": "Allow", + "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}", + "Action": "eks:DescribeCluster" + } + ] +} \ No newline at end of file diff --git a/website/content/en/preview/upgrading/v1beta1-reference.md b/website/content/en/preview/upgrading/v1beta1-reference.md new file mode 100644 index 000000000000..8b21dcf530e7 --- /dev/null +++ b/website/content/en/preview/upgrading/v1beta1-reference.md @@ -0,0 +1,661 @@ +--- +title: "v1beta1 Upgrade Reference" +linkTitle: "v1beta1 Upgrade Reference" +weight: 30 +description: > + API information for upgrading Karpenter +--- + +Significant changes to the Karpenter APIs have been introduced in Karpenter v0.32.x. +In this release, Karpenter APIs have advanced to v1beta1, in preparation for Karpenter v1 in the near future. +The v1beta1 changes are meant to simplify and improve ease of use of those APIs, as well as solidify the APIs for the v1 release. +Use this document as a reference to the changes that were introduced in the current release and as a guide to how you need to update the manifests and other Karpenter objects you created in previous Karpenter releases. + +The [Upgrade Guide]({{< relref "upgrade-guide" >}}) steps you through the process of upgrading Karpenter for the latest release. +For a more general understanding of Karpenter's compatibility, see the [Compatibility Document]({{< relref "compatibility" >}}). + +## CRD Upgrades + +Karpenter ships with a few Custom Resource Definitions (CRDs). Starting with v0.32.0, CRDs representing Provisioners, Machines, and AWS Node Templates are replaced by those for NodePools, NodeClaims, and EC2Nodeclasses, respectively. +You can find these CRDs by visiting the [Karpenter GitHub repository](https://github.com/aws/karpenter/tree{{< githubRelRef >}}pkg/apis/crds). + +The [Upgrade Guide]({{< relref "upgrade-guide" >}}) describes how to install the new CRDs. + +## Annotations, Labels, and Status Conditions + +Karpenter v1beta1 introduces changes to some common labels, annotations, and status conditions that are present in the project. The tables below lists the v1alpha5 values and their v1beta1 equivalent. + +| Karpenter Labels | | +|---------------------------------|-----------------------------| +| **v1alpha5** | **v1beta1** | +| karpenter.sh/provisioner-name | karpenter.sh/nodepool | + + +| Karpenter Annotations | | +|-------------------------------------|----------------------------------| +| **v1alpha5** | **v1beta1** | +| karpenter.sh/provisioner-hash | karpenter.sh/nodepool-hash | +| karpenter.k8s.aws/nodetemplate-hash | karpenter.k8s.aws/nodeclass-hash | +| karpenter.sh/do-not-consolidate | karpenter.sh/do-not-disrupt | +| karpenter.sh/do-not-evict | karpenter.sh/do-not-disrupt | + +> **Note**: Karpenter dropped the `karpenter.sh/do-not-consolidate` annotation in favor of the `karpenter.sh/do-not-disrupt` annotation on nodes. This annotation specifies that no voluntary disruption should be performed by Karpenter against this node. + +| StatusCondition Types | | +|---------------------------------|----------------| +| **v1alpha5** | **v1beta1** | +| MachineLaunched | Launched | +| MachineRegistered | Registered | +| MachineInitialized | Initialized | +| MachineEmpty | Empty | +| MachineExpired | Expired | +| MachineDrifted | Drifted | + +## Provisioner to NodePool + +Karpenter v1beta1 moves almost all top-level fields under the `NodePool` template field. Similar to Deployments (which template Pods that are orchestrated by the deployment controller), Karpenter NodePool templates NodeClaims (that are orchestrated by the Karpenter controller). Here is an example of a `Provisioner` (v1alpha5) migrated to a `NodePool` (v1beta1): + +Note that: +* The `Limits` and `Weight` fields sit outside of the template section. The `Labels` and `Annotations` fields from the Provisioner are now under the `spec.template.metadata` section. All other fields including requirements, taints, kubelet, and so on, are specified under the `spec.template.spec` section. +* Support for `spec.template.spec.kubelet.containerRuntime` has been dropped. If you are using EKS 1.23 you should upgrade to containerd before using Karpenter v0.32.0, as this field in the kubelet block of the NodePool is not supported. EKS 1.24+ only supports containerd as a supported runtime. + +**Provisioner example (v1alpha)** + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner + ... +spec: + providerRef: + name: default + annotations: + custom-annotation: custom-value + labels: + team: team-a + custom-label: custom-value + requirements: + - key: karpenter.k8s.aws/instance-generation + operator: Gt + values: ["3"] + - key: karpenter.k8s.aws/instance-category + operator: In + values: ["c", "m", "r"] + - key: karpenter.sh/capacity-type + operator: In + values: ["spot"] + taints: + - key: example.com/special-taint + value: "true" + effect: NoSchedule + startupTaints: + - key: example.com/another-taint + value: "true" + effect: NoSchedule + kubelet: + systemReserved: + cpu: 100m + memory: 100Mi + ephemeral-storage: 1Gi + maxPods: 20 + limits: + resources: + cpu: 1000 + memory: 1000Gi + weight: 50 +``` + +**NodePool example (v1beta1)** + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +... +spec: + template: + metadata: + annotations: + custom-annotation: custom-value + labels: + team: team-a + custom-label: custom-value + spec: + requirements: + - key: karpenter.k8s.aws/instance-generation + operator: Gt + values: ["3"] + - key: karpenter.k8s.aws/instance-category + operator: In + values: ["c", "m", "r"] + - key: karpenter.sh/capacity-type + operator: In + values: ["spot"] + taints: + - key: example.com/special-taint + value: "true" + effect: NoSchedule + startupTaints: + - key: example.com/another-taint + value: "true" + effect: NoSchedule + kubelet: + systemReserved: + cpu: 100m + memory: 100Mi + ephemeral-storage: 1Gi + maxPods: 20 + limits: + cpu: 1000 + memory: 1000Gi + weight: 50 +``` + +### Provider + +The Karpenter `spec.provider` field has been deprecated since version v0.7.0 and is now removed in the new `NodePool` resource. Any of the fields that you could specify within the `spec.provider` field are now available in the separate `NodeClass` resource. + +**Provider example (v1alpha)** + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner +... +spec: + provider: + amiFamily: Bottlerocket + tags: + test-tag: test-value +``` + +**Nodepool example (v1beta1)** + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +... +nodeClassRef: + name: default +``` + +**EC2NodeClass example (v1beta1)** + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +metadata: + name: default +spec: + amiFamily: Bottlerocket + tags: + test-tag: test-value +``` + +### TTLSecondsAfterEmpty + +The Karpenter `spec.ttlSecondsAfterEmpty` field has been removed in favor of a `consolidationPolicy` and `consolidateAfter` field. + +As part of the v1beta1 migration, Karpenter has chosen to collapse the concepts of emptiness and underutilization into a single concept: `consolidation`. +You can now define the types of consolidation that you want to support in your `consolidationPolicy` field. +The current values for this field are `WhenEmpty` or `WhenUnderutilized` (defaulting to `WhenUnderutilized` if not specified). +If specifying `WhenEmpty`, you can define how long you wish to wait for consolidation to act on your empty nodes by tuning the `consolidateAfter` parameter. +This field works the same as the `ttlSecondsAfterEmpty` field except this field now accepts either of the following values: + +* `Never`: This disables empty consolidation. +* Duration String (e.g. “10m”, “1s”): This enables empty consolidation for the time specified. + +**ttlSecondsAfterEmpty example (v1alpha)** + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner +... +spec: + ttlSecondsAfterEmpty: 120 +``` + +**consolidationPolicy and consolidateAfter examples (v1beta1)** + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +... +spec: + disruption: + consolidationPolicy: WhenEmpty + consolidateAfter: 2m + +``` + +### Consolidation + +The Karpenter `spec.consolidation` block has also been shifted under `consolidationPolicy`. If you were previously enabling Karpenter’s consolidation feature for underutilized nodes using the `consolidation.enabled` flag, you now enable consolidation through the `consolidationPolicy`. + +**consolidation enabled example (v1alpha)** + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner +... +spec: + consolidation: + enabled: true +``` + +**consolidationPolicy WhenUnderutilized example (v1beta1)** + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +... +spec: + disruption: + consolidationPolicy: WhenUnderutilized +``` + +> Note: You currently can’t set the `consolidateAfter` field when specifying `consolidationPolicy: WhenUnderutilized`. Karpenter will use a 15s `consolidateAfter` runtime default. + +### TTLSecondsUntilExpired + +The Karpenter `spec.ttlSecondsUntilExpired` field has been removed in favor of the `expireAfter` field inside of the `disruption` block. This field works the same as it did before except this field now accepts either of the following values: + +* `Never`: This disables expiration. +* Duration String (e.g. “10m”, “1s”): This enables expiration for the time specified. + +**consolidation ttlSecondsUntilExpired example (v1alpha)** + +```yaml +apiVersion: karpenter.sh/v1alpha5 +kind: Provisioner +... +spec: + ttlSecondsUntilExpired: 2592000 # 30 Days = 60 * 60 * 24 * 30 Seconds +``` + +**consolidationPolicy WhenUnderutilized example (v1beta1)** + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +... +spec: + disruption: + expireAfter: 720h # 30 days = 30 * 24 Hours + +``` + +### Defaults + +Karpenter now statically defaults some fields in the v1beta1 if they are not specified when applying the `NodePool` configuration. The following fields are defaulted if unspecified. + +| Field | Default | +|--------------------------------------|------------------------------------------------------------------| +| spec.disruption | {"consolidationPolicy: WhenUnderutilized", expireAfter: "720h"} | +| spec.disruption.consolidationPolicy | WhenUnderutilized | +| spec.disruption.expireAfter | 720h | + + +### spec.template.spec.requirements Defaults Dropped + +Karpenter v1beta1 drops the defaulting logic for the node requirements that were shipped by default with Provisioners in v1alpha5. Previously, Karpenter would create dynamic defaulting in the following cases. If multiple of these cases were satisfied, those default requirements would be combined: + +* If you didn't specify any instance type requirement: + + ```yaml + spec: + requirements: + - key: karpenter.k8s.aws/instance-category + operator: In + values: ["c", "m", "r"] + - key: karpenter.k8s.aws/instance-generation + operator: In + values: ["2"] + ``` + +* If you didn’t specify any capacity type requirement (`karpenter.sh/capacity-type`): + + ```yaml + spec: + requirements: + - key: karpenter.sh/capacity-type + operator: In + values: ["on-demand"] + ``` + +* If you didn’t specify any OS requirement (`kubernetes.io/os`): + ```yaml + spec: + requirements: + - key: kubernetes.io/os + operator: In + values: ["linux"] + ``` + +* If you didn’t specify any architecture requirement (`kubernetes.io/arch`): + ```yaml + spec: + requirements: + - key: kubernetes.io/arch + operator: In + values: ["amd64"] + ``` + +If you were previously relying on this defaulting logic, you will now need to explicitly specify these requirements in your `NodePool`. + +## AWSNodeTemplate to EC2NodeClass + +To configure AWS-specific settings, AWSNodeTemplate (v1alpha) is being changed to EC2NodeClass (v1beta1). Below are ways in which you can update your manifests for the new version. + +{{% alert title="Note" color="warning" %}} +Tag-based [AMI Selection](https://karpenter.sh/v0.31/concepts/node-templates/#ami-selection) is no longer supported for the new v1beta1 `EC2NodeClass` API. That feature allowed users to tag their AMIs using EC2 tags to express “In” requirements on selected. We recommend using different NodePools with different EC2NodeClasses with your various AMI requirement constraints to appropriately constrain your AMIs based on the instance types you’ve selected for a given NodePool. +{{% /alert %}} + +{{% alert title="Note" color="primary" %}} +Karpenter will now tag EC2 instances with their node name instead of with `karpenter.sh/provisioner-name: $PROVISIONER_NAME`. This makes it easier to map nodes to instances if you are not currently using [resource-based naming](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-naming.html). +{{% /alert %}} + + +### InstanceProfile + +The Karpenter `spec.instanceProfile` field has been removed from the EC2NodeClass in favor of the `spec.role` field. Karpenter is also removing support for the `defaultInstanceProfile` specified globally in the karpenter-global-settings, making the `spec.role` field required for all EC2NodeClasses. + +Karpenter will now auto-generate the instance profile in your EC2NodeClass, given the role that you specify. To find the role, type: + +```bash +export INSTANCE_PROFILE_NAME=KarpenterNodeInstanceProfile-bob-karpenter-demo +aws iam get-instance-profile --instance-profile-name $INSTANCE_PROFILE_NAME --query "InstanceProfile.Roles[0].RoleName" +KarpenterNodeRole-bob-karpenter-demo +``` + +**instanceProfile example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + instanceProfile: KarpenterNodeInstanceProfile-karpenter-demo +``` + +**role example (v1beta1)** + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + role: KarpenterNodeRole-karpenter-demo +``` + +### SubnetSelector, SecurityGroupSelector, and AMISelector + +Karpenter’s `spec.subnetSelector`, `spec.securityGroupSelector`, and `spec.amiSelector` fields have been modified to support multiple terms and to first-class keys like id and name. If using comma-delimited strings in your `tag`, `id`, or `name` values, you may need to create separate terms for the new fields. + + +**subnetSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + subnetSelector: + karpenter.sh/discovery: karpenter-demo +``` + +**SubnetSelectorTerms.tags example (v1beta1)** + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + subnetSelectorTerms: + - tags: + karpenter.sh/discovery: karpenter-demo +``` + +**subnetSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + subnetSelector: + aws::ids: subnet-123,subnet-456 +``` + +**subnetSelectorTerms.id example (v1beta1)** + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + subnetSelectorTerms: + - id: subnet-123 + - id: subnet-456 +``` + +**securityGroupSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + securityGroupSelector: + karpenter.sh/discovery: karpenter-demo +``` + +**securityGroupSelectorTerms.tags example (v1beta1)** + +```yaml +apiVersion: compute.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + securityGroupSelectorTerms: + - tags: + karpenter.sh/discovery: karpenter-demo +``` + +**securityGroupSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + securityGroupSelector: + aws::ids: sg-123, sg-456 +``` + + +**securityGroupSelectorTerms.id example (v1beta1)** + +```yaml +apiVersion: compute.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + securityGroupSelectorTerms: + - id: sg-123 + - id: sg-456 +``` + + +**amiSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + amiSelector: + karpenter.sh/discovery: karpenter-demo +``` + + +**amiSelectorTerms.tags example (v1beta1)** + +```yaml +apiVersion: compute.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + amiSelectorTerms: + - tags: + karpenter.sh/discovery: karpenter-demo +``` + +**amiSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + amiSelector: + aws::ids: ami-123,ami-456 +``` + +**amiSelectorTerms example (v1beta1)** + +```yaml +apiVersion: compute.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + amiSelectorTerms: + - id: ami-123 + - id: ami-456 +``` + + +**amiSelector example (v1alpha)** + +```yaml +apiVersion: karpenter.k8s.aws/v1alpha1 +kind: AWSNodeTemplate +... +spec: + amiSelector: + aws::name: my-name1,my-name2 + aws::owners: 123456789,amazon +``` + +**amiSelectorTerms.name example (v1beta1)** + +```yaml +apiVersion: compute.k8s.aws/v1beta1 +kind: EC2NodeClass +... +spec: + amiSelectorTerms: + - name: my-name1 + owner: 123456789 + - name: my-name2 + owner: 123456789 + - name: my-name1 + owner: amazon + - name: my-name2 + owner: amazon +``` + +### LaunchTemplateName + +The `spec.launchTemplateName` field for referencing unmanaged launch templates within Karpenter has been removed. +Find a discussion of the decision to remove `spec.launchTemplateName`, see [RFC: Unmanaged LaunchTemplate Removal](https://github.com/aws/karpenter/blob/main/designs/unmanaged-launch-template-removal.md). + +### AMIFamily + +The AMIFamily field is now required. If you were previously not specifying the AMIFamily field, having Karpenter default the AMIFamily to AL2, you will now have to specify AL2 explicitly. + +## Metrics + +The following table shows v1alpha5 metrics and the v1beta1 version of each metric. All metrics on this table will exist simultaneously, while both v1alpha5 and v1beta1 are supported within the same version. + +| **v1alpha5 Metric Name** | **v1beta1 Metric Name** | +|-----------------------------------------------------------------------|-----------------------------------------------------------------| +| karpenter_machines_created | karpenter_nodeclaims_created | +| karpenter_machines_disrupted | karpenter_nodeclaims_disrupted | +| karpenter_machines_drifted | karpenter_nodeclaims_drifted | +| karpenter_machines_initialized | karpenter_nodeclaims_initialized | +| karpenter_machines_launched | karpenter_nodeclaims_launched | +| karpenter_machines_registered | karpenter_nodeclaims_registered | +| karpenter_machines_terminated | karpenter_nodeclaims_terminated | +| karpenter_provisioners_limit | karpenter_nodepools_limit | +| karpenter_provisioners_usage | karpenter_nodepools_usage | +| karpenter_provisioners_usage_pct | Dropped | +| karpenter_deprovisioning_evaluation_duration_seconds | karpenter_disruption_evaluation_duration_seconds | +| karpenter_deprovisioning_eligible_machines | karpenter_disruption_eligible_nodeclaims | +| karpenter_deprovisioning_replacement_machine_initialized_seconds | karpenter_disruption_replacement_nodeclaims_initialized_seconds | +| karpenter_deprovisioning_replacement_machine_launch_failure_counter | karpenter_disruption_replacement_nodeclaims_launch_failed_total | +| karpenter_deprovisioning_actions_performed | karpenter_disruption_actions_performed_total | +| karpenter_deprovisioning_consolidation_timeouts | karpenter_disruption_consolidation_timeouts_total | +| karpenter_nodes_leases_deleted | karpenter_leases_deleted | + +In addition to these metrics, the MachineNotFound error returned by the `karpenter_cloudprovider_errors_total` values in the error label has been changed to `NodeClaimNotFound`. This is agnostic to the version of the API (Machine or NodeClaim) that actually owns the instance. + +## Global Settings + +The v1beta1 specification removes the `karpenter-global-settings` ConfigMap in favor of setting all Karpenter configuration using environment variables. Along, with this change, Karpenter has chosen to remove certain global variables that can be configured with more specificity in the EC2NodeClass . These values are marked as removed below. + + +| **`karpenter-global-settings` ConfigMap Key** | **Environment Variable** | **CLI Argument** +|---------------------------------------------------|---------------------------------|-------------------------------| +| batchMaxDuration | BATCH_MAX_DURATION | --batch-max-duration | +| batchIdleDuration | BATCH_IDLE_DURATION | --batch-idle-duration | +| assumeRoleARN | ASSUME_ROLE_ARN | --assume-role-arn | +| assumeRoleDuration | ASSUME_ROLE_DURATION | --assume-role-duration | +| clusterCABundle | CLUSTER_CA_BUNDLE | --cluster-ca-bundle | +| clusterName | CLUSTER_NAME | --cluster-name | +| clusterEndpoint | CLUSTER_ENDPOINT | --cluster-endpoint | +| defaultInstanceProfile | Dropped | Dropped | +| enablePodENI | Dropped | Dropped | +| enableENILimitedPodDensity | Dropped | Dropped | +| isolatedVPC | ISOLATED_VPC | --isolated-vpc | +| vmMemoryOverheadPercent | VM_MEMORY_OVERHEAD_PERCENT | --vm-memory-overhead-percent | +| interruptionQueueName | INTERRUPTION_QUEUE_NAME | --interruption-queue-name | +| reservedENIs | RESERVED_ENIS | --reserved-enis | +| featureGates.enableDrift | FEATURE_GATE="Drift=true" | --feature-gates Drift=true | + +## Drift Enabled by Default + +The drift feature will now be enabled by default starting from v0.33.0. If you don’t specify the Drift featureGate, the feature will be assumed to be enabled. You can disable the drift feature by specifying --feature-gates Drift=false. This feature gate is expected to be dropped when core APIs (NodePool, NodeClaim) are bumped to v1. + +## Logging Configuration is No Longer Dynamic + +As part of this deprecation, Karpenter will no longer call out to the APIServer to discover the ConfigMap. Instead, Karpenter will expect the ConfigMap to be mounted on the filesystem at `/etc/karpenter/logging/zap-logger-config`. You can also still choose to override the individual log level of components of the system (webhook and controller) at the paths `/etc/karpenter/logging/loglevel.webhook` and `/etc/karpenter/logging/loglevel.controller`. + +What you do to upgrade this feature depends on how you install Karpenter: + +* If you are using the helm chart to install Karpenter, you won’t need to make any changes for Karpenter to begin using this new mechanism for loading the config. + +* If you are manually configuring the deployment for Karpenter, you will need to add the following sections to your deployment: + + ```yaml + apiVersion: apps/v1 + kind: Deployment + spec: + template: + spec: + ... + containers: + - name: controller + volumeMounts: + - name: config-logging + mountPath: /etc/karpenter/logging + volumes: + - name: config-logging + configMap: + name: config-logging + ``` + +Karpenter will drop support for ConfigMap discovery through the APIServer starting in v0.33.0, meaning that you will need to ensure that you are mounting the config file on the expected filepath by that version. + +## Webhook Support Deprecated in Favor of CEL + +Karpenter v1beta1 APIs now support Common Expression Language (CEL) for validaiton directly through the APIServer. This change means that Karpenter’s validating webhooks are no longer needed to ensure that Karpenter’s NodePools and EC2NodeClasses are configured correctly. + +As a result, Karpenter will now disable webhooks by default by setting the `DISABLE_WEBHOOK` environment variable to `true` starting in v0.33.0. If you are currently on a version of Kubernetes < less than 1.25, CEL validation for Custom Resources is not enabled. We recommend that you enable the webhooks on these versions with `DISABLE_WEBHOOK=false` to get proper validation support for any Karpenter configuration. \ No newline at end of file diff --git a/website/static/_redirects b/website/static/_redirects index 239a5d444163..5655bc1466e9 100644 --- a/website/static/_redirects +++ b/website/static/_redirects @@ -8,3 +8,8 @@ /preview/concepts/provisioners /preview/concepts/nodepools /preview/concepts/node-templates /preview/concepts/nodeclasses /preview/concepts/deprovisioning /preview/concepts/disruption +/preview/upgrade/* /preview/upgrading/* +/preview/concepts/instance-types /preview/reference/instance-types +/preview/concepts/metrics /preview/reference/metrics +/preview/concepts/settings /preview/reference/settings +/preview/concepts/threat-model /preview/reference/threat-model