v0.28.0

Known Issues

Karpenter v0.28.0 is incompatible with Kubernetes version 1.26+, which can result in additional node scale outs when using --cloudprovider=external, which is the default for the EKS Optimized AMI. See: kubernetes-sigs/karpenter#375

Features

• support trn1 instance neuron accelerators (#3902) #3902 (Sarthak)
• Adds vcpu and memory metrics per instance type (#3824) #3824 (Valdis Rigdon)
• Include knative webhooks in readiness probes (#3879) #3879 (Ellis Tarn)
• Add SecurityGroup Name to AWSNodeTemplate Status (#3866) #3866 (Amanuel Engeda)
• Machine Migration (#3408) #3408 (Jonathan Innis)
• Adding AMI status into AWSNodeTemplate (#3551) #3551 (Amanuel Engeda)
• adds new karpenter_cloudprovider_instance_price_estimate prometheus metric (#3613) #3613 (Valdis Rigdon)
• support reservedENIs for vpc cni custom networking (#3516) #3516 (Brandon Wagner)

Bug Fixes

• add labels on PodDisruptionBudget (#3981) #3981 (Maxime Guillet)
• Edit the cache for install-deps to use K8s version (#3987) #3987 (Jonathan Innis)
• remove data field from Secret (#4025) #4025 (Tore)
• Convert karpenter.sh/managed-by label to annotation (#4034) #4034 (Jonathan Innis)
• Update E2E old drift AMI (#3897) #3897 (Amanuel Engeda)
• Revert v0.27.5 values.yaml changes (#3916) #3916 (Jonathan Innis)
• Explicitly configure AssignPublicIp in EC2 Launch Templates (#3814) #3814 (lorenzadia)
• Enable IMDSv2 (#3931) #3931 (Amanuel Engeda)
• undefined blockdevicemappings volumesize would panic if not defaulted (#3898) #3898 (Nick Tran)
• update graviton allocatable memory calculation (#3940) #3940 (Todd Neal)
• Ensure that our cluster Machine list is more updated than our CloudProvider Machine list (#3943) #3943 (Jonathan Innis)
• Don't Delete Instances if Orphaned (#3951) #3951 (Jonathan Innis)
• Stringify the kubeletExtraArgs values for AL2/Ubuntu (#3841) #3841 (Jonathan Innis)

Documentation

• Metric breaking change note for v0.28.0 and add Metrics (#3984) #3984 (Jonathan Innis)
• Update docs to switch ttlSecondsAfterEmpty to consolidation (#3964) #3964 (Amanuel Engeda)
• add AppsFlyer to adopters (#3993) #3993 (Daniel)
• Updated EKS Karpenter workshop links in getting started page (#4002) #4002 (Bill Rayburn)
• fix github-relref for /docs (#4021) #4021 (Brandon Wagner)
• Update Drift Docs (#4014) #4014 (Amanuel Engeda)
• Troubleshooting Docs for IP Address Exhaustion (#4037) #4037 (Jonathan Innis)
• Add a note on karpenter removing defaulting.webhook.karpenter.sh (#3861) #3861 (Jonathan Innis)
• Add docs updates for v0.27.5 (#3911) #3911 (Jonathan Innis)
• Updated getting-started guide to add logout of helm registry (#3880) #3880 (Apoorva Kulkarni)
• add pods preventing eviction point in control flow (#3782) #3782 (Nick Tran)
• Fix a typo in settings.md (#3780) #3780 (Apoorva Kulkarni)
• add detail on spot interruption behavior (#3789) #3789 (Josh Hart)
• Update AWSNodeTemplate docs for ami selectors (#3786) #3786 (Vishal Vazkar)
• Add a note on Karpenter's dependency on tags for IAM (#3832) #3832 (Jonathan Innis)
• Add upgrade note on CRDs for machines (#3847) #3847 (Jonathan Innis)

Tests

• adjust upgrade instance size (#3959) #3959 (Nick Tran)
• Enable scripts to be referenced in configMapGenerator and Enable Sweeper (#3961) #3961 (Jonathan Innis)
• Fix alpine package resolution (#3966) #3966 (Jonathan Innis)
• Fix cleanup.sh script failures from empty responses (#3968) #3968 (Jonathan Innis)
• Add periodic trigger for scale testing (#3970) #3970 (Jonathan Innis)
• fix consolidation replace test (#3982) #3982 (Nick Tran)
• separate cleanup objects for finalizable and non finalizable objects (#3983) #3983 (Nick Tran)
• fix test labels and testing failures (#4019) #4019 (Nick Tran)
• Add TEST_TIMEOUT for scale test run (#4038) #4038 (Jonathan Innis)
• Disable prefix delegation (#3990) #3990 (Jonathan Innis)
• Enable GitHub Actions for E2Es (#3893) #3893 (Jonathan Innis)
• Fix syntax error in GHA yq command (#4040) #4040 (Jonathan Innis)
• Update region for GHA runs (#4042) #4042 (Jonathan Innis)
• Fix node template reference for drift AWSNodeTemplate (#4045) #4045 (Jonathan Innis)
• Fix missed Eventually() checks in happiness check (#3855) #3855 (Jonathan Innis)
• Bump Tektoncli for e2etests (#3892) #3892 (Amanuel Engeda)
• Properly wait for a full pod restart on E2Es (#3891) #3891 (Jonathan Innis)
• Cleanup Subnet tests (#3890) #3890 (Amanuel Engeda)
• Scale testing (#3874) #3874 (Jonathan Innis)
• Run the tests scoped to the test suite with E2E_TEST_SUITE (#3923) #3923 (Jonathan Innis)
• Fix failures in Scale Testing Suite (#3926) #3926 (Jonathan Innis)
• Update the spot interruption test to use the FIS client directly (#3938) #3938 (Jonathan Innis)
• fix consolidation test timing and add commit ref in metrics (#3942) #3942 (Nick Tran)
• add in more scale test dimensions (#3946) #3946 (Nick Tran)
• fix cloudwatch importing with missing env variable (#3950) #3950 (Nick Tran)
• reduce scale test instance size (#3958) #3958 (Nick Tran)
• Fix upgrading CRDS from older version (#3790) #3790 (Jonathan Innis)
• E2ETesting Delete launch templates in cleanup step (#3808) #3808 (Jonathan Innis)
• Ignore empty output when running xargs (#3811) #3811 (Jonathan Innis)
• Fix E2ETest Flakiness on Labels and Machines (#3821) #3821 (Jonathan Innis)
• Fix settings override logic in AfterEach (#3825) #3825 (Jonathan Innis)

Chores

• fix machine controller names (#3945) #3945 (Guillaume Delacour)
• deps: Bump github.com/aws/aws-sdk-go from 1.44.267 to 1.44.273 in /test (#3972) #3972 (dependabot[bot])
• deps: Bump postcss from 8.4.23 to 8.4.24 in /website (#3971) #3971 (dependabot[bot])
• deps: Bump github.com/onsi/ginkgo/v2 from 2.9.2 to 2.9.7 in /test (#3973) #3973 (dependabot[bot])
• deps: Bump github.com/onsi/ginkgo/v2 from 2.9.2 to 2.9.7 (#3975) #3975 (dependabot[bot])
• deps: Bump golang.org/x/sync from 0.1.0 to 0.2.0 (#3976) #3976 (dependabot[bot])
• deps: Bump github.com/aws/aws-sdk-go from 1.44.267 to 1.44.273 (#3978) #3978 (dependabot[bot])
• deps: Bump github.com/pelletier/go-toml/v2 from 2.0.7 to 2.0.8 (#3977) #3977 (dependabot[bot])
• only show minor versions on website (#4001) #4001 (Brandon Wagner)
• fix version links on website (#4009) #4009 (Brandon Wagner)
• Update data from AWS APIs (#4027) #4027 (github-actions[bot])
• update mod for release (#4047) #4047 (Nick Tran)
• Fix change monitor changing on different node templates (#3809) #3809 (Jonathan Innis)
• Update data from AWS APIs (#3865) #3865 (github-actions[bot])
• Add SQS Server Side Encryption to interruption queue in getting started guide (#3886) #3886 (Ellis Tarn)
• Update data from AWS APIs (#3888) #3888 (github-actions[bot])
• Move AMI drift to a drift.go (#3904) #3904 (Amanuel Engeda)
• up aws-sdk-go to v1.267 (#3930) #3930 (Brandon Wagner)
• Bump karpenter-core and use v1alpha5.MachineLinked annotation for Delete (#3932) #3932 (Jonathan Innis)
• Release v0.27.3 (#3783) #3783 (github-actions[bot])
• Add additional machine e2e for node finalizer compatability (#3792) #3792 (Jonathan Innis)
• Sort missing instanceTypes for Bandwidth Codegen (#3801) #3801 (Amanuel Engeda)
• Handle re-owned nodes linking (#3793) #3793 (Jonathan Innis)
• Add an upgrade note for machine migration (#3797) #3797 (Jonathan Innis)
• deps: Bump yaml from 2.1.1 to 2.2.2 in /website (#3810) #3810 (dependabot[bot])
• Fix folder cleanup for new versions of the website (#3817) #3817 (Jonathan Innis)
• Add machine permission to aggregate clusterrole (#3831) #3831 (Jonathan Innis)
• deps: Bump github.com/aws/aws-sdk-go-v2/config from 1.18.19 to 1.18.22 in /test (#3838) #3838 (dependabot[bot])
• deps: Bump postcss from 8.4.21 to 8.4.23 in /website (#3834) #3834 (dependabot[bot])
• deps: Bump github.com/aws/aws-sdk-go from 1.44.234 to 1.44.253 in /test (#3837) #3837 (dependabot[bot])
• Update data from AWS APIs (#3839) #3839 (github-actions[bot])
• deps: Bump github.com/aws/aws-sdk-go from 1.44.195 to 1.44.253 (#3833) #3833 (dependabot[bot])
• deps: Bump github.com/prometheus/client_golang from 1.14.0 to 1.15.0 (#3836) #3836 (dependabot[bot])
• deps: Bump github.com/go-playground/validator/v10 from 10.11.2 to 10.13.0 (#3835) #3835 (dependabot[bot])
• Fix escaping regex special characters (#3840) #3840 (Jonathan Innis)
• Restore GPU labels and use DoesNotExist (#3845) #3845 (Jonathan Innis)
• Bump karpenter-core to latest (#3846) #3846 (Jonathan Innis)
• Remove aws.nodeNameConvention from Karpenter settings (#3851) #3851 (Jonathan Innis)

Commits

• 2a7d02a: Threat model (#3969) (Micah Hausler) #3969
• 4cc6790: added Next Insurance to adopters list (#3994) (Moshe Ben Shoham) #3994
• 78ac493: use github raw links for grafana and prometheus values (#4010) (Brandon Wagner) #4010
• 346e9a7: fix Provisioner.spec.limits in scheduling.md (#4031) (Yahav Biran) #4031
• 92c5b26: fix bottlerocket node-labels merge in user-data (#4030) (Brandon Wagner) #4030
• ed9b0ac: v0.27.6 website updates (#4033) (Brandon Wagner) #4033
• c2a915e: Add PITS Global Data Recovery Services (#3858) (Frederic) #3858
• 61657bb: Ignore make toolchain when hitting Cache (#3883) (Jonathan Innis) #3883
• 999690c: update maintainers (#3901) (Brandon Wagner) #3901
• 6e44eef: update doc (#3887) (Prabhjot Singh) #3887
• 0e04b48: update k8s versions (#3935) (Todd Neal) #3935
• bf0e7f6: Update faq.md (#3956) (uditsidana) #3956
• cfc9213: add k8s 1.27.x to CI testing (#3957) (Todd Neal) #3957
• 2be6616: Bump karpenter-core to latest for v0.28.0-rc.2 (#3947) (Jonathan Innis) #3947
• 49a318e: BREAKING CHANGE: Add tag key checking to AWSNodeTemplate validation (#3784) (Jonathan Innis) #3784
• 0d9dc0b: BREAKING CHANGE: Remove .Values.extraObjects (#3803) (Jonathan Innis) #3803