Mega Issue: Deprovisioning Controls

[njtran]

Tell us about your request
Karpenter provisions nodes and deprovisions nodes as described in the docs.

Some users are asking for more control on how to specify when Karpenter should disrupt nodes, and controls on how to rate-limit these disruptions. Opening this issue to aggregate the existing issues.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

** Karpenter disruption conditions:**

• Automatic Update of Nodes #1716 - Karpenter should watch for AMI version updates and roll nodes to update to the new version.
• Support specifying AMI family release version in provisioner #1495 - allow specifying AMI Release version in Provisioner
• Reconcile nodes that are out of spec #1457 - Provisioner Drift
• Native Support for Spot Termination #702 - Consume EC2 events similar to Node Termination Handler
• Proposal: introduce opt-in support for orphaned node destruction #1268 - Support for orphaned node destruction (both present in the cluster and outside)
• Add jitter over nodes TTL #1884 - Include jitter for TTLSecondsUntilExpired

Control over how Karpenter should disrupt nodes:

• Provide more control over node upgrading #1018 - TTLSecondsUntilExpired can result in a lot of node churn at unexpected times. Rate-limit some termination decisions to reduce downtime.
• Machine Disruption Gate kubernetes-sigs/karpenter#753, Provide ability on when to actually rotate expired nodes #903 - Time-based disruption controls
• Node-based Disruption Budget for Deprovisioning/Termination kubernetes-sigs/karpenter#679 - Node Disruption Budget
• Exponential / logarithmic decay for cluster desired size kubernetes-sigs/karpenter#696 - Use a defined "expected cluster size" to drive deprovisioning
• Merge Expiration/Drift in the Deprovisioning controller kubernetes-sigs/karpenter#710 - Consider deprovisioning differently per-deprovisioning method?
• Consolidation ttl: spec.disruption.consolidateAfter kubernetes-sigs/karpenter#735 - Per-node TTL for Consolidation
• MinAge for NodeClaims kubernetes-sigs/karpenter#1030 - MinAge for NodeClaims

Control over Karpenter's Eviction Policy:

• Add a gracePeriod for the do-not-disrupt pod annotation kubernetes-sigs/karpenter#752 - Allow specifying a non-permanent time frame do-not-evict for a pod, for un-programmatic removal use-cases.
• Add "allow-eviction" functionality  #1532 - Include allow-evict annotation for pods for emptiness condition.
• Do not evict running jobs kubernetes-sigs/karpenter#701 - Don't evict running jobs
• Provision to specify the max Drain / Pod evictions attempts in Provisioner  #3052 - Max eviction tries
• support drain timeout kubernetes-sigs/karpenter#743 - Drain/Eviction Timeout

Additional context

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[vasylenko]

Heavy ➕ for #1716 - Karpenter should watch for AMI version updates and roll nodes to update to the new version.

Use case: rotate (refresh) running instances when new security or other important update is released to the AMI used by the project/organization.

[prashil-g]

+1

[aavileli]

How does ami refresh or OS refresh looks like in karpentar as it does not use asg node groups.
One of our needs is to refresh nodes every 3 weeks. we would use a similar public script https://github.com/hellofresh/eks-rolling-update which used asg and cluster autoscaler to
to cordon , drain and delete the nodes . With karpentar we could use ttlSecondsUntilExpired to expire the node. What happens if a new vulnerability is discovered we need to refresh all nodes
solution is to change the ttlSecondsUntilExpired for small cluster is fine but for a large cluster this would create a high churn so one idea is to have a max_number_of_node flag to control the refresh capacity. Is there a better way to achieve this ?

[hajdukd]

Any update on this ? Pls

[blakepettersson]

Would it be possible to do something like a subset of #1841, which doesn't necessarily have to re-provision nodes? For example, updating instance tags shouldn't need to re-provision new nodes.

[hawkesn]

Another example of #1716 , I upgraded the EKS control plane and would like the nodes to upgrade accordingly.

[njtran]

Hey all. Check out #2569 for the design doc for node upgrades.

[karma-git]

@ellistarn Hi,

Maybe you could specify the ETA of NodeDrift implementation?
Thank you

[johngmyers]

My view is that if a node could have been created from the current spec then it has not drifted, whereas if it could not have been created it should be deprovisioned (subject to rate limits, etc.)

So if the zone requirements changed from ["a"] to ["a", "b"] then no nodes need be deprovisioned. But if the opposite change is made then all nodes in zone b should be deprovisioned.

[njtran]

In these cases drift is undesirable.

@johngmyers, sorry I'm a little confused here by your comment. I thought previously you were advocating for label and tag drift, but in your message here it looks like you're saying drift is not desired?

[johngmyers]

I am saying I want drift to be remediated by karpenter.

[grandich]

@ellistarn Regarding "ability to control when and when not to expire nodes":

• (this issue) 8) #903 - Provide ability on when to actually rotate expired nodes
• Adding Maintenance Windows feature for advanced node control TTL (#1302) #1920

Just to understand: Will the proposed way to address this be using Node Disruption Budget and k8s cronjobs updating them to achieve "maintenance windows"?

[njtran]

Changed the title and updated this issue to be more accurate of the current state of affairs. Added a design doc on how we're thinking Drift should work for the rest of the known fields here: kubernetes-sigs/karpenter#366

[hendryanw]

Hi, I’ve searched online and in Github, but can’t find the documentation covering the node patch / update. It’s related to #1716, which I saw it has already been ticked.

Can you please help guide me where can I the documentation?

[njtran]

@hendryanw you can drive node patches/updates by deprovisioning. As of v0.29.0, Karpenter automatically upgrades nodes through Drift for AMIs, Security Groups, and Subnets.

[calvinbui]

can we add kubernetes-sigs/karpenter#735 to this list?

[engedaam]

Just released v0.30.0-rc.0 which contains the full set of drift expansion. Checkout the full release notes in karpenter and karpenter-core

[njtran]

Hey all, I've linked an RFC here that has some of the API decisions we're thinking about for some of the linked items in this mega issue.

• Machine Disruption Gate kubernetes-sigs/karpenter#753, Provide ability on when to actually rotate expired nodes #903 - Maintenance Windows
• Node-based Disruption Budget for Deprovisioning/Termination kubernetes-sigs/karpenter#679 - Node Disruption Budget
• Consolidation ttl: spec.disruption.consolidateAfter kubernetes-sigs/karpenter#735 - Per-node TTL for Consolidation

Please take a read and give a review if you can!