Karpenter is terminating nodes that have pods with the do-not-disrupt annotation

[tatyana12345]

Description

Observed Behavior:
Many of our pods have the annotation karpenter.sh/do-not-disrupt: "true", but karpenter still deleted the node on which these pods were running. Reason: AMIDrift
Example log:

{"level":"INFO","time":"2025-02-05T06:01:03.248Z","logger":"controller","caller":"disruption/controller.go:183","message":"disrupting nodeclaim(s) via replace, terminating 1 nodes (7 pods) ip-xx-xx-xx-58.ec2.internal/m5.xlarge/on-demand and replacing with on-demand node from types m5.xlarge, m5.2xlarge","commit":"6174c75","controller":"disruption","namespace":"","name":"","reconcileID":"bebd3e56-c70c-4d3c-9a01-6de66e1e6d78","command-id":"c346a6c4-f550-4cba-ae2e-ebde75f4f8f9","reason":"drifted"}

{"level":"DEBUG","time":"2025-02-05T05:27:37.777Z","logger":"controller","caller":"disruption/controller.go:91","message":"marking drifted","commit":"6174c75","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"ondemand-pt85t"},"na
```mespace":"","name":"ondemand-pt85t","reconcileID":"962b44d0-4ee5-48d0-b649-083bbee95d7c","reason":"AMIDrift"}

Configuration:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: private
  namespace: ${local.karpenter.namespace}
spec:
  amiFamily: Bottlerocket
  userData:  |
    [settings.kubernetes]
    "shutdown-grace-period" = "50s"
    "shutdown-grace-period-for-critical-pods" = "50s"
    eviction-max-pod-grace-period = 60
    [settings.kubernetes.eviction-hard]
    "memory.available" = "2.5%"
    "nodefs.available" = "15%"
    "nodefs.inodesFree" = "10%"
    "imagefs.available" = "20%"
    [settings.kubernetes.eviction-soft]
    "memory.available" = "5%"
    [settings.kubernetes.eviction-soft-grace-period]
    "memory.available" = "5m30s"
  role: ${var.node_group_default_iam_role_name}
  subnetSelectorTerms:
    - tags:
        Name: "${var.name}"
  securityGroupSelectorTerms:
    - tags:
       Name: "${var.name}"
  amiSelectorTerms:
    - alias: bottlerocket@latest
  tags:
    karpenter.sh/discovery: ${var.name}
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 2Gi
        volumeType: gp3
    - deviceName: /dev/xvdb
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
=====================
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  annotations:
    ...
spec:
  disruption:
    budgets:
    - nodes: 5%
      reasons:
      - Drifted
      - Underutilized
    - nodes: 100%
      reasons:
      - Empty
    consolidateAfter: 15m
    consolidationPolicy: WhenEmptyOrUnderutilized
  limits:
    cpu: "250"
    memory: 1000Gi
  template:
    metadata:
      labels:
        node.kubernetes.io/lifecycle: ondemand
    spec:
      expireAfter: 720h
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: private
      requirements:
        ...
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
      terminationGracePeriod: 15m

Expected Behavior:
If I understand the official documentation correctly, annotation karpenter.sh/do-not-disrupt: "true" should block nodes from being deleted during drift - but it doesn't.

Reproduction Steps :
Follow the official installation instructions for helm chart karpenter version 1.0.6
https://artifacthub.io/packages/helm/aws-karpenter/karpenter/1.0.6

Add annotation to pods karpenter.sh/do-not-disrupt: "true" according to this documentation

The problem will be reproduced when changing the AMI-image

Versions:

• Chart Version: 1.0.6
• Kubernetes Version (kubectl version): 1.31
• AMI image: Bottlerocket

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

There's a docs miss here, when terminationGracePeriod is configured drift is not blocked by PDBs or the do-not-disrupt annotation. This enables cluster administrators to ensure crucial updates aren't blocked by misconfigured applications. Given you have a 15 minute terminationGracePeriod, this is intended behavior. I've opened a PR with the docs updates: #7710.

[imunhatep]

The problem is not with the docs, karpenter node rotation logic is broken.