Description
Observed Behavior:
After upgrading to Karpenter 1.0, we tried to enact a policy to terminate nodes after 7d with a 4h terminationGracePeriod. However, Karpenter still refuses to terminate a pod at the deadline if the PDB does not allow for disruption. This results in us having large instances running with just a single workload pod as Karpenter has already evicted other workloads and tainted the node karpenter.sh/disrupted:NoSchedule 💸 .
Repeated events are generated against the node:
Normal DisruptionBlocked 14m (x1329 over 47h) karpenter Cannot disrupt Node: state node is marked for deletion
Warning FailedDraining 3m48s (x1407 over 47h) karpenter Failed to drain node, 12 pods are waiting to be evicted
11 DaemonSet pods and 1 pod from a Deployment. The Deployment's PDB is configured to not allow normal termination of the pod.
Karpenter itself is logging:
{
"level": "ERROR",
"time": "2024-10-25T12:46:56.452Z",
"logger": "controller",
"message": "consistency error",
"commit": "6174c75",
"controller": "nodeclaim.consistency",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "test-vxtgb"
},
"namespace": "",
"name": "test-vxtgb",
"reconcileID": "2a7b8ffd-80cf-4fbf-b612-870a33adec27",
"error": "can't drain node, PDB \"default/test\" is blocking evictions"
}Expected Behavior:
A node owned by Karpenter reaches expireAfter + terminationGracePeriod and all pods are removed. Node is terminated.
I'm not sure if this is actually a documentation bug? But the documentation certainly implies, to my reading, that PDBs get overridden when the grace period expires: terminationGracePeriod
Pods blocking eviction like PDBs and do-not-disrupt will block full draining until the terminationGracePeriod is reached.
Reproduction Steps (Please include YAML):
- Have a NodePool with forceful termination enabled. e.g.
spec:
template:
spec:
expireAfter: 1h
terminationGracePeriod: 1h
- Create a Deployment:
kubectl create deployment test --image=nginx --replicas=1
- Add a PDB that won't allow termination:
kubectl create poddisruptionbudget test --selector=app=test --min-available=1
- Wait. Node won't get terminated by Karpenter
Versions:
- Chart Version: 1.0.6
- Kubernetes Version (
kubectl version):
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.4-eks-a737599
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Description
Observed Behavior:
After upgrading to Karpenter 1.0, we tried to enact a policy to terminate nodes after 7d with a 4h terminationGracePeriod. However, Karpenter still refuses to terminate a pod at the deadline if the PDB does not allow for disruption. This results in us having large instances running with just a single workload pod as Karpenter has already evicted other workloads and tainted the node
karpenter.sh/disrupted:NoSchedule💸 .Repeated events are generated against the node:
11 DaemonSet pods and 1 pod from a Deployment. The Deployment's PDB is configured to not allow normal termination of the pod.
Karpenter itself is logging:
{ "level": "ERROR", "time": "2024-10-25T12:46:56.452Z", "logger": "controller", "message": "consistency error", "commit": "6174c75", "controller": "nodeclaim.consistency", "controllerGroup": "karpenter.sh", "controllerKind": "NodeClaim", "NodeClaim": { "name": "test-vxtgb" }, "namespace": "", "name": "test-vxtgb", "reconcileID": "2a7b8ffd-80cf-4fbf-b612-870a33adec27", "error": "can't drain node, PDB \"default/test\" is blocking evictions" }Expected Behavior:
A node owned by Karpenter reaches
expireAfter + terminationGracePeriodand all pods are removed. Node is terminated.I'm not sure if this is actually a documentation bug? But the documentation certainly implies, to my reading, that PDBs get overridden when the grace period expires: terminationGracePeriod
Reproduction Steps (Please include YAML):
Versions:
kubectl version):