Description
Observed Behavior:
We are running an EKS cluster with 1 node per zone. We have 1 Statefulset (2 replicas) using PersistentVolume dynamicaly assigned to a zone by EBS CSI Driver. We wanted to setup pod's node affinity selecting one that is not handled by Karpenter (e.g. the one on zone B). During a rolling update where we added a new soft affinity with karpenter.sh/nodepool key and DoesNotExist operator, pod-1 got rescheduled on an appropriate node (i.e. on zone C corresponding to its PV zonal constraint). Yet, pod-0 — which has a PV zonal constaint on zone A — got stuck on Pending state with the following error message : 2 node(s) had volume node affinity conflict. Checking the events we saw that Karpenter drained the node on zone A after we setup this new affinity. Karpenter's log shows that the pods from the statefulset got ignored as we were using this specific node affinity.
{"level":"DEBUG","time":"2024-06-18T14:32:52.068Z","logger":"controller.provisioner","message":"ignoring pod, configured to not run on a Karpenter provisioned node via the karpenter.sh/nodepool DoesNotExist requirement","commit":"6b868db","pod":"monitoring/prometheus-prometheus-kube-prometheus-prometheus-0"}
Pod-0 was the only resource with a zonal constaint on zone A. As Karpenter started to ignore this pod and so its zonal constraints, the node has been eligible for draining. Other nodes were running workloads that had zonal contraints to zone B or C and also had enough capacity to handle the pods from node of zone A, so this node form zone A got drained and pods rescheduled to the remaining ones.
Expected Behavior:
It makes sense that the pod gets ignored by Karpenter when there's a hard affinity on such key/operator as it's guaranteed that the pod would never be scheduled on a node managed by Karpenter. Yet, I feel like this shouldn't necessarily be the case with soft affinities. As it's a preference and not a requirement, there's no guarantee that the pod will be scheduled on a node that is not managed by Karpenter and I feel like instead of ignoring the pod, Karpenter should have behaved normaly and detect that we shouldn't drain node A as pod-0 would most likely be scheduled on such node due to its PV zonal constraints.
N.B : I must confess we had setup an incoherent scenario where the prefered node is on a zone that doesn't match its PV zonal constraint. Yet, as it's a preference, I'm still expecting the scheduler to ignore pod's prefered node affinity and prioritize PV's required affinity meaning this case is still valid from my point of view.
Reproduction Steps (Please include YAML):
I don't have the capacity to test this, but I suppose that should do :
- Setup an EKS cluster with Karpenter installed. Ensure log level is set on DEBUG.
- Do what's necessary to have 1 node created by Karpenter on a zone != from the initial node.
- Create a PV with a zonal constraint to the same zone as the node created by Karpenter
- Setup up a pod with a PVC referencing the PV we just created. I'm expecting the pod to be scheduled on the node managed by Karpenter as per PV constraint.
- Add the following node affinity to the pod :
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
weight: 50
- Check Karpenter's logs. It should now state that it ignores the pod. If the initial node is properly sized (meaning that it can handled the pods from karpenter's node) and there no extra contraints on karpenter's node, i'm expecting the node to get drained and the pod to get stuck on Pending as per PV constraint violation.
Versions:
- Chart Version: 0.36.0
- Kubernetes Version (
kubectl version):
Client Version: v1.29.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4-eks-036c24b
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Description
Observed Behavior:
We are running an EKS cluster with 1 node per zone. We have 1 Statefulset (2 replicas) using
PersistentVolumedynamicaly assigned to a zone by EBS CSI Driver. We wanted to setup pod's node affinity selecting one that is not handled by Karpenter (e.g. the one on zone B). During a rolling update where we added a new soft affinity withkarpenter.sh/nodepoolkey andDoesNotExistoperator, pod-1 got rescheduled on an appropriate node (i.e. on zone C corresponding to its PV zonal constraint). Yet, pod-0 — which has a PV zonal constaint on zone A — got stuck on Pending state with the following error message :2 node(s) had volume node affinity conflict. Checking the events we saw that Karpenter drained the node on zone A after we setup this new affinity. Karpenter's log shows that the pods from the statefulset got ignored as we were using this specific node affinity.Pod-0 was the only resource with a zonal constaint on zone A. As Karpenter started to ignore this pod and so its zonal constraints, the node has been eligible for draining. Other nodes were running workloads that had zonal contraints to zone B or C and also had enough capacity to handle the pods from node of zone A, so this node form zone A got drained and pods rescheduled to the remaining ones.
Expected Behavior:
It makes sense that the pod gets ignored by Karpenter when there's a hard affinity on such key/operator as it's guaranteed that the pod would never be scheduled on a node managed by Karpenter. Yet, I feel like this shouldn't necessarily be the case with soft affinities. As it's a preference and not a requirement, there's no guarantee that the pod will be scheduled on a node that is not managed by Karpenter and I feel like instead of ignoring the pod, Karpenter should have behaved normaly and detect that we shouldn't drain node A as pod-0 would most likely be scheduled on such node due to its PV zonal constraints.
N.B : I must confess we had setup an incoherent scenario where the prefered node is on a zone that doesn't match its PV zonal constraint. Yet, as it's a preference, I'm still expecting the scheduler to ignore pod's prefered node affinity and prioritize PV's required affinity meaning this case is still valid from my point of view.
Reproduction Steps (Please include YAML):
I don't have the capacity to test this, but I suppose that should do :
Versions:
kubectl version):