Karpenter consolidateAfter is not working as expected

[mallow111]

Description

For consolidateAfter, can we make -> consolidateAfter is the amount of time Karpenter should wait to have a node as part of the consolidation process. The amount of time Karpenter has to wait until pods can move into the node and out of the node due to consolidation. What we want is that when a node has a pod movement (pod added or removed for any reason - k8s scale up/down, consolidation, drift, etc.) then this node is invisible for the Karpenter underutilized consolidation process

• E.g How can we avoid nodes are disturbed before the 30min(consolidateAfter setting) configuration?

• E.g We want to know what is the logic behind picking a node for consolidation, for those perfectly packed nodes, can we avoid picking them for consolidation before 30min(the consolidateAfter period)?It is ok to consolidate these nodes after 30min.

• E.g We also concerned the new nodes who joined in the cluster recently can get consolidated in a short period of time again(it's OK to consolidate them after 30min)

Observed Behavior:
We have the following settings:

disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30m
    budgets:
    # Drifted nodes limited to terminate 10% in parallel
       - nodes: "10%"
       reasons:
         - Drifted
    # Underutilized nodes always compacting to avoid a low packing growth
      - nodes: "5%"
      reasons:
        - Underutilized
    # Empty nodes can be terminated any time

• Some well-packed nodes get interrupted
• Several pods are disrupted unnecessarily.
• The interruption is too frequent even if we set consolidateAfter: 30m

Expected Behavior:

• We expect to see only 3 nodes get consolidated at this moment since they are empty now, other nodes and pods get untouched

Reproduction Steps (Please include YAML):
Nodepool yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: aaa
spec:
    disruption:
      consolidationPolicy: WhenEmptyOrUnderutilized
      consolidateAfter: 30m
      budgets:
      # Drifted nodes limited to terminate 10% in parallel
         - nodes: "10%"
         reasons:
           - Drifted
      # Underutilized nodes always compacting to avoid a low packing growth
        - nodes: "5%"
        reasons:
          - Underutilized
      # Empty nodes can be terminated any time
    template:
      spec:
        expireAfter: Never
        nodeClassRef:
          group: karpenter.k8s.aws
          kind: EC2NodeClass
          name: nearcorepool2
        requirements:
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64
          - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - on-demand
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - m7i.8xlarge
        taints:
        - effect: NoSchedule
          key: dedicated
          value: aaa

In the workload, we just set the toleration to use this nodepool

  Node-Selectors:  nodepool=aaa
  Tolerations:  dedicated=aaa:NoSchedule

Versions:

• Chart Version: 1.3.3
• Kubernetes Version (kubectl version): The bug is not related to k8s version.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[burgerjeffrey]

We had to limit our consolidation windows to two one hour (twice a day) windows for drifted and underutlized nodes since the constant churn of consolidating nodes was unnecessary and the interruptions to our workloads constantly wasn't ideal. Below is the disruption policy we decided on. Also, narrowing down the available EC2 instance types can help as well.

spec:
disruption:
budgets:
- duration: 7h
nodes: '0'
reasons:
- Drifted
- Underutilized
schedule: 0 1 * * *
- duration: 15h
nodes: '0'
reasons:
- Drifted
- Underutilized
schedule: 0 9 * * *
- nodes: 10%
reasons:
- Empty
consolidateAfter: 10m
consolidationPolicy: WhenEmptyOrUnderutilized

[mallow111]

@burgerjeffrey Thx for the info, does this change help you guys have less disruption, this change basically blocked Drifted and Underutilized, we want to have consolidation for drift and Underutilized but not so frequent.
Also, narrowing down the available EC2 instance types can help as well. do you have a sample yaml we can refer? Thx!

[burgerjeffrey]

We used it for more controlled disruption times since we didn't want to narrow down our EC2 instance types.

[jmdeal]

Our customers dont want to have frequent interruptions, no matter it is consolidation/drift/empty

Both consolidation and emptiness respect consolidate after, the only one of these mechanisms that doesn't is drift. That being said drift should not be occurring all that frequently and if it is that indicates to me an issue in the environment (e.g. an actor continuously updating NodePools or NodeClasses). If that's the case that should be addressed as the root cause IMO. Can you elaborate more on your need for it to apply to drift?

As for the issue with consolidation applying to packed nodes, we would need to understand a few things:

• What pods were scheduled to each of the nodes? Was there anything preventing the temporary workload from scheduling together?
• Were all of the nodes consolidatable (according to the status condition)? What's the consolidateAfter setting for those nodes?

A general recommendation I've given before is to separate high-churn workloads (e.g. periodic jobs) and stable workloads into separate NodePools. High-churn workloads are going to naturally result in higher churn for the nodes they're on, whereas Karpenter's more likely to reach and maintain a steady state for more stable workloads. Hard to say if this applies to your scenario without more details though.

[jmdeal]

/triage needs-information

[mallow111]

@jmdeal thanks for the input, do you also have insight about these 2 questions?

• We want to know what is the logic behind picking a node for consolidation, for those perfectly packed nodes, can we avoid picking them for consolidation?

• We also concerned the new nodes who joined in the cluster recently can get consolidated in a short period of time again

[ellistarn]

Are you running the default KubeScheduler or a custom one that uses NodeResourcesFit: MostAllocated? We have a working theory that the kube scheduler's default spreads all pods to the least allocated node, which triggers the ConsolidateAfter waiting period on all of your smallest nodes. This results in a scenario where the largest nodes are the only allowable targets for consolidation, and can trigger the behavior you describe in your picture above.

The cluster autoscaler does not face this issue as describe because its simulations use a simpler threshold-based approach. There are tradeoffs to both approaches.

I suspect that this problem would be significantly mitigated by swapping the kube scheduler's strategy to pack instead of spread pods.

[mallow111]

@ellistarn We are using default KubeScheduler and, we have been talking about this during our sync meeting(salesfoce and aws sync meeting), what do you recommend for the configuration of default KubeScheduler, any sample yaml we can follow

[sushraju]

Just also make sure the workloads you care about have cpu / memory requests. If they are configured less with higher limits (burstable), it's quite likely they will get moved around quite frequently based on our observation.

[mallow111]

@jmdeal @ellistarn Can you provide some thought about how to avoid nodes being disrupted before 30m(our consolidateAfter: 30m), we just don't want to have the consolidation process looking at the new node for 30min. It's like the new node (or the node that just got a pod movement) is invisible, it's not there for 30min. After 30min it's all good to consolidate into the new node or out of the new node.