The getTargetNodes in RemoveDuplicates does not respect node resources and can mistakenly evict pods

[bingzheliu]

descheduler version

0.27.1

k8s version

v1.28.0

Problem

The RemoveDuplicate plugin is supposed to get the feasible nodes that can place the duplicatePods before calculating which pods to delete. In particular, this line of code.

However, it currently only checks on the tolerance/taint and nodeSelector (code here). The node can be infeasible to the pod due to other reasons, i.e., the node does not have enough resources for more pods.

This issue can cause more pods to be mistakenly evicted if nodes can't hold the pods due to limited resources. And unending cycle of scheduling and eviction can occur.

What is expected

The getTargetNodes should consider the node resource fits and other scheduling restrictions of the pods (like NodePorts).

[lowang-bh]

/assign

[a7i]

@bingzheliu nodeFit by default is false so you may want to enable that.

If you are using descheduler/v1alpha2, the docs can be found here to configure the DefaultEvictor.

profiles:
  - name: ProfileName
    pluginConfig:
    - name: "DefaultEvictor"
      args:
        nodeFit: true

If you are using descheduler/v1alpha, the docs can be found here

  RemoveDuplicates:
     enabled: true
     params:
       removeDuplicates:
         nodeFit: true

[a7i]

However nodePorts is not one of our nodeFit checks. Would you be open to proposing a design for that?

[bingzheliu]

Thanks for the reply @a7i ! Sure, I can create a new issue for the nodePorts later.

Regarding the resource fit, I understand there is the nodeFit in the evictor. However, not implementing the node resource fit in this getTargetNodes function inside the RemoveDuplicates plugins can still cause unexpected issues.

Because the result of targetNodes is used to calculate the upperAvg that determines which pods to evict.
upperAvg := int(math.Ceil(float64(ownerKeyOccurence[ownerKey]) / float64(len(targetNodes)))) code here.

This example shows how the issue can occur:

In this cluster, node 1 can only schedule one more pod due to being occupied by other deployments; and node 1/2 has less weight than node 3 for nodeAffinity. Hence, the scheduler decides to schedule the 5 pods as shown in the above graph.

If the getTargetNodes consider the resource, targetNodes = 2 and upperAvg = ceil(5/2) = 3; then none of the pods should be evicted.

However, as now the getTargetNodes only considered taint and nodeSelector, the targetNodes = 3 and upperAvg = ceil(5/3) = 2, causing one pod to be evicted on node 3 and scheduled again, and an ending eviction/scheduling cycle perpetuated.

Even setting the NodeFit in the eviction plugin does not help, as there is another node (node 2) in this case that can host the pod.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.