Pause Pod Evictions during Full Zonal Disruption

[sumukha-radhakrishna]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Currently, we pause pod evictions when a zone experiences partial disruption. This is to prevent unnecessary evictions due to temporary or localized issues.

However, during a full zonal disruption, our current logic continues evicting pods, which can be counterproductive since the entire zone is unavailable.

All workloads in a zone gets evicted during FullZonalDisruption. I understand the idea to evict pods in a zone where we are not able to allocate/provision more capacity since its down.

I am not opposed to the idea of having a flag to turn this on/off, looking for opinion the on the idea.

Which issue(s) this PR fixes:

Fixes #131314

Special notes for your reviewer:

Does this PR introduce a user-facing change?

No

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.33 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.33.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Mon Apr 14 13:34:58 UTC 2025.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

Hi @rsumukha. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dims]

/ok-to-test

[sumukha-radhakrishna]

I am not opposed to having a flag to turn this on/off, looking for opinion the on the idea.

[k8s-triage-robot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sumukha-radhakrishna
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• api/OWNERS
• cmd/kube-controller-manager/OWNERS
• pkg/controller/apis/config/OWNERS
• pkg/controller/nodelifecycle/OWNERS
• pkg/generated/openapi/OWNERS
• staging/src/k8s.io/kube-controller-manager/config/OWNERS
• test/integration/node/OWNERS
• test/integration/scheduler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rphillips]

However, during a full zonal disruption, our current logic continues evicting pods, which can be counterproductive since the entire zone is unavailable.

If there is a full zonal disruption, then the zone is not guaranteed to come back. Thus the pods are left scheduled to those pods within the dead zone. I'm inclined to say the behavior is correct today by evicting the pods within the dead zone, so they have an opportunity to run in a different zone as a self-healing mechanism.

Thoughts?

[sumukha-radhakrishna]

I'm inclined to say the behavior is correct today by evicting the pods within the dead zone, so they have an opportunity to run in a different zone as a self-healing mechanism.

You're right, evicting pods during zonal disruption can trigger rescheduling elsewhere, which is a good self-healing strategy.
But it needs to be done carefully, and configurably, because in some clusters, it can degrade availability if capacity isn't there. This is why I made it configurable via cli flag.

[siyuanfoundation]

/remove-sig api-machinery

[dims]

@cartermckinnon @kmala please help review!

[lmktfy]

If I had a magic coding wand (and, specifically, one significantly better than Claude):

• On detecting full zonal disruption, [k-c-m would] capture of snapshot of all nodes in that zone. Using that snapshot, taint any node in that zone that isn't obviously healthy, using a newly minted, official Kubernetes taint. This taint is to indicate that the node is in an unhealthy zone. This first controller can, at least initially, be off-by-default.
• We also add a second, on-by-default controller to drop that taint once the zone appears to partially or fully recover (and of course, if that specific taint is never set, this controller won't appear to be doing anything). However, you can opt out of the second controller and remove that taint using your own implementation.

Lots of extensibility options and the possibility to prototype it out-of-tree as well. Plus, the mechanism uses existing Kubernetes concepts so it will be very easy to teach.