Skip to content

fix: update deletionTimestamp on terminating pods when after nodeDeletionTimestamp #2316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

cosimomeli
Copy link
Contributor

Description
When a node receives the unreachable taint, the Kubernetes taint controller triggers the deletion of all pods after 5 minutes. When the Node Repair threshold is reached, Karpenter's drain procedure waits for all pods to be evicted or to be stuck on termination (when they have passed their deletionTimestamp), but if a Pod has a long termination grace period (RabbitMQ operator pods have 7 days, for example) the node will wait too long before being deleted.

To improve the forced termination, I added the terminating pods with a deletionTimestamp after the nodeTerminationTimestamp to be deleted again, so their deletionTimestamp can be aligned with the nodeTerminationTimestamp.

How was this change tested?
I added a unit test for this and also tested the change with both an Unhealthy Node on AWS (dead kubelet) and a simple node deletion.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link

linux-foundation-easycla bot commented Jun 17, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jun 17, 2025
@k8s-ci-robot
Copy link
Contributor

Welcome @cosimomeli!

It looks like this is your first PR to kubernetes-sigs/karpenter 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/karpenter has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 17, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @cosimomeli. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 17, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 17, 2025
@coveralls
Copy link

coveralls commented Jun 17, 2025

Pull Request Test Coverage Report for Build 16201702039

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 6 of 6 (100.0%) changed or added relevant lines in 2 files are covered.
  • 171 unchanged lines in 13 files lost coverage.
  • Overall coverage decreased (-0.1%) to 81.916%

Files with Coverage Reduction New Missed Lines %
pkg/controllers/disruption/consolidation.go 3 88.14%
pkg/controllers/disruption/drift.go 4 87.76%
pkg/controllers/disruption/singlenodeconsolidation.go 4 93.62%
pkg/controllers/disruption/emptiness.go 5 87.3%
pkg/controllers/state/statenode.go 5 87.05%
pkg/controllers/controllers.go 9 0.0%
pkg/controllers/disruption/multinodeconsolidation.go 10 86.76%
pkg/test/ratelimitinginterface.go 10 0.0%
pkg/controllers/disruption/helpers.go 11 87.43%
pkg/controllers/disruption/validation.go 15 81.92%
Totals Coverage Status
Change from base Build 15692799185: -0.1%
Covered Lines: 10219
Relevant Lines: 12475

💛 - Coveralls

@jonathan-innis
Copy link
Member

/assign @engedaam

Amanuel implemented Node Autorepair so assigning him since he's the relevant owner

@chicco785
Copy link

hey @engedaam any estimated time for the review? thx!

@engedaam
Copy link
Contributor

engedaam commented Jul 9, 2025

When the Node Repair threshold is reached, Karpenter's drain procedure waits for all pods to be evicted or to be stuck on termination (when they have passed their deletionTimestamp), but if a Pod has a long termination grace period (RabbitMQ operator pods have 7 days, for example) the node will wait too long before being deleted.

Currently, Karpenter does not immediately drain pods when initiating a Node Repair action. Instead, it relies on a tolerationDuration configured by the cloud provider. For example, in the AWS Provider, unreachable nodes are given a 30-minute toleration duration before Karpenter begins the process of deleting the node. During this termination period, Karpenter waits for pods to be terminated, which is handled by the drain logic implemented in the terminator.go file (specifically at this line: https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/node/termination/terminator/terminator.go#L140). The behavior you're describing in this PR aligns with our current expectations. To better understand any potential issues, could you provide a specific example where you've observed Karpenter taking longer than the configured toleration duration to terminate an unhealthy node?

To improve the forced termination, I added the terminating pods with a deletionTimestamp after the nodeTerminationTimestamp to be deleted again, so their deletionTimestamp can be aligned with the nodeTerminationTimestamp.

Can you help me understand why this would help here? We only really look at the deletionTimestamp for filtering pods, not for case when to force delete

@chicco785
Copy link

Currently, Karpenter does not immediately drain pods when initiating a Node Repair action. Instead, it relies on a tolerationDuration configured by the cloud provider. For example, in the AWS Provider, unreachable nodes are given a 30-minute toleration duration before Karpenter begins the process of deleting the node. During this termination period, Karpenter waits for pods to be terminated, which is handled by the drain logic implemented in the terminator.go file (specifically at this line: https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/node/termination/terminator/terminator.go#L140). The behavior you're describing in this PR aligns with our current expectations. To better understand any potential issues, could you provide a specific example where you've observed Karpenter taking longer than the configured toleration duration to terminate an unhealthy node?

As far as I understood, in case a Pod has a long termination grace period the node will not be removed at the end of the node toleration duration, but it will wait for the pod termination grace period. For example, in RabbitMQ operator pods have 7 days termination period, so the node won't be terminated before 7 days.

@cosimomeli can explain better.

@cosimomeli
Copy link
Contributor Author

To better understand any potential issues, could you provide a specific example where you've observed Karpenter taking longer than the configured toleration duration to terminate an unhealthy node?

Hello @engedaam, thanks for the answer.
When a node becomes unreachable, Karpenter triggers Node Repair after 30 minutes (on AWS). Meanwhile, the Kubernetes taint controller starts evicting pods after 5 minutes when a node receives the node.kubernetes.io/unreachable taint.

Karpenter's terminator logic immediately drains every pod in the Node, as the node.health controller sets the node termination timestamp to the current timestamp, this is actually a forced shutdown, and the termination goes as expected, but there is one exception:
The podsToDelete here filters out every pod that is already terminating, and has not passed its graceful termination period. This means that if I have a pod with a very long termination period (my example was RabbitMQ with 7 days), the termination controller will not touch it, and Karpenter will wait for its natural termination before deleting the node.

To improve the forced termination, I added the terminating pods with a deletionTimestamp after the nodeTerminationTimestamp to be deleted again, so their deletionTimestamp can be aligned with the nodeTerminationTimestamp.

Can you help me understand why this would help here? We only really look at the deletionTimestamp for filtering pods, not for case when to force delete

My change has the effect of adding inside podsToDelete the pods with a graceful shutdown period that ends after the deadline of the node termination, this way they get deleted again with a different graceful period, to be compatible with the node termination deadline.

@cosimomeli
Copy link
Contributor Author

Considering that after 30 minutes, excluding the ones with an explicit toleration of node.kubernetes.io/unreachable, all the pods are already deleted by the toleration controller as documented here, it's not uncommon that a terminating pod needs another delete to change its default deletion time to speed up the draining process.

@engedaam
Copy link
Contributor

Considering that after 30 minutes, excluding the ones with an explicit toleration of node.kubernetes.io/unreachable, all the pods are already deleted by the toleration controller as documented here, it's not uncommon that a terminating pod needs another delete to change its default deletion time to speed up the draining process.

This clears things up and thanks for the thorough explanation!

@engedaam
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 10, 2025
Copy link
Contributor

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small nit

Co-authored-by: Amanuel Engeda <74629455+engedaam@users.noreply.github.com>
@engedaam
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 10, 2025
@jmdeal
Copy link
Member

jmdeal commented Jul 11, 2025

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cosimomeli, jmdeal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 11, 2025
@k8s-ci-robot k8s-ci-robot merged commit 58bf160 into kubernetes-sigs:main Jul 11, 2025
16 checks passed
rlanhellas pushed a commit to rlanhellas/karpenter that referenced this pull request Jul 12, 2025
…tionTimestamp (kubernetes-sigs#2316)

Co-authored-by: Amanuel Engeda <74629455+engedaam@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants