-
Notifications
You must be signed in to change notification settings - Fork 302
fix: update deletionTimestamp on terminating pods when after nodeDeletionTimestamp #2316
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: update deletionTimestamp on terminating pods when after nodeDeletionTimestamp #2316
Conversation
Welcome @cosimomeli! |
Hi @cosimomeli. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Pull Request Test Coverage Report for Build 16201702039Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
/assign @engedaam Amanuel implemented Node Autorepair so assigning him since he's the relevant owner |
hey @engedaam any estimated time for the review? thx! |
Currently, Karpenter does not immediately drain pods when initiating a Node Repair action. Instead, it relies on a tolerationDuration configured by the cloud provider. For example, in the AWS Provider, unreachable nodes are given a 30-minute toleration duration before Karpenter begins the process of deleting the node. During this termination period, Karpenter waits for pods to be terminated, which is handled by the drain logic implemented in the terminator.go file (specifically at this line: https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/node/termination/terminator/terminator.go#L140). The behavior you're describing in this PR aligns with our current expectations. To better understand any potential issues, could you provide a specific example where you've observed Karpenter taking longer than the configured toleration duration to terminate an unhealthy node?
Can you help me understand why this would help here? We only really look at the deletionTimestamp for filtering pods, not for case when to force delete |
As far as I understood, in case a Pod has a long termination grace period the node will not be removed at the end of the node toleration duration, but it will wait for the pod termination grace period. For example, in RabbitMQ operator pods have 7 days termination period, so the node won't be terminated before 7 days. @cosimomeli can explain better. |
Hello @engedaam, thanks for the answer. Karpenter's terminator logic immediately drains every pod in the Node, as the node.health controller sets the node termination timestamp to the current timestamp, this is actually a forced shutdown, and the termination goes as expected, but there is one exception:
My change has the effect of adding inside |
Considering that after 30 minutes, excluding the ones with an explicit toleration of |
This clears things up and thanks for the thorough explanation! |
/ok-to-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one small nit
Co-authored-by: Amanuel Engeda <74629455+engedaam@users.noreply.github.com>
/lgtm |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cosimomeli, jmdeal The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…tionTimestamp (kubernetes-sigs#2316) Co-authored-by: Amanuel Engeda <74629455+engedaam@users.noreply.github.com>
Description
When a node receives the unreachable taint, the Kubernetes taint controller triggers the deletion of all pods after 5 minutes. When the Node Repair threshold is reached, Karpenter's drain procedure waits for all pods to be evicted or to be stuck on termination (when they have passed their deletionTimestamp), but if a Pod has a long termination grace period (RabbitMQ operator pods have 7 days, for example) the node will wait too long before being deleted.
To improve the forced termination, I added the terminating pods with a deletionTimestamp after the nodeTerminationTimestamp to be deleted again, so their deletionTimestamp can be aligned with the nodeTerminationTimestamp.
How was this change tested?
I added a unit test for this and also tested the change with both an Unhealthy Node on AWS (dead kubelet) and a simple node deletion.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.