Graceful Node Shutdown does not update endpoints for terminating pods #116965
Labels
kind/bug
Categorizes issue or PR as related to a bug.
needs-triage
Indicates an issue or PR lacks a `triage/foo` label and requires one.
sig/node
Categorizes an issue or PR as relevant to SIG Node.
Comments
k8s-ci-robot
added
needs-sig
|
/sig node |
k8s-ci-robot
added
sig/node
|
/assign |
i don't know your backend implementation of service，it’s significant feature to route backend healthy backend。the terminating pod in EndpointSlice is not ready as described below，shouldn't route new traffic to it。 but it‘s unreasonable terminating pod in EndpointSlice，should like Endpoints。 |
It will be important to check the logs of the endpointslice controller in the controller-manager and correlate with the pod and node states |
SergeyKanzhelev
added this to Triage
in SIG Node Bugs (migrated to https://github.com/orgs/kubernetes/projects/185)
|
Terminating pods appearing in EndpointSlice might be intentional due to KEP-1672, though that's just a guess from the name. However, I'm not seeing what you posted above. What I'm seeing is both I updated the repro a little bit to print the preStop hook sequence in the log & an increase the grace period. It doesn't materially change anything. Here's the video of what happens from the repo I linked on my computer, with controller logs: https://drive.google.com/file/d/1-_hsvkJq3cOtLxEzQ2R8Fy4MEXka2Z82/view?usp=sharing . Both |
|
Do you see the terminating flag value on Endpoint slice (https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#terminating) changed for the terminating pod? /triage needs-information Also if you can share kubelet.log and @aojea requested the logs of the endpointslice controller - it will be helpful |
k8s-ci-robot
added
the
triage/needs-information
SergeyKanzhelev
moved this from Triage
to Needs Information
in SIG Node Bugs (migrated to https://github.com/orgs/kubernetes/projects/185)
|
The I captured the logs in the video I posted above as well. Here are the logs pasted (from a second attempt). I shut down the node @ kube-controller-manager.logkubelet.log |
all the components can increase the verbosity level with the |
The endpointslice reflects the state of the Pod, it literally checks the pod state, same you are doing with you'll need to check the pod at that time, it should be ready, if not, there is a problem we need to investigate more |
|
The pod also reports everything is great while shutting down: Afaict the pod never even gets marked as |
|
Seems the code around pod status merging is at least partially responsible for the dropped api server update. The shutdown manager code only sets Not sure what the right fix is here, but I think that's the root cause? If I'm understanding the code right, this would affect pod eviction and any other kubelet-led graceful pod shutdown. |
|
@SergeyKanzhelev would you need anything else from me to triage this? This may be a bit presumptuous but I noticed @mimowo & @smarterclayton worked on this area recently. Would either of you have the time to see if my diagnosis in the previous comment is correct/useful? |
|
/triage accepted xref to the KEP: kubernetes/enhancements#2000 to consider when going GA Will this problem be the same when we try to delete Pod due to some eviction reasons than? The PR is changing the existing tests - need to understand that the proposed fix is not regressing some other behavior. |
k8s-ci-robot
added
triage/accepted
SergeyKanzhelev
moved this from Needs Information
to Triaged
in SIG Node Bugs (migrated to https://github.com/orgs/kubernetes/projects/185)
|
/remove-triage needs-information |
k8s-ci-robot
removed
the
triage/needs-information
|
Hi, we also encounter this issue on 1.27.6. What we see, is that after the node deletion triggered, pod is still Running during the prestop lifecycle, also when sigterm is sent. Then the pod transition to status to completed. Endpoint is updated when the pod pass in terminating state, but because during shutdown it never transition to Terminating, it stills send traffic to this pod until the pod is Completed, which cause errors. |
|
/remove-triage accepted Can someone from SIG node re-triage this? IIUC, the question is how/when does kubelet report to the apiserver that the Pod is terminating when doing a graceful node shutdown, so that other controllers can react. |
k8s-ci-robot
added
needs-triage
|
Double check that your service has a sigterm handler that sets the readiness endpoint to false. Some of the behavior described on this ticket is expected behavior. If the readiness endpoint is never set to false, then the pod is expected to stay ready until the entire lifecycle is terminated. The kubelet also needs to be setup in a way where if it is running under systemd the dependency chain has the kubelet terminate prior to networking. |
|
Also note, the pre-stop hook blocks the pod termination until it has completed.
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution |
|
@rphillips I don't think this is specced behavior: readiness probes auto-fail when pods start shutdown normally. See this page: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
|
Because of a bug(#124648), readiness probe doesn’t work during pod termination triggered in kubelet such as eviction and node shutdown. This may be one of reasons why an endpoint is not updated early. |
@rphillips @smarterclayton it seems the main problem here is the lack of update on the status, once the status on eviction is propagated Terminating or Failing, Services will work correctly, we had a similar issue in the past #110255 |
|
So zooming out: GracefulShutdown by itself does not say whether the node is coming back or not - that's the responsibility of the agent that "owns" the node (cloud provider, human admin, vm orchestrator, whatever) to determine whether the node is coming back or not. So using graceful shutdown by itself can make no decision on whether to delete the pods or not, only to correctly represent that the pod is shutting down and being not ready. GracefulShutdown was not intended to replace a controlled drain operation on the node. If the use case described here is assuming that graceful shutdown will guarantee that pods are terminated, that is not generally true and not the design we intended for GracefulShutdown. Any "set delete timestamp" action would have to be opt in and probably not the responsibility of the kubelet generally - the responsibility today belongs to the initiator of the "node going away permanently" action (cloud provider, orchestrator, human) and I could argue it should stay there. I think we can say that GracefulShutdown is intended to end the processes of pods as best effort within a specific timeframe and report the status of those pods appropriately. At minimum, the pods should be described as non ready, and the containers should be described as stopped with terminal exit codes and/or phase (RestartAlways pods should be left in running phase, because there was no signal that the node is not coming back). But it is best effort - it cannot guarantee due to being part of a distributed system that those status updates will propagate. The remaining question is whether the node is doing a good enough job during graceful shutdown. I think the answer given this thread is "no" - I would expect the following (as i commented on the other bug):
Anything I'm missing? Does this match everyone's experience and expectations? |
|
One piece I might want to clarify is point 2:
This isn't entirely complete - the endpoint slices that include these pods need to be updated correctly. Specifically, there are 3 booleans currently exposed: "ready", "serving", and "terminating". For pods on nodes undergoing graceful node shutdown these should be:
Unfortunately doing this in the endpoint controller is impossible since the kubelet doesn't expose any indicator that the pod is shutting down. There may be a disruption condition now - so maybe this bug is about having the endpoint controller use that. Separately - and this is just a non-advanced end user opinion that you can definitely ignore - IMO kubelet eviction should use pod deletion. The problem ultimately is doing anything else violates the "principle of least surprise". During eviction the pod is "Terminating" but unless everyone remembers that there are 2 ways of checking that - deletion timestamp & a disruption condition (or something similar) - there will be bugs everywhere. Even kubectl doesn't do this right now (to show "Terminating"). Nobody expects "kubectl drain" and "sudo halt" to do anything different - it's a nasty surprise. I'm not totally sure I understand what's gained by leaving the undead pods around. In theory you might be able to restart them if the node comes back up, but even if that doesn't happen - the parent controller should be recreating them, right? Then is this done to protect "fully uncontrolled" pods (ie non-static and non-daemonset/replicaset/statefulset/etc)? These are super weird - you can't even use "kubectl delete" to restart them - right? I readily admit I have not run kube on bare metal or with pet nodes (only cattle nodes) - so I'd love to understand how this feature helps folks. Ultimately if allowing kubelets to actually delete pods is a breaking change in the kube API guarantees, IMO it's worth doing. It likely fixes more bugs than it creates. All that said, the endpoint slice controller behavior is what this bug report is about. So long as that readiness starts reporting false when termination begins - that works to fix this issue. |
6 tasks
|
Addressing this first:
In the specific, we can't rely on there being a parent controller because there are many scenarios where automation creates singular pods - all pods are mortal, but we try to avoid disrupting them unnecessarily (which is indeed surprising to users when they do get disrupted). A node restart is not a guarantee that a pod can't happily continue running past that restart. More generally, I should clarify that graceful node shutdown is an an additive feature to Kubernetes - it added the concept of a node being shutdown with the intent that the node may not come back (i.e. to support spot nodes on cloud primarily). However, the default and historical behavior of Kubernetes is the exact opposite - nodes can go away and come back at any time and Kubernetes' was designed to be resilient to all of those scenarios (so we're paranoid about specific aspects of "stopping early" to avoid failing to restart things). The addition of graceful shutdown thus had to support both scenarios, and does so with two different intersecting configurables:
The design thus had to support both of these and the full intersection with features like graceful termination of endpoints (which serving=x was added to support). While kubelet has never initiated graceful deletions for pods, there's nothing that says it couldn't when the node isn't coming back. We discussed it in the graceful shutdown implementation, several of us myself included were nervous because we didn't understand all of the issues. It's certainly possible, and might be something we can consider in the future. However, if we had, it would have masked the particular issue you found for restartable workloads, and we probably would not have detected the issue here, which is: There is no reliable signal sent by the kubelet during graceful node shutdown to ensure that the endpoints controller can eagerly removing pods from rotation. |
|
Now addressing the second part: The key difference with spot instances is that they are time limited - we have a fixed budget of time in which to ensure all of the right signals are sent to other components before permanent disruption occurs (e.g. 25s before the machine is forcibly powered off in some examples). Appropriate signals:
We need a solution for permanent graceful node shutdown that maximizes the chance that the endpoints controller can eagerly remove pods from rotation - we need to both ensure a low latency signal is sent in the normal case, and a high reliability signal is sent. Problem 1: The Kubelet may not be able to reliably update all pods in timeThe Kubelet has a lot of loops and work going on it and it it is possible that the Kubelet receives the signal but is unable to act upon it before 25s are over. The kubelet status loop is single threaded today - it updates each pod status sequentially. A single blip of apiserver connectivity, a bad load balancer, or a network error on the Kubelet could contribute 10s of seconds of delay, which if there are enough pods on the node may mean that some pods are not signaled before the time limit is up. Problem 2: The Kubelet may not receive the signalDepending on the cloud and environment, the signal for graceful termination may not be delivered to the node, so the actor who receives the signal is also capable of sending it. If different enviroments have different signals, it is hard to justify doing this only in the Kubelet. Problem 3: Once the node is terminated, the Kubelet can no longer actOther actors in the system are responsible post termination for minimizing the failure of the Kubelet to act (specifically pod garbage collection and node controller). A concern with adding support to the Kubelet for deletion is that it is duplicative to the above. |
|
Between the two signals, graceful deletion is only appropriate for nodes that aren't coming back ever, so if we changed we would only fix a portion of the problem you identified. We should consider improving that, but given the scope of the change it's probably better to leverage the other signal and look at how we could propagate ready=false more correctly. Propagating ready=false does make problem 1 worse if the implementation of post-node cleanup for a given provider is lacking (because it increases the chance that the completed pod update does not make it out). But I think practically that's a gap in the provider and correct propagation of ready=false is necessary for all footprints no matter what. Providers should be aware of that. My recommendationThe minimum Kube fix here seems to be to ensure that once readiness probes are stopped for a container, the container is marked as not ready, and the kubelet propagates that status rapidly to the api. I think by definition when we are no longer probing we are assuming the pod is not ready, but a risk involved in this fix that may need to be mitigated is that we'll be more rapidly marking the pod not ready during node-initiated shutdown, and then if we fix the early probe stopping at a later time we'd have another impact on users. We might need to identify which early probe-stopping improvements we make at the same time. |
|
Something @bobbypage noted - graceful node shutdown should be triggering the node to go not ready, which should propagate to the control plane, which should cause the node lifecycle controller to detect the node going unready, which should then result in the pods being marked as not ready (should = we see the code for that, it should be working). That would then at least temporarily result in pod status changing to not ready, which should then trigger an endpoint restart. It might not if the kubelet sees the status change and fixes it rapidly, but a watch result would show the ready state changing. @mikekap, when you ran your reproducer, did you see the pod status changing during a watch during live shutdown? |
No I didn't see anything change in the pod description (except the disruption condition). In the original PR I had traced the reason for that to kubernetes/pkg/kubelet/status/status_manager.go Lines 1088 to 1103 in 3f9b79f |
|
Re: node controller disruption - David indicates there seems to be a bug in the controller that has prevented this from working ever, so can discount for now. Re: your other code change I tried to think about all the ways the linked fix could interact with the cluster, and I believe it corrects the issue you are seeing because of an unintended side effect of setting the state to be failed in the kubelet. However, I believe (but have not tested) breaks the expected behavior of the Kubelet allowing readiness probes to keep a pod in the serving rotation after it is marked deleted, so it works to fix your scenario but breaks others (please correct me if my review is inaccurate). Re: this issue My triage of this issue is that as described, "kubernetes is working as intended" in this case. There is a gap in that there is no recommendation for cloud-portable shutdown for workloads, but expecting delete pod semantics is a provider specific behavior that graceful shutdown does not provide today (it's not an unreasonable ask, it's just not supported by Kubernetes but is instead something provider specific). The Kubernetes-portable behavior is to use the readiness endpoint to control being in rotation during graceful shutdown - being deleted is not required to be drained in a conformant distro today. So in the scenario you describe, your readiness endpoint should start returning false during your preStop hook if you want to be taken out of rotation fast. Details:
Graceful shutdown as a feature is intended to work for all node shutdown scenarios, not just spot instances. One scenario is "node isn't coming back", and the other is "node is coming back". The current design of graceful shutdown does not include a way for the kubelet to know for certain that "node shutting down" = "not coming back", and so the kubelet can't take the action of "deleting pods" today (and thus can't behave like kubectl delete pod). In general a reboot signal is probably not sufficient to result in a delete pods action even on spot instances - there may be other use cases for transient spot-like nodes where there may be multiple types of shutdown (live migration, suspend for forensic analysis, etc). At the current time, it's the responsibility of the infrastructure provider using the graceful shutdown feature to implement the "delete pod if node isn't coming back" semantic. I think we as a project should be open to an improvement to graceful node shutdown that standardized that signal (like
The behavior of graceful node shutdown is that Kubernetes respects the readiness probe during the shutdown process. To be removed from rotation eagerly, you must return false from your readiness probe during the preStop phase. Deleting the pod during a "node isn't coming back" is a separate signal that has the same behavior as above, but would not be portable to all scenarios (naive node reboots when graceful shutdown is used to avoid disruption of workloads without draining). |
Would you happen to have an example of where this is used? I can't claim to have run a lot of different things on kube, but I have never seen naked, unmanaged pods used for anything other than demos (i.e. One issue here might be that if you want this to continue working as before graceful shutdown, the container might need to know if it has a chance of coming back. e.g. some sort of long-running distributed data crunching task would likely signal a "full disconnect" on a SIGTERM since that's the deletion signal too.
This is what I meant by "if this is a breaking change, IMO it's worth doing." This default does not match with the majority of kube clusters deployed today. Not to say the new default here should be to "break" the current behavior and prevent nodes from rejoining clusters entirely under their old identity -- the breaking change would be to start with no pods after graceful restart.
That sounds entirely right - we don't use unready endpoints anywhere in our clusters so "breaking" it probably just doesn't affect us. Your review seems accurate to me. Definitely happy to close this PR - I realized I was wrong in my diagnosis later in the thread here.
A few points here:
|
Correct - what the KEP failed to implement (and is obvious in retrospect) was there was no design to implement the desirable property that “gracefully shutdown pods are aggressively removed from load balanced services eagerly unless the workload owner opts in to a slower removal”. Deleting the pod isn’t the right general choice than that, but in the KEP we failed to design the mechanism that would result in graceful pods temporarily stopping. The right fix is to correct that oversight via a new signal that indicates the kubelet will be stopping the pod (desirable for other use cases) and have endpoints treat that as identical to “deleting the pod”. A secondary fix would be to standardize node drain via a control plane signal, which then all implementations could honor (being discussed on sig arch mailing list for other reasons). A provider could then correctly map their specific instance termination signals to that request, avoiding divergent drain implementations. Regarding pods without controllers, it’s fairly common to orchestrate pods directly via non-kube-centric orchesrtators (batch frameworks, CI/CD systems) vs workload controllers, and in many cases those are run in environments where pods survive power outages. Deleting the pod would violate current behavior and potentially lead to data loss for any metadata attached to the pod. |
Kube as a project has chosen to not change default behavior over time so as to reduce "surprise" for existing users, and that puts a higher burden on new users (more new specific things to learn). Because we have existing users with large fleets in "forever" node environments, we can't just delete the pods - it would be user-hostile and hostile to their non-controller automation. I agree with you that this is surprising and I agree we should be finding a way to either improve graceful node shutdown via the KEP for GA or spawn a new KEP to formally define an a way to indicate workloads will be temporarily disrupted and endpoints should properly react (that'll be new API surface area). |
|
Not sure whether the issue we are seeing is similar to this report or not. Using 1.24 version |
|
In reviewing the behavior of the kubelet in this scenario, I'm taking a look at the code paths for termination to identify whether we should be marking the pod unready from the kubelet if we know it will be removed. my previous concern was around static pods, but static pods should propagate local deletion to the apiserver consistent with regular pods, and marking the pod not ready aggressively might be correct for the graceful case. |
|
Updated: @mikekap your triage in #116965 (comment) is accurate - the deferral of reporting a terminal phase implemented by #108366 (which is what allows components on the cluster to know for certain when a pod has cleaned up all of its running resources, which is required for safety and scheduling new capacity to the node) exposed a flaw in how the pod prober manager (which owns readiness state) tracks readiness. The pod is still considered running, so prober_manager.go#UpdatePodStatus continues to leave the pod as ready because it doesn't know that the pod will be terminal. As mentioned earlier up thread, when we are certain a pod will be terminating, that means that it will eventually be disrupted no matter how long the pod reports ready (in graceful node shutdown, for some period of time). I was originally concerned that static pods could be disrupted if we were to return unready too early, but we have a mitigation for that which is But a different and more important argument that we neglected to focus on is that ready propagation is a one way signal for basic services - it should be fast, but we cannot be certain when it will fully propagate to all consuming network components. This requires us to design unready propagation to be "eager" so that we maximize the chance that clients are updated before an expected disruption happens. Endpoint controller originally added the deletionTimestamp as a signal for removing a pod from endpoints eagerly for this reason - the pod will eventually be disrupted, in advance of when the controller observes the kubelet set the ready state (the kubelet may also be down). However, the ready state on the pod is owned by the kubelet, so if we can be certain that there is no existing use case that depends on pods remaining ready during graceful termination, we can eagerly set ready=false.
I need some additional review to ensure that there aren't legitimate workloads that would be impacted by eagerly going unready when the kubelet knows the pod will be shut down, but if we get that certainty then the fix will probably be:
Will open a PoC PR to see what else might break. The e2e test would be a simple variation of the yaml at the top of this PR and confirmation that ready propagates quickly and potentially verify the same conditions as #125404 |
8 tasks
|
As part of the exploration of this, I've created a design proposal for the solution that would address this - please read https://docs.google.com/document/d/1t25jgO_-LRHhjRXf4KJ5xY_t8BZYdapv7MDAxVGY6R8/edit?usp=sharing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
When graceful node shutdown is enabled, I expect that services are drained appropriately during pod shutdown (i.e. the same way that
kubectl delete podwould work). However this doesn't seem to be the case - the pod never enters theTerminatingstate according tokubectland doesn't get removed from the service endpoints until it enters theCompletedorFailedstate. This causes connection errors and lost requests inCluster-routed services.What did you expect to happen?
After pod termination starts, the EndpointSlice should be updated to have the pod that is terminating removed or marked terminating, even before the pod is fully shut down. This is what happens when a pod gets removed for any other reason (e.g.
kubectl delete pod).How can we reproduce it (as minimally and precisely as possible)?
You can see the repro at https://github.com/mikekap/vagrant-kubeadm-kubernetes/tree/graceful-shutdown-repro . The repro sets up a 2 node cluster with graceful shutdown enabled and a simple deployment/service.
Here is the deployment/service yaml pasted again
To reproduce:
watch -n 1 'kubectl describe endpointslice my-service-5j2nw'in a separate terminal (you'll need to figure out the right name unfortunately)Within 20 seconds (the preStop hook duration), the EndpointSlice should be updated to have the pod that is terminating removed. This is what happens when a pod gets removed for any other reason (e.g.
kubectl delete pod).To ensure that everything else worked, you can reboot the node and see the journal logs (via something like
journalctl -b -1) of the shutdown sequence triggering the kubelet graceful shutdown logic.Anything else we need to know?
No response
Kubernetes version
Cloud provider
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
The text was updated successfully, but these errors were encountered: