Volume hang on Karpenter Node Consolidation/Termination #1665
Assignees
Labels
kind/bug
Categorizes issue or PR as related to a bug.
Comments
|
We're running into similar issues that may not be exactly the same but related: #1677. |
|
@torredil In regards to this PR #1736 which was merged into 1.23.0. After updating the EBS CSI Driver Addon to 1.23.0 we can still observe the volume hanging containers issue (due to Volume Attachments still existing). Steps:
This led us down a bit of a rabbit hole regarding the behaviour of Looking at the nodes being provisioned by Karpenter, we saw GracefulShutdown was not enabled (I'm not sure if this behaviour is different by default on AL2?): ubuntu@ip-10-112-249-47:~$ systemd-inhibit --list
WHO UID USER PID COMM WHAT WHY
MODE
Unattended Upgrades Shutdown 0 root 555 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delayIn the UserData, adding the following enabled the Graceful Shutdown handling: echo "$(jq '.shutdownGracePeriod="30s"' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq '.shutdownGracePeriodCriticalPods="10s"' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
====
ubuntu@ip-10-112-249-141:~$ systemd-inhibit --list
WHO UID USER PID COMM WHAT WHY
MODE
Unattended Upgrades Shutdown 0 root 561 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay
kubelet 0 root 4169 kubelet shutdown Kubelet needs time to handle node shutdown delay
However, the same behaviour is observed. I imagine this is because Karpenter is removing the finalizer too early, which in turns deletes the Node object https://github.com/aws/karpenter-core/blob/main/pkg/controllers/termination/controller.go#L103-L106. Thus not letting the ebs node process update the Node object. Increasing the Graceful Node Timeout: While tailing the logs and initialising a terminate via the AWS CLI - we observed:
Mentioned in the OP, was the need for an ebs managed finializer in order to prevent the object deletion from the k8s api before it was ready. However, I'm sure this will get a bit problematic if Graceful Shutdown isn't configured correctly and the node is nuked too quickly, leaving over stale objects). Attempting to add a new finaliser - called "ebs.io/martytest". In this case, the pod seems to be successfully rescheduled on another host within a few minutes. There was an error in one case: Hope this helps |
|
We have come across similar issues but I don't think this is Karpenter related. I don't think PreStop hook works at times for multiple reasons. If a node is drained first which effectively unmounts and detaches the volumes successfully, we don't see this problem (Karpenter would do the draining for consolidation/termination). So, I would consider this an edge case. On the pre-stop-hook side;
These were the issues I've come across around pre-stop-hook and I don't think It will work as intended when an instance is terminated by an AWS call unless the node is drained first (which is the time I'm not sure if the pre-stop-hook is adding any value). As you will see from the logs below, the PVCs are unmounted/detached successfully and yet pre-stop-hook still finds the volume attachments from the k8s API (I also checked the API directly, they're left over until controller manager kicks in 6 minutes later). I think somehow the unmounts/detachments are not communicated to k8s API timely (maybe due to kubelet node status not being able to find the node anymore?) |
|
Hi @martysweet @egeturgay - appreciate the comprehensive feedback, this is really helpful 👍 A few things to cover here:
First and most importantly: the pre-stop LCH primarily aims to address the following bug in Kubernetes: Graceful node shutdown doesn't wait for volume teardown #115148. Michelle explains it very well - due to a race condition between That is to say, the pre-stop LCH is useful and important in preventing delayed attachments - especially when using the driver with Karpenter and Spot Instances - even when Separately (important distinction), configuring
Regarding the Kubelet config point:
This is addressed by awslabs/amazon-eks-ami#1544
Thank you for pointing this out! I will address this in a PR shortly.
That message observed in the logs is misleading - Kubelet is not responsible for detaching volumes, the Attach/Detach controller is, see the relevant code here: https://github.com/kubernetes/kubernetes/blob/3f7a50f38688eb332e2a1b013678c6435d539ae6/pkg/kubelet/volumemanager/reconciler/reconciler_common.go#L298 The pre-stop LCH correctly finds volume attachments from the K8s API because the associated volumes have not been fully detached from the instance. |
|
Addressed by #1895, which will be part of the upcoming /close |
|
@torredil: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@torredil I still get problem on newest version 1.28, after node has been terminate by karpenter, it still waited 6 minutes before be detached and useable in another node. What did I do
|
|
@levanlongktmt: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This was referenced Mar 2, 2024
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
/kind bug
As discussed in #1302, some users experience a considerable delay when using PersistentVolumes in combination with Karpenter.
This ticket is to capture that specific issue, highlight where the error is coming from, and discuss possible ways forward.
What happened?
During a Karpenter scaling event, it’s possible that the Kubelet cannot update its Node.Status.VolumesAttached and Node.Status.VolumesInUse values, either because EC2 TerminateInstances has been executed or the Node object has been removed. The Kubelet VolumeManager processes jobs asynchronously, so a terminated pod does not mean all the volumes have been fully unmounted and detached from the node.
Due to how the controllermanager attachdetach-controller uses this node information, the VolumeAttachment objects are not cleaned up immediately, often waiting for around 6+ minutes. Once they are cleaned up by the control plane, the EBS-CSI Controller does its job as expected, typically throwing the error “Already detached” in the logs (as the EC2 instance had detached the volumes on termination).
What you expected to happen?
If a node is deleted and the underlying EBS volumes become detached in the real world, the ebs-csi driver should be able to reflect the actual state in Kubernetes before the timeout condition triggers in the controllermanager attachdetach-controller.
Ideally, we can then catch this dangling volume condition, and reconcile the state within seconds instead of 6+ minutes. Allowing the volumes to be remounted quickly on another node.
How to reproduce it (as minimally and precisely as possible)?
Reproduction Walkthrough
make py-watch-nodesandmake py-watch-pods, to listen to updates to the applied Kubernetes objectsip-10-227-165-182.eu-central-1.compute.internal2023-06-30T09:04:30.218Z INFO controller.termination cordoned node {"commit": "26e2d35-dirty", "node": "ip-10-227-165-182.eu-central-1.compute.internal"} │ 2023-06-30T09:04:31.769Z INFO controller.termination deleted node {"commit": "26e2d35-dirty", "node": "ip-10-227-165-182.eu-central-1.compute.internal"} │ 2023-06-30T09:04:32.049Z INFO controller.machine.termination deleted machine {"commit": "26e2d35-dirty", "machine": "nodes-ubuntu-j67qz", "node": "ip-10-227-165-182.eu-central-1.compute.internal",Here is a snippet of VolumeAttachment objects after the node has been deleted - referencing the stale node. After 6 minutes, these objects will be cleaned up and recreated for the newly scheduled node.
Versions
kubectl version): v1.25.10-eks-c12679a running Ubuntu Kubelet v1.25.10Possible Solutions
There are lots of components at play in this specific issue.
To help explain all the components involved, we put together a simplified diagram.
As the ebs-csi controller, we have access to the real-world EBS Volume attachments information in the EC2 API. As such, the driver should be able to update the VolumeAttachment object to notify the attachdetach-controller of the correct state.
One possible solution to this problem could be:
csi-ebsfinalizer to the Nodecsi-ebsvolumes are detached before removing the finalizerThe advantage of the above flow, is we are just forcing a wait and allowing the standard CSI flows to happen. However, looking at the Karpenter logic, it seems that it wouldn’t care about our finalizer, terminating the instance regardless https://github.com/aws/karpenter-core/blob/1a05bdff59ced95c8868c75efdddfa77a3540092/pkg/controllers/termination/controller.go#L84-L102. Perhaps that is a bug on their side, where they should be waiting for a final delete event before proceeding with the TerminateInstance call?
Another solution could be:
In either case, missing events (say during a Controller restart) will fall back to the 6-minute stale timeout controlled by the attachdetach-controller. The aim is to make this reconciliation quicker and in a safe manner.
The text was updated successfully, but these errors were encountered: