Skip to content

Commit

Permalink
Merge pull request #56093 from lpettyjo/OSDOCS-3790
Browse files Browse the repository at this point in the history
OSDOCS-3790:CSI driver vol detach w/ non-graceful node shutdown
  • Loading branch information
lpettyjo committed Apr 19, 2023
2 parents 9d09ac3 + b86bcb9 commit 98a6ab8
Show file tree
Hide file tree
Showing 4 changed files with 83 additions and 0 deletions.
2 changes: 2 additions & 0 deletions _topic_maps/_topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1517,6 +1517,8 @@ Topics:
File: persistent-storage-csi-cloning
- Name: CSI automatic migration
File: persistent-storage-csi-migration
- Name: Detach CSI volumes after non-graceful node shutdown
File: persistent-storage-csi-vol-detach-non-graceful-shutdown
- Name: AliCloud Disk CSI Driver Operator
File: persistent-storage-csi-alicloud-disk
- Name: AWS Elastic Block Store CSI Driver Operator
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc
//

:_content-type: CONCEPT
[id="persistent-storage-csi-vol-detach-non-graceful-overview_{context}"]
= Overview

A graceful node shutdown occurs when the kubelet's node shutdown manager detects the upcoming node shutdown action. Non-graceful shutdowns occur when the kubelet does not detect a node shutdown action, which can occur because of system or hardware failures. Also, the kubelet may not detect a node shutdown action when the shutdown command does not trigger the Inhibitor Locks mechanism used by the kubelet on Linux, or because of a user error, for example, if the shutdownGracePeriod and shutdownGracePeriodCriticalPods details are not configured correctly for that node.

With this feature, when a non-graceful node shutdown occurs, you can manually add an `out-of-service` taint on the node to allow volumes to automatically detach from the node.
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc
//

:_content-type: PROCEDURE
[id="persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure_{context}"]
= Adding an out-of-service taint manually for automatic volume detachment

.Prerequisites

* Access to the cluster with cluster-admin privileges.
.Procedure

To allow volumes to detach automatically from a node after a non-graceful node shutdown:

. After a node is detected as unhealthy, shut down the worker node.

. Ensure that the node is shutdown by running the following command and checking the status:
+
[source, terminal]
----
oc get node <node name> <1>
----
<1> <node name> = name of the non-gracefully shutdown node

. Taint the corresponding node object by running the following command:
+
[source, terminal]
----
oc adm taint node <node name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute <1>
----
<1> <node name> = name of the non-gracefully shutdown node
+
After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node.
+
.Example
+
The resulting YAML file resembles the following:
+
[source, yaml]
----
spec:
taints:
- effect: NoExecute
key: node.kubernetes.io/out-of-service
value: nodeshutdown
----

. Restart the node.

. Remove the taint.
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
:_content-type: ASSEMBLY
[id="ephemeral-storage-csi-vol-detach-non-graceful-shutdown"]
= Detach CSI volumes after non-graceful node shutdown
include::_attributes/common-attributes.adoc[]
:context: ephemeral-storage-csi-vol-detach-non-graceful-shutdown

toc::[]

This feature allows Container Storage Interface (CSI) drivers to automatically detach volumes when a node goes down non-gracefully.

:FeatureName: Detach CSI volumes after non-graceful node shutdown
include::snippets/technology-preview.adoc[leveloffset=+1]

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc[leveloffset=+1]

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc[leveloffset=+1]

0 comments on commit 98a6ab8

Please sign in to comment.