Skip to content

Commit

Permalink
OSDOCS-3790:CSI driver vol detach w/ non-graceful node shutdown
Browse files Browse the repository at this point in the history
  • Loading branch information
Lisa Pettyjohn committed Feb 17, 2023
1 parent 4e60745 commit 1a5c128
Show file tree
Hide file tree
Showing 4 changed files with 76 additions and 0 deletions.
2 changes: 2 additions & 0 deletions _topic_maps/_topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1460,6 +1460,8 @@ Topics:
File: persistent-storage-csi-cloning
- Name: CSI automatic migration
File: persistent-storage-csi-migration
- Name: CSI driver volume detach with non-graceful node shutdown
File: persistent-storage-csi-vol-detach-non-graceful-shutdown
- Name: AliCloud Disk CSI Driver Operator
File: persistent-storage-csi-alicloud-disk
- Name: AWS Elastic Block Store CSI Driver Operator
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc
//

:_content-type: CONCEPT
[id="persistent-storage-csi-vol-detach-non-graceful-overview_{context}"]
= Overview

A node shutdown can be graceful only if the kubelet's node shutdown manager detects the upcoming node shutdown action. Non-graceful shutdowns occur when the kubelet does not detect a node shutdown action. This can happen when the shutdown command does not trigger the Inhibitor Locks mechanism used by the kubelet on Linux, or because of a user error. For example, if the shutdownGracePeriod and shutdownGracePeriodCriticalPods details are not configured correctly for that node.

With this feature, when a non-graceful node shutdown occurs, you can manually add an `out-of-service` taint on the node to allow volumes to automatically detach from the node.
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Module included in the following assemblies:
//
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc
//

:_content-type: PROCEDURE
[id="persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure_{context}"]
= Manually adding an out-of-service taint for automatic volume detachment

.Prerequisites

* Access to the cluster with cluster-admin privileges.
.Procedure

To allow volumes to detach automatically from a node after a non-graceful node shutdown:

. After a node is detected as unhealthy, shut down the worker node.

. Taint the corresponding node object with by running the following command:
+
[source, terminal]
----
oc adm taint node/<node name> out-of-service=nodeshutdown:NoExecute <1>
----
<1> <node name> = name of the non-gracefully shutdown node
+
After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node.
+
The resulting YAML file resembles the following:
+
[source, yaml]
----
spec:
taints:
- effect: NoSchedule
key: out-of-service=nodeshutdown:NoExecute
timeAdded: "2023-02-17T17:23:03Z"
----

. After restarting the node, remove the taint.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
:_content-type: ASSEMBLY
[id="ephemeral-storage-csi-vol-detach-non-graceful-shutdown"]
= CSI driver volume detach with non-graceful node shutdown
include::_attributes/common-attributes.adoc[]
:context: ephemeral-storage-csi-vol-detach-non-graceful-shutdown

toc::[]

This feature allows Container Storage Interface (CSI) drivers to automatically detach volumes when a node goes down non-gracefully.

:FeatureName: CSI driver volume detach with non-graceful node shutdown
include::snippets/technology-preview.adoc[leveloffset=+1]

[NOTE]
====
This feature is enabled by default.
====

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc[leveloffset=+1]

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc[leveloffset=+1]

0 comments on commit 1a5c128

Please sign in to comment.