From 6e0e6501546b05182aae196216c2724584424683 Mon Sep 17 00:00:00 2001 From: Lisa Pettyjohn Date: Fri, 17 Feb 2023 10:41:09 -0500 Subject: [PATCH] OSDOCS-3790:CSI driver vol detach w/ non-graceful node shutdown --- _topic_maps/_topic_map.yml | 2 + ...detach-non-graceful-shutdown-overview.adoc | 12 +++++ ...etach-non-graceful-shutdown-procedure.adoc | 53 +++++++++++++++++++ ...-csi-vol-detach-non-graceful-shutdown.adoc | 16 ++++++ 4 files changed, 83 insertions(+) create mode 100644 modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc create mode 100644 modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc create mode 100644 storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index 72b890ee1c27..e052eb4b2f79 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -1516,6 +1516,8 @@ Topics: File: persistent-storage-csi-cloning - Name: CSI automatic migration File: persistent-storage-csi-migration + - Name: Detach CSI volumes after non-graceful node shutdown + File: persistent-storage-csi-vol-detach-non-graceful-shutdown - Name: AliCloud Disk CSI Driver Operator File: persistent-storage-csi-alicloud-disk - Name: AWS Elastic Block Store CSI Driver Operator diff --git a/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc b/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc new file mode 100644 index 000000000000..ad545f520aee --- /dev/null +++ b/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc @@ -0,0 +1,12 @@ +// Module included in the following assemblies: +// +// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc +// + +:_content-type: CONCEPT +[id="persistent-storage-csi-vol-detach-non-graceful-overview_{context}"] += Overview + +A graceful node shutdown occurs when the kubelet's node shutdown manager detects the upcoming node shutdown action. Non-graceful shutdowns occur when the kubelet does not detect a node shutdown action, which can occur because of system or hardware failures. Also, the kubelet may not detect a node shutdown action when the shutdown command does not trigger the Inhibitor Locks mechanism used by the kubelet on Linux, or because of a user error, for example, if the shutdownGracePeriod and shutdownGracePeriodCriticalPods details are not configured correctly for that node. + +With this feature, when a non-graceful node shutdown occurs, you can manually add an `out-of-service` taint on the node to allow volumes to automatically detach from the node. \ No newline at end of file diff --git a/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc b/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc new file mode 100644 index 000000000000..63ddb616eed1 --- /dev/null +++ b/modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc @@ -0,0 +1,53 @@ +// Module included in the following assemblies: +// +// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc +// + +:_content-type: PROCEDURE +[id="persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure_{context}"] += Adding an out-of-service taint manually for automatic volume detachment + +.Prerequisites + +* Access to the cluster with cluster-admin privileges. + +.Procedure + +To allow volumes to detach automatically from a node after a non-graceful node shutdown: + +. After a node is detected as unhealthy, shut down the worker node. + +. Ensure that the node is shutdown by running the following command and checking the status: ++ +[source, terminal] +---- +oc get node <1> +---- +<1> = name of the non-gracefully shutdown node + +. Taint the corresponding node object by running the following command: ++ +[source, terminal] +---- +oc adm taint node node.kubernetes.io/out-of-service=nodeshutdown:NoExecute <1> +---- +<1> = name of the non-gracefully shutdown node ++ +After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node. ++ +.Example ++ +The resulting YAML file resembles the following: ++ +[source, yaml] +---- +spec: + taints: + - effect: NoExecute + key: node.kubernetes.io/out-of-service + value: nodeshutdown +---- + +. Restart the node. + +. Remove the taint. diff --git a/storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc b/storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc new file mode 100644 index 000000000000..d519eb9e9bb9 --- /dev/null +++ b/storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc @@ -0,0 +1,16 @@ +:_content-type: ASSEMBLY +[id="ephemeral-storage-csi-vol-detach-non-graceful-shutdown"] += Detach CSI volumes after non-graceful node shutdown +include::_attributes/common-attributes.adoc[] +:context: ephemeral-storage-csi-vol-detach-non-graceful-shutdown + +toc::[] + +This feature allows Container Storage Interface (CSI) drivers to automatically detach volumes when a node goes down non-gracefully. + +:FeatureName: Detach CSI volumes after non-graceful node shutdown +include::snippets/technology-preview.adoc[leveloffset=+1] + +include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc[leveloffset=+1] + +include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc[leveloffset=+1] \ No newline at end of file