-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #56093 from lpettyjo/OSDOCS-3790
OSDOCS-3790:CSI driver vol detach w/ non-graceful node shutdown
- Loading branch information
Showing
4 changed files
with
83 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
12 changes: 12 additions & 0 deletions
12
modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
// Module included in the following assemblies: | ||
// | ||
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc | ||
// | ||
|
||
:_content-type: CONCEPT | ||
[id="persistent-storage-csi-vol-detach-non-graceful-overview_{context}"] | ||
= Overview | ||
|
||
A graceful node shutdown occurs when the kubelet's node shutdown manager detects the upcoming node shutdown action. Non-graceful shutdowns occur when the kubelet does not detect a node shutdown action, which can occur because of system or hardware failures. Also, the kubelet may not detect a node shutdown action when the shutdown command does not trigger the Inhibitor Locks mechanism used by the kubelet on Linux, or because of a user error, for example, if the shutdownGracePeriod and shutdownGracePeriodCriticalPods details are not configured correctly for that node. | ||
|
||
With this feature, when a non-graceful node shutdown occurs, you can manually add an `out-of-service` taint on the node to allow volumes to automatically detach from the node. |
53 changes: 53 additions & 0 deletions
53
modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
// Module included in the following assemblies: | ||
// | ||
// * storage/container_storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc | ||
// | ||
|
||
:_content-type: PROCEDURE | ||
[id="persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure_{context}"] | ||
= Adding an out-of-service taint manually for automatic volume detachment | ||
|
||
.Prerequisites | ||
|
||
* Access to the cluster with cluster-admin privileges. | ||
.Procedure | ||
|
||
To allow volumes to detach automatically from a node after a non-graceful node shutdown: | ||
|
||
. After a node is detected as unhealthy, shut down the worker node. | ||
|
||
. Ensure that the node is shutdown by running the following command and checking the status: | ||
+ | ||
[source, terminal] | ||
---- | ||
oc get node <node name> <1> | ||
---- | ||
<1> <node name> = name of the non-gracefully shutdown node | ||
|
||
. Taint the corresponding node object by running the following command: | ||
+ | ||
[source, terminal] | ||
---- | ||
oc adm taint node <node name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute <1> | ||
---- | ||
<1> <node name> = name of the non-gracefully shutdown node | ||
+ | ||
After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node. | ||
+ | ||
.Example | ||
+ | ||
The resulting YAML file resembles the following: | ||
+ | ||
[source, yaml] | ||
---- | ||
spec: | ||
taints: | ||
- effect: NoExecute | ||
key: node.kubernetes.io/out-of-service | ||
value: nodeshutdown | ||
---- | ||
|
||
. Restart the node. | ||
|
||
. Remove the taint. |
16 changes: 16 additions & 0 deletions
16
..._storage_interface/persistent-storage-csi-vol-detach-non-graceful-shutdown.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
:_content-type: ASSEMBLY | ||
[id="ephemeral-storage-csi-vol-detach-non-graceful-shutdown"] | ||
= Detach CSI volumes after non-graceful node shutdown | ||
include::_attributes/common-attributes.adoc[] | ||
:context: ephemeral-storage-csi-vol-detach-non-graceful-shutdown | ||
|
||
toc::[] | ||
|
||
This feature allows Container Storage Interface (CSI) drivers to automatically detach volumes when a node goes down non-gracefully. | ||
|
||
:FeatureName: Detach CSI volumes after non-graceful node shutdown | ||
include::snippets/technology-preview.adoc[leveloffset=+1] | ||
|
||
include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc[leveloffset=+1] | ||
|
||
include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc[leveloffset=+1] |