Add support scripts to drain a node and backup old resources (#741)

These scripts will be useful for cleanups and support. Have added necessary documentation for these tools.
minio · Apr 13, 2023 · 366f244 · 366f244
1 parent 8e8a598
commit 366f244
Show file tree

Hide file tree

Showing 6 changed files with 177 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -92,6 +92,7 @@ Please review the [security checklist](./security-checklist.md) before deploying
 - [Scheduling Guide](./docs/scheduling.md)
 - [Drive Replacement Guide](./docs/drive-replacement.md)
 - [Volume Expansion](./docs/volume-expansion.md)
+- [Drain a node](./docs/drain-node.md)
 - [Driver Specification](./docs/specification.md)
 - [Monitoring & Metrics](./docs/metrics.md)
 - [Developer Guide](./docs/development-and-testing.md)

diff --git a/docs/drain-node.md b/docs/drain-node.md
@@ -0,0 +1,27 @@
+Drain a node
+-------------
+
+Draining will forcefully remove the DirectPV resources from a node. This [bash script](./tools/drain.sh) can be used for draining and has to be executed cautiously as it is an irreversible operation and may incur data loss.
+
+You can consider draining a node in the following circumstances
+
+#### When a node is detached from kubernetes
+
+If a node which was used by DirectPV is detached from kubernetes, the DirectPV resources from that node will remain intact until the resources are drained.
+
+#### When DirectPV is unselected to run on a specific node
+
+If a node which was used by DirectPV is decided to be a "non-storage" node.
+
+For example, If DirectPV is decided to not run on a specific node by changing the node-selectors like the following example
+
+```sh
+$ kubectl directpv uninstall
+$ kubectl directpv install --node-selector node-label-key=node-label-value
+```
+
+the resources from the detached node can then be cleaned up by
+
+```sh
+./drain.sh <node-name>
+```
diff --git a/docs/tools/drain.sh b/docs/tools/drain.sh
@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+#
+# This file is part of MinIO DirectPV
+# Copyright (c) 2023 MinIO, Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# This script drains the DirectPV resources from a selected node
+# 
+# **CAUTION**
+#
+# This operation is irreversible and may incur data loss if not used cautiously.
+#
+
+set -e -C -o pipefail
+
+
+function drain() {
+    selector="directpv.min.io/node=${2}"
+
+    # unset the finalizers
+    kubectl get "${1}" --selector="${selector}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r resource_name; do
+        kubectl patch "${1}" "${resource_name}" -p '{"metadata":{"finalizers":null}}' --type=merge
+    done
+
+    # delete the objects
+    kubectl delete "${1}" --selector="${selector}" --ignore-not-found=true
+}
+
+function init() {
+
+    if [[ $# -ne 1 ]]; then
+        echo "usage: drain.sh <NODE>"
+        echo
+        echo "This script forcefully removes all the DirectPV resources from the node"
+        echo "This operation is irreversible and may incur data loss if not used cautiously."
+        exit 255
+    fi
+
+    if ! which kubectl >/dev/null 2>&1; then
+        echo "kubectl not found; please install"
+        exit 255
+    fi
+
+    if kubectl get csinode "${1}" -o go-template="{{range .spec.drivers}}{{if eq .name \"directpv-min-io\"}}{{.name}}{{end}}{{end}}" --ignore-not-found | grep -q .; then
+        echo "the node is still under use by DirectPV CSI Driver; please remove DirectPV installation from the node to drain"
+        exit 255
+    fi
+}
+
+function main() {
+    node="$1"
+
+    drain "directpvvolumes" "${node}"
+    drain "directpvdrives" "${node}"
+    drain "directpvinitrequests" "${node}"
+    kubectl delete directpvnode "${node}" --ignore-not-found=true
+}
+
+init "$@"
+main "$@"
diff --git a/docs/tools/remove-directcsi.sh b/docs/tools/remove-directcsi.sh
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+#
+# This file is part of MinIO DirectPV
+# Copyright (c) 2023 MinIO, Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# This script removes direct-csi drives and volumes after taking backup YAMLs
+# to directcsidrives.yaml and directcsivolumes.yaml
+#
+
+set -e -C -o pipefail
+
+function init() {
+    if [[ $# -ne 0 ]]; then
+        echo "usage: remove-directcsi.sh"
+        echo
+        echo "This script removes direct-csi drives and volumes after taking backup YAMLs"
+        echo "to directcsidrives.yaml and directcsivolumes.yaml"
+        exit 255
+    fi
+
+    if ! which kubectl >/dev/null 2>&1; then
+        echo "kubectl not found; please install"
+        exit 255
+    fi
+}
+
+# usage: unset_object_finalizers <resource>
+function unset_object_finalizers() {
+    kubectl get "${1}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r resource_name; do
+        kubectl patch "${1}" "${resource_name}" -p '{"metadata":{"finalizers":null}}' --type=merge
+    done
+}
+
+function main() {
+    kubectl get directcsivolumes -o yaml > directcsivolumes.yaml
+    kubectl get directcsidrives -o yaml > directcsidrives.yaml
+
+    # unset the finalizers
+    unset_object_finalizers "directcsidrives"
+    unset_object_finalizers "directcsivolumes"
+
+    # delete the resources
+    kubectl delete directcsivolumes --all
+    kubectl delete directcsidrives --all
+}
+
+init "$@"
+main "$@"
diff --git a/docs/tools/replace.sh b/docs/tools/replace.sh
@@ -16,6 +16,10 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+#
+# This script replaces source drive to destination drive in the specified node
+#
+
 set -e
 
 # usage: get_drive_id <node> <drive-name>
@@ -47,6 +51,8 @@ function get_pod_namespace() {
 function init() {
     if [[ $# -eq 4 ]]; then
         echo "usage: replace.sh <NODE> <SRC-DRIVE> <DEST-DRIVE>"
+        echo
+        echo "This script replaces source drive to destination drive in the specified node"
         exit 255
     fi
 
@@ -93,8 +99,8 @@ function main() {
     fi
 
     mapfile -t volumes < <(get_volumes "${src_drive_id}")
-
-    for volume in "${volumes[@]}"; do
+    IFS=' ' read -r -a volumes_arr <<< "${volumes[@]}"
+    for volume in "${volumes_arr[@]}"; do
         pod_name=$(get_pod_name "${volume}")
         pod_namespace=$(get_pod_namespace "${volume}")
 
@@ -104,10 +110,11 @@ function main() {
         fi
     done
 
-    if [ "${#volumes[@]}" -gt 0 ]; then
+    if [ "${#volumes_arr[@]}" -gt 0 ]; then
         # Wait for associated DirectPV volumes to be unbound
-        while kubectl directpv list volumes --no-headers "${volumes[@]}" | grep -q Bounded; do
+        while kubectl directpv list volumes --no-headers "${volumes_arr[@]}" | grep -q Bounded; do
             echo "...waiting for volumes to be unbound"
+            sleep 10
         done
     else
         echo "no volumes found in source drive ${src_drive} on node ${node}"

diff --git a/docs/upgrade.md b/docs/upgrade.md
@@ -34,6 +34,8 @@ quay.io/minio/csi-resizer:v1.7.0
 
 - If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18`
 
-If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes.
+- If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes.
 
 You need to change `direct.csi.min.io/access-tier: <your_access_tier_value>` to `directpv.min.io/access-tier: <your_access_tier_value>` in the respective storage class parameters section.
+
+- The older CRDs (directcsidrives and directcsivolumes) are deprecated and not used in versions > v4.0.0, it can be removed after upgrading. Please use the [bash script](./tools/remove-directcsi.sh) to remove the older objects after upgrading to latest.