Update Node Maintenance and Kubernetes Upgrade Guide

longhorn-3304
longhorn-5549
longhorn-5585

Signed-off-by: Phan Le <phan.le@suse.com>

content/docs/1.3.3/deploy/important-notes/index.md

@@ -7,7 +7,8 @@ This page lists important notes for Longhorn v{{< current-version >}}.
 Please see [here](https://github.com/longhorn/longhorn/releases/tag/v{{< current-version >}}) for the full release note.
 
 ## Notes
-1. Please ensure your Kubernetes cluster is at least v1.18 and at most v1.24 before upgrading to Longhorn v{{< current-version >}} because the supported Kubernetes version has been updated (>= v1.18 and <= v1.24) in v{{< current-version >}}.
+1. We introduce a new setting [Node Drain Policy](../../references/settings#node-drain-policy) which will deprecate the old setting [Allow Node Drain with the Last Healthy Replica](../../references/settings#allow-node-drain-with-the-last-healthy-replica), so it is recommended to keep the old setting [Allow Node Drain with the Last Healthy Replica](../../references/settings#allow-node-drain-with-the-last-healthy-replica) as default `false` value and use the new setting [Node Drain Policy](../../references/settings#node-drain-policy) which has more options and controls.
+2. Please ensure your Kubernetes cluster is at least v1.18 and at most v1.24 before upgrading to Longhorn v{{< current-version >}} because the supported Kubernetes version has been updated (>= v1.18 and <= v1.24) in v{{< current-version >}}.
 2. After the upgrade, the recurring job settings of volumes will be migrated to new recurring job resources, and the `RecurringJobs` field in the volume spec will be deprecated. [[doc](https://longhorn.io/docs/{{< current-version >}}/deploy/upgrade/#4-automatically-migrate-recurring-jobs)]
 3. When upgrading from a Longhorn version >= 1.2.0 and <= v1.2.3 to v{{< current-version >}}, if your cluster has many backups, you may expect to have a long upgrade time (with 22000 backups, it could take a few hours). Helm upgrade may timeout and you may need to re-run the upgrade command or set a longer timeout. This is [a known issue](https://github.com/longhorn/longhorn/issues/3890)
 4. There are two important fixes that will prevent rarely potential data corruption during replica rebuilding, and also improve write performance via the specific filesystem block size.

content/docs/1.3.3/references/settings.md

@@ -5,7 +5,7 @@ weight: 1
 
 - [Customizing Default Settings](#customizing-default-settings)
 - [General](#general)
-  - [Allow Node Drain with the Last Healthy Replica](#allow-node-drain-with-the-last-healthy-replica)
+  - [Node Drain Policy](#node-drain-policy)
   - [Automatically Cleanup System Generated Snapshot](#automatically-cleanup-system-generated-snapshot)
   - [Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly](#automatically-delete-workload-pod-when-the-volume-is-detached-unexpectedly)
   - [Automatic Salvage](#automatic-salvage)
@@ -57,20 +57,24 @@ weight: 1
 - [Deprecated](#deprecated)
   - [Guaranteed Engine CPU](#guaranteed-engine-cpu)
   - [Disable Replica Rebuild](#disable-replica-rebuild)
+  - [Allow Node Drain with the Last Healthy Replica](#allow-node-drain-with-the-last-healthy-replica)
 
 ### Customizing Default Settings
 
 To configure Longhorn before installing it, see [this section](../../advanced-resources/deploy/customizing-default-settings) for details.
 
 ### General
 
-#### Allow Node Drain with the Last Healthy Replica
-
-> Default: `false`
+#### Node Drain Policy
 
-By default, Longhorn will block `kubectl drain` action on a node if the node contains the last healthy replica of a volume.
+> Default: `block-if-contains-last-replica`
 
-If this setting is enabled, Longhorn will not block `kubectl drain` action on a node even if the node contains the last healthy replica of a volume.
+Define the policy to use when a node with the last healthy replica of a volume is drained. Available options:
+- `block-if-contains-last-replica`: Longhorn will block the drain when the node contains the last healthy replica of a volume.
+- `allow-if-replica-is-stopped`: Longhorn will allow the drain when the node contains the last healthy replica of a volume but the replica is stopped.
+  WARNING: possible data loss if the node is removed after draining. Select this option if you want to drain the node and do in-place upgrade/maintenance.
+- `always-allow`: Longhorn will allow the drain even though the node contains the last healthy replica of a volume.
+  WARNING: possible data loss if the node is removed after draining. Also possible data corruption if the last replica was running during the draining.
 
 #### Automatically Cleanup System Generated Snapshot
 
@@ -515,3 +519,12 @@ This deprecated setting is replaced by 2 new settings "Guaranteed Engine Manager
 > Default: `false`
 
 This deprecated setting is replaced by the new setting 'Concurrent Replica Rebuild Per Node Limit'. Once the new setting value is 0, it means rebuilding disable.
+
+#### Allow Node Drain with the Last Healthy Replica
+
+> Default: `false`
+
+By default, Longhorn will block `kubectl drain` action on a node if the node contains the last healthy replica of a volume. If this setting is enabled, Longhorn will not block `kubectl drain` action on a node even if the node contains the last healthy replica of a volume.
+
+
+This deprecated setting is replaced by the new setting [Node Drain Policy](#node-drain-policy)

content/docs/1.3.3/volumes-and-nodes/maintenance.md

@@ -1,59 +1,52 @@
 ---
-title: Node Maintenance Guide
+title: Node Maintenance and Kubernetes Upgrade Guide
 weight: 6
 ---
 
-This section describes how to handle planned maintenance of nodes.
+This section describes how to handle planned node maintenance or upgrading Kubernetes version for the cluster.
 
 - [Updating Node OS or Container Runtime](#updating-the-node-os-or-container-runtime)
-- [Updating Kubernetes](#updating-kubernetes)
 - [Removing a Disk](#removing-a-disk)
 - [Removing a Node](#removing-a-node)
+- [Updating Kubernetes](#upgrading-kubernetes)
 
 ## Updating the Node OS or Container Runtime
 
 1. Cordon the node. Longhorn will automatically disable the node scheduling when a Kubernetes node is cordoned.
 
 1. Drain the node to move the workload to somewhere else.
 
-    You will need to use `--ignore-daemonsets` and `--pod-selector='app!=csi-attacher,app!=csi-provisioner,app!=longhorn-admission-webhook,app!=longhorn-conversion-webhook,app!=longhorn-driver-deployer'` options to drain the node.
-    The `--ignore-daemonsets` is needed because Longhorn deployed some daemonsets such as `Longhorn manager`, `Longhorn CSI plugin`, `engine image`.
-    The `--pod-selector='app!=csi-attacher,app!=csi-provisioner,app!=longhorn-admission-webhook,app!=longhorn-conversion-webhook,app!=longhorn-driver-deployer'` is needed so that Longhorn can properly detaches Longhorn volumes (see the [GitHub issue](https://github.com/longhorn/longhorn/issues/3304) for more detail).
+   You will need to use `--ignore-daemonsets` and `--force` to drain the node.
+   The `--ignore-daemonsets` is needed because Longhorn deployed some daemonsets such as `Longhorn manager`, `Longhorn CSI plugin`, `engine image`.
+   The `--force` is needed because some Longhorn pods (instance manager and share manager pods) are not managed by a ReplicationController, Job, or DaemonSet.
 
-    The replica processes on the node will be stopped at this stage. Replicas on
-    the node will be shown as `Failed`.
+   The running replicas on the node will be stopped at this stage. They will be shown as `Failed`.
 
-        Note: By default, if there is one last healthy replica for a volume on
-        the node, Longhorn will prevent the node from completing the drain
-        operation, to protect the last replica and prevent the disruption of the
-        workload. You can either override the behavior in the setting, or evict
-        the replica to other nodes before draining.
+   > **Note:**
+   > By default, if there is one last healthy replica for a volume on
+   > the node, Longhorn will prevent the node from completing the drain
+   > operation, to protect the last replica and prevent the disruption of the
+   > workload. You can control this behavior in the setting [Node Drain Policy](../../references/settings#node-drain-policy), or [evict
+   > the replica to other nodes before draining](../disks-or-nodes-eviction).
 
-    The engine processes on the node will be migrated with the Pod to other nodes.
+   The engine processes on the node will be migrated with the Pod to other nodes.
+   > **Note:** For volumes that are not attached through the CSI flow on the node (for example, manually attached using UI),
+   > they will not be automatically attached to new nodes by Kubernetes during the draining.
+   > Therefore, Longhorn will prevent the node from completing the drain operation.
+   > User would need to handle detachment for these volumes to unblock the draining.
 
-        Note: If there are volumes not created by Kubernetes on the node,
-        Longhorn will prevent the node from completing the drain operation, to
-        prevent the potential workload disruption.
+   After the `drain` is completed, there should be no engine or replica process running on the node. Two instance managers will still be running on the node, but they're stateless and won't cause interruption to the existing workload.
 
-    After the `drain` is completed, there should be no engine or replica process running on the node. Two instance managers will still be running on the node, but they're stateless and won't cause interruption to the existing workload.
-
-        Note: Normally you don't need to evict the replicas before the drain
-        operation, as long as you have healthy replicas on other nodes. The replicas
-        can be reused later, once the node back online and uncordoned.
+   > **Note:** Normally you don't need to evict the replicas before the drain
+   > operation, as long as you have healthy replicas on other nodes. The replicas
+   > can be reused later, once the node back online and uncordoned.
 
 1. Perform the necessary maintenance, including shutting down or rebooting the node.
 1. Uncordon the node. Longhorn will automatically re-enable the node scheduling.
-
-    If there are existing replicas on the node, Longhorn might use those
-    replicas to speed up the rebuilding process. You can set the `Replica
-    Replenishment Wait Interval` setting to customize how long Longhorn should
-    wait for potentially reusable replica to be available.
-
-## Updating Kubernetes
-
-Follow the official [Kubernetes upgrade documentation.](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)
-
-* If Longhorn is installed as a Rancher catalog app, follow [Rancher's Kubernetes upgrade guide](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/upgrading-kubernetes/#upgrading-the-kubernetes-version) to upgrade Kubernetes.
+   If there are existing replicas on the node, Longhorn might use those
+   replicas to speed up the rebuilding process. You can set the [Replica
+   Replenishment Wait Interval](../../references/settings#replica-replenishment-wait-interval) setting to customize how long Longhorn should
+   wait for potentially reusable replica to be available.
 
 ## Removing a Disk
 To remove a disk:
@@ -71,14 +64,37 @@ To remove a node:
 1. Evict all the replicas on the node.
 1. Detach all the volumes on the node.
 
-    If the node has been drained, all the workloads should be migrated to another node already.
+   If the node has been drained, all the workloads should be migrated to another node already.
 
-    If there are any other volumes remaining attached, detach them before continuing.
+   If there are any other volumes remaining attached, detach them before continuing.
 
 1. Remove the node from Longhorn using the `Delete` in the `Node` tab.
 
-    Or, remove the node from Kubernetes, using:
+   Or, remove the node from Kubernetes, using:
 
         kubectl delete node <node-name>
 
 1. Longhorn will automatically remove the node from the cluster.
+
+## Upgrading Kubernetes
+
+### In-place Upgrade
+In-place upgrade is upgrading method in which nodes are upgraded without being removed from the cluster.
+Some example solutions that use this upgrade methods are [k3s automated upgrades](https://docs.k3s.io/upgrades/automated), [Rancher's Kubernetes upgrade guide](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/upgrading-kubernetes/#upgrading-the-kubernetes-version),
+[Kubeadm upgrade](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/), etc...
+
+With the assumption that node and disks are not being deleted/removed, the recommended upgrading guide is:
+1. You should cordon and drain a node before upgrading Kubernetes components on a node.
+   Draining instruction is similar to the drain instruction at [Updating the Node OS or Container Runtime](#updating-the-node-os-or-container-runtime)
+2. The drain `--timeout` should be big enough so that replica rebuildings on healthy node can finish.
+   The more Longhorn replicas you have on the draining node, the more time it takes for the Longhorn replicas to be rebuilt on other healthy nodes.
+   We recommending you to test and select a big enough value or set it to 0 (aka never timeout).
+3. The number of nodes doing upgrade at a time should be smaller than the number of Longhorn replicas for each volume.
+   This is so that a running Longhorn volume has at least one healthy replica running at a time.
+4. Set the setting [Node Drain Policy](../../references/settings#node-drain-policy) to `allow-if-replica-is-stopped` so that the drain is not blocked by the last healthy replica of a detached volume.
+
+
+### Managed Kubernetes
+See the instruction at [Support Managed Kubernetes Service](../../advanced-resources/support-managed-k8s-service)
+
+

content/docs/1.4.2/deploy/important-notes/index.md

@@ -70,3 +70,6 @@ Fast replica rebuilding is supported by Longhorn, and is enabled by default. The
 ### Each Kubernetes Node Must Have a Unique Hostname for RWX Volumes
 Longhorn has a dedicated recovery backend service for NFS servers in the share-manager pods used by the RWX volumes. The clients' information, including its hostname, will be stored in the recovery backend. The information will be used for connection recovery if the share-manager pod is abnormally terminated and a new one is created. The [environment check script](https://raw.githubusercontent.com/longhorn/longhorn/v{{< current-version >}}/scripts/environment_check.sh) helps users to check all nodes have unique hostnames.
 More information please refer to [ReadWriteMany (RWX) Volume](../../advanced-resources/rwx-workloads/index.html).
+
+### Node Drain Policy
+We introduce a new setting [Node Drain Policy](../../references/settings#node-drain-policy) which will deprecate the old setting [Allow Node Drain with the Last Healthy Replica](../../references/settings#allow-node-drain-with-the-last-healthy-replica), so it is recommended to keep the old setting [Allow Node Drain with the Last Healthy Replica](../../references/settings#allow-node-drain-with-the-last-healthy-replica) as default `false` value and use the new setting [Node Drain Policy](../../references/settings#node-drain-policy) which has more options and controls.

content/docs/1.4.2/references/settings.md

@@ -5,7 +5,7 @@ weight: 1
 
 - [Customizing Default Settings](#customizing-default-settings)
 - [General](#general)
-  - [Allow Node Drain with the Last Healthy Replica](#allow-node-drain-with-the-last-healthy-replica)
+  - [Node Drain Policy](#node-drain-policy)
   - [Automatically Cleanup System Generated Snapshot](#automatically-cleanup-system-generated-snapshot)
   - [Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly](#automatically-delete-workload-pod-when-the-volume-is-detached-unexpectedly)
   - [Automatic Salvage](#automatic-salvage)
@@ -71,20 +71,24 @@ weight: 1
 - [Deprecated](#deprecated)
   - [Disable Replica Rebuild](#disable-replica-rebuild)
   - [Custom mkfs.ext4 parameters](#custom-mkfsext4-parameters)
+  - [Allow Node Drain with the Last Healthy Replica](#allow-node-drain-with-the-last-healthy-replica)
 
 ### Customizing Default Settings
 
 To configure Longhorn before installing it, see [this section](../../advanced-resources/deploy/customizing-default-settings) for details.
 
 ### General
 
-#### Allow Node Drain with the Last Healthy Replica
-
-> Default: `false`
+#### Node Drain Policy
 
-By default, Longhorn will block `kubectl drain` action on a node if the node contains the last healthy replica of a volume.
+> Default: `block-if-contains-last-replica`
 
-If this setting is enabled, Longhorn will not block `kubectl drain` action on a node even if the node contains the last healthy replica of a volume.
+Define the policy to use when a node with the last healthy replica of a volume is drained. Available options:
+- `block-if-contains-last-replica`: Longhorn will block the drain when the node contains the last healthy replica of a volume.
+- `allow-if-replica-is-stopped`: Longhorn will allow the drain when the node contains the last healthy replica of a volume but the replica is stopped.
+  WARNING: possible data loss if the node is removed after draining. Select this option if you want to drain the node and do in-place upgrade/maintenance.
+- `always-allow`: Longhorn will allow the drain even though the node contains the last healthy replica of a volume.
+  WARNING: possible data loss if the node is removed after draining. Also possible data corruption if the last replica was running during the draining.
 
 #### Automatically Cleanup System Generated Snapshot
 
@@ -332,7 +336,7 @@ Hashing snapshot disk files impacts the performance of the system. The immediate
 
 > Default: `0 0 */7 * *`
 
-Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files. 
+Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files.
 > **Warning**
 > Hashing snapshot disk files impacts the performance of the system. It is recommended to run data integrity checks during off-peak times and to reduce the frequency of checks.
 
@@ -650,4 +654,13 @@ This deprecated setting is replaced by the new setting `Concurrent Replica Rebui
 
 Allows setting additional filesystem creation parameters for ext4. For older host kernels it might be necessary to disable the optional ext4 metadata_csum feature by specifying `-O ^64bit,^metadata_csum`.
 
-This deprecated setting is replaced by the new setting [`mkfsParams` in the StorageClass](../../volumes-and-nodes/create-volumes/#creating-longhorn-volumes-with-kubectl) and planned removed from v1.5.0. 
+This deprecated setting is replaced by the new setting [`mkfsParams` in the StorageClass](../../volumes-and-nodes/create-volumes/#creating-longhorn-volumes-with-kubectl) and planned removed from v1.5.0.
+
+#### Allow Node Drain with the Last Healthy Replica
+
+> Default: `false`
+
+By default, Longhorn will block `kubectl drain` action on a node if the node contains the last healthy replica of a volume. If this setting is enabled, Longhorn will not block `kubectl drain` action on a node even if the node contains the last healthy replica of a volume.
+
+
+This deprecated setting is replaced by the new setting [Node Drain Policy](#node-drain-policy)