doc: describe upgrade and recovery for raft topology

Document the manual upgrade procedure that is required to enable
consistent cluster management in clusters that were upgraded from an
older version to ScyllaDB Open Source 6.0. This instruction is placed in
previously placeholder "Enable Raft-based Topology" page which is a part
of the upgrade instructions to ScyllaDB Open Source 6.0.

Add references to the new description in the "Raft Consensus Algorithm
in ScyllaDB" document in relevant places.

Extend the "Handling Node Failures" document so that it mentions steps
required during recovery of a ScyllaDB cluster running version 6.0.

Fixes: #17341

Closes #17624

docs/architecture/raft.rst

@@ -54,6 +54,10 @@ When all the nodes in the cluster are upgraded to ScyllaDB Open Source 5.5 or Sc
 Verifying that the Raft upgrade procedure finished successfully
 ========================================================================
 
+.. note::
+    After the procedure described in this section finishes, you must perform manual action in order to enable consistent topology changes.
+    See :doc:`the guide for enabling consistent topology changes</upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology>` for more details.
+
 The Raft upgrade procedure requires **full cluster availability** to correctly setup the Raft algorithm; after the setup finishes, Raft can proceed with only a majority of nodes, but this initial setup is an exception.
 An unlucky event, such as a hardware failure, may cause one of your nodes to fail. If this happens before the Raft upgrade procedure finishes, the procedure will get stuck and your intervention will be required.
 
@@ -150,6 +154,10 @@ If some nodes are **dead and irrecoverable**, you'll need to perform a manual re
 Verifying that Raft is enabled
 ===============================
 
+.. _schema-on-raft-enabled:
+
+**Schema on Raft**
+
 You can verify that Raft is enabled on your cluster by performing the following query on each node:
 
 .. code-block:: sql
@@ -174,6 +182,38 @@ If ``value`` is ``recovery``, it means that the cluster is in the middle of the
 
 If ``value`` is anything else, it might mean data corruption or a mistake when performing the manual recovery procedure. The value will be treated as if it was equal to ``recovery`` when the node is restarted.
 
+.. _verifying-consistent-topology-changes-enabled:
+
+**Consistent topology changes**
+
+You can verify that consistent topology management is enabled on your cluster in two ways:
+
+#. By querying the ``system.topology`` table:
+
+    .. code-block:: cql
+
+        cqlsh> SELECT upgrade_state FROM system.topology;
+
+   The query should return ``done`` after upgrade is complete:
+
+    .. code-block:: console
+
+        upgrade_state
+        ---------------
+                done
+
+        (1 rows)
+
+    An empty result or a value of ``not_upgraded`` means that upgrade has not started yet. Any other value means that upgrade is in progress.
+
+#. By sending a GET HTTP request to the `/`storage_service/raft_topology/upgrade`` endpoint. For example, you can do it with ``curl`` like this:
+
+    .. code-block:: bash
+
+        curl -X GET "http://127.0.0.1:10000/storage_service/raft_topology/upgrade"
+
+   It returns a JSON string, with the same meaning and value as the ``upgrade_state`` column in ``system.topology`` (see the previous point).
+
 .. _raft-schema-changes:
 
 Safe Schema Changes with Raft

docs/troubleshooting/handling-node-failures.rst

@@ -70,8 +70,9 @@ Manual Recovery Procedure
 You can follow the manual recovery procedure when:
 
 * The majority of nodes (for example, 2 out of 3) failed and are irrecoverable.
-* :ref:`The Raft upgrade procedure <verify-raft-procedure>` got stuck because one 
-  of the nodes failed in the middle of the procedure and is irrecoverable.
+* :ref:`The Raft upgrade procedure <verify-raft-procedure>`
+  or :doc:`the procedure for enabling consistent topology changes</upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology>`
+  got stuck because one of the nodes failed in the middle of the procedure and is irrecoverable.
 
 .. warning::
 
@@ -130,6 +131,7 @@ in the past and then had Raft enabled, and to clusters that were bootstrapped us
 
    .. code-block:: cql
 
+        cqlsh> TRUNCATE TABLE system.topology;
         cqlsh> TRUNCATE TABLE system.discovery;
         cqlsh> TRUNCATE TABLE system.group0_history;
         cqlsh> DELETE value FROM system.scylla_local WHERE key = 'raft_group0_id';
@@ -144,4 +146,6 @@ in the past and then had Raft enabled, and to clusters that were bootstrapped us
 
 #. Perform a :doc:`rolling restart </operating-scylla/procedures/config-change/rolling-restart/>` of your alive nodes.
 
-#. The Raft upgrade procedure will start anew. :ref:`Verify <verify-raft-procedure>` that it finishes successfully.
+#. The Raft upgrade procedure will start anew. :ref:`Verify <verify-raft-procedure>` that it finishes successfully.
+
+#. Perform :doc:`the procedure for enabling consistent topology changes </upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology>`.

docs/upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology.rst

@@ -3,4 +3,108 @@ Enable Consistent Topology Updates
 =====================================
 
 This article explains how to enable consistent topology changes
-when you upgrade from version 5.4 to 6.0.
+when you upgrade from version 5.4 to 6.0.
+
+Introduction
+============
+
+ScyllaDB Open Source 6.0 introduces :ref:`consistent topology changes based on Raft <raft-topology-changes>`.
+Newly created clusters use consistent topology changes right from the start. However - unlike in the case
+of schema managed on Raft - consistent topology changes are *not* automatically enabled after the cluster
+was upgraded from an older version of ScyllaDB. If you have such a cluster, then you need to enable
+consistent topology changes manually with a dedicated upgrade procedure.
+
+Before running the procedure, you **must** check that the cluster meets some prerequisites
+and you **must** ensure that some administrative procedures will not be run
+while the upgrade procedure is in progress.
+
+.. _enable-raft-topology-6.0-prerequisites:
+
+Prerequisites
+=============
+
+* Make sure that all nodes in the cluster are upgraded to ScyllaDB Open Source 6.0.
+* Verify that :ref:`schema on raft is enabled <schema-on-raft-enabled>`.
+* Make sure that all nodes enabled ``SUPPORTS_CONSISTENT_TOPOLOGY_CHANGES`` cluster feature.
+  One way to verify this is to look for the following message in the log:
+
+  .. code-block:: none
+
+    features - Feature SUPPORTS_CONSISTENT_TOPOLOGY_CHANGES is enabled
+
+  Alternatively, this can be checked programmatically by checking whether ``value`` column under the key ``enabled_features`` contains the name of the feature in the ``system.scylla_local`` table.
+  For example, this can be done with the following bash command:
+
+  .. code-block:: bash
+
+    until cqlsh -e "select value from system.scylla_local where key = 'enabled_features'" | grep "SUPPORTS_CONSISTENT_TOPOLOGY_CHANGES"
+    do
+        echo "Upgrade didn't finish yet on the local node, waiting 10 seconds before checking again..."
+        sleep 10
+    done
+    echo "Upgrade completed on the local node"
+
+* Make sure that all nodes are alive for the duration of the upgrade.
+
+.. _enable-raft-topology-6.0-forbidden-operations:
+
+Administrative operations which must not be running during upgrade
+==================================================================
+
+Make sure that administrative operations will not be running while upgrade is in progress.
+In particular, you must abstain from:
+
+* :doc:`Cluster management procedures </operating-scylla/procedures/cluster-management/index>` (adding, replacing, removing, decommissioning nodes etc.).
+* Running :doc:`nodetool repair </operating-scylla/nodetool-commands/repair>`.
+* Running :doc:`nodetool checkAndRepairCdcStreams </operating-scylla/nodetool-commands/checkandrepaircdcstreams>`.
+
+Running the procedure
+=====================
+
+.. warning::
+
+  Before proceeding, make sure that all the :ref:`prerequisites <enable-raft-topology-6.0-prerequisites>` are met
+  and no :ref:`forbidden administrative operations <enable-raft-topology-6.0-forbidden-operations>` will run
+  during upgrade. Failing to do so may put the cluster in an inconsistent state.
+
+Starting the upgrade procedure is done by issuing an POST HTTP request to the ``/storage_service/raft_topology/upgrade`` endpoint,
+to any of the nodes in the cluster.
+
+For example, you can do it via ``curl``, like this:
+
+.. code-block:: bash
+
+	curl -X POST "http://127.0.0.1:10000/storage_service/raft_topology/upgrade"
+
+Next, wait until all nodes report that upgrade is complete. You can check that a single node finished upgrade in one of two ways:
+
+* By sending a HTTP ``GET`` request on the ``/storage_service/raft_topology/upgrade`` endpoint. For example, you can do it with ``curl`` like this:
+
+  .. code-block:: bash
+
+      curl -X GET "http://127.0.0.1:10000/storage_service/raft_topology/upgrade"
+
+  It will return a JSON string which will be equal to ``done`` after the upgrade is complete on this node.
+
+* By querying the ``upgrade_state`` column in the ``system.topology`` table. You can use ``cqlsh`` to get the value of the column like this:
+
+  .. code-block:: bash
+
+      cqlsh -e "select upgrade_state from system.topology"
+
+  The ``upgrade_state`` column should be set to ``done`` after the upgrade is complete on this node:
+
+After the upgrade is complete on all nodes, wait at least one minute before issuing any topology changes in order to avoid data loss from writes that were started before the upgrade.
+
+What if upgrade gets stuck?
+===========================
+
+If the process gets stuck at some point, first check the status of your cluster:
+
+- If there are some nodes that are not alive, try to restart them.
+- If all nodes are alive, ensure that the network is healthy and every node can reach all other nodes.
+- If all nodes are alive and the network is healthy, perform a :doc:`rolling restart </operating-scylla/procedures/config-change/rolling-restart/>` of the cluster.
+
+If none of the above solves the issue, perform :ref:`the Raft recovery procedure <recovery-procedure>`.
+During recovery, the cluster will switch back to gossip-based topology management mechanism.
+After exiting recovery, you should upgrade the cluster to consistent topology updates using the procedure described in this document.