fix(upgrade): upgrade with raft topology procedure #7295

aleksbykov · 2024-03-25T09:56:06Z

After upgrade to latest master(6.0) raft topology feature or
tablets + raft topology features will be enabled by default
To switch cluster from gossiper to raft topology, raft topology
procedure should be executed. It is described here:
https://github.com/scylladb/scylladb/blob/c5601a749e21fc710958a7c84316ecdf5943022c/docs/dev/topology-over-raft.md
section: Upgrade from legacy topology to raft-based topology

parameter 'enable_tablets_on_upgradeis used to control upgrade with/without tablets. By default tablets are enabled with scylla parameterenable_tablets` and it is added to scylla.yaml starting from 6.0. Upon upgrade this parameter should be set explicitly to true to enable tablets or set to false or not be added to scylla.yaml so tablets stay disabled after upgrade

raft topology is enabled by default, and could be disabled only with parameter enforce_gossip_topology_changes. if this parameter is added to scylla.yaml before upgrade, then raft feature: consistent topology changes will not be enabled

Testing

run upgrade from 5.4 -> 6.0.

PR pre-checks (self review)

I added the relevant backport labels
I didn't leave commented-out/debugging code

Reminders

Add New configuration option and document them (in sdcm/sct_config.py)
Add unit tests to cover my changes (under unit-test/ folder)
Update the Readme/doc folder relevent to this change (if needed)

sdcm/rest/raft_upgrade_procedure.py

upgrade_test.py

sdcm/sct_config.py

sdcm/rest/raft_upgrade_procedure.py

upgrade_test.py

temichus · 2024-04-24T11:16:57Z

blocked by scylladb/scylladb#18098

kbr-scylla · 2024-05-16T10:24:33Z

I'm unassigning myself -- it is blocked by scylladb/scylladb#18098 but that issue does not "belong" to me

mykaul · 2024-05-27T07:33:15Z

blocked by scylladb/scylladb#18098

Is this still the case?

kbr-scylla · 2024-05-27T15:00:30Z

Should be no longer -- @aleksbykov please retest

(on nightly build -- the fix is not backported to 6.0 yet)

After upgrade to latest master(6.0) raft topology feature or tablets + raft topology features will be enabled by default To switch cluster from gossiper to raft topology, raft topology procedure should be executed. It is described in scylla doc upgrade from 5.4-> 6.0 section: Upgrade from legacy topology to raft-based topology

soyacz

Is it right to put it as default and not providing tests with false?
Especially, that false path is current default for enterprise (IIRC it was not decided yet to turn on tablets by default in enterprise).

soyacz · 2024-06-10T06:19:37Z

upgrade_test.py

-            scylla_yaml_updates.update({"experimental_features": ["tablets", "consistent-topology-changes"]})
+            scylla_yaml_updates.update({"enable_tablets": True})
+
+        if self.params.get("enable_force_`gossip_topology_changes"):


typo here in param - so it looks this path was never tested

soyacz · 2024-06-10T06:29:16Z

sdcm/provision/scylla_yaml/scylla_yaml.py

@@ -339,6 +339,8 @@ def set_authorizer(cls, authorizer: str):
    audit_categories: str = None  # None
    audit_tables: str = None  # None
    audit_keyspaces: str = None  # None
+    force_gossip_topology_changes: bool = None  # False
+    enable_tablets: bool = None  # False


bit misleading comment - 6.0.0 comes with this set explicitly to True - I think worth to add this info (like we do for e.g. rpc_port).

soyacz · 2024-06-10T06:32:48Z

sdcm/sct_config.py

@@ -1550,6 +1550,10 @@ class SCTConfiguration(dict):

        dict(name="kafka_connectors", env="SCT_KAFKA_CONNECTORS", type=str_or_list_or_eval,
             help="configuration for setup up kafka connectors"),
+
+        dict(name="enable_force_gossip_topology_changes", env="SCT_ENABLE_FORCE_GOSSIP_TOPOLOGY_CHANGES", type=boolean,


this param is looking like global one while it is only used in upgrade tests - misleading.

Btw. Can't we just add force_gossip_topology_changes to append_scylla_yaml? Now when it is used as dict and merged with other append_scylla_yaml params, maybe we don't need this one?

yarongilor · 2024-06-10T15:37:21Z

defaults/test_default.yaml

@@ -250,3 +250,5 @@ teardown_validators:

 kafka_backend: null
 kafka_connectors: []
+
+enable_force_gossip_topology_changes: false


why enable_force_gossip_topology_changes and not force_gossip_topology_changes?

yarongilor · 2024-06-10T15:38:59Z

sdcm/sct_config.py

@@ -1550,6 +1550,10 @@ class SCTConfiguration(dict):

        dict(name="kafka_connectors", env="SCT_KAFKA_CONNECTORS", type=str_or_list_or_eval,
             help="configuration for setup up kafka connectors"),
+
+        dict(name="enable_force_gossip_topology_changes", env="SCT_ENABLE_FORCE_GOSSIP_TOPOLOGY_CHANGES", type=boolean,
+             help="""Enable gossip topology changes (disable raft topology)"""),


--> """Force gossip topology changes (disable raft topology)""" ?

yarongilor · 2024-06-10T15:43:12Z

upgrade_test.py

+            scylla_yaml_updates.update({"enable_tablets": True})
+
+        if self.params.get("enable_force_`gossip_topology_changes"):
+            scylla_yaml_updates.update({"force_gossip_topology_changes": True, "enable_tablets": False})


enable_tablets is "false" by default of binary.

github-actions bot assigned aleksbykov Mar 25, 2024

aleksbykov force-pushed the raft-topology-exp-feature-upgrade branch 2 times, most recently from eec8251 to f80b512 Compare March 27, 2024 07:54

aleksbykov mentioned this pull request Mar 29, 2024

Schema mismatch after rollback during rolling upgrade from 5.4 to 5.5/6.0 (SCT, dtest) scylladb/scylladb#18098

Closed

aleksbykov force-pushed the raft-topology-exp-feature-upgrade branch from f80b512 to e8d533a Compare April 15, 2024 09:14

aleksbykov requested review from soyacz, fruch and temichus April 15, 2024 09:16

aleksbykov added the backport/none Backport is not required label Apr 15, 2024

aleksbykov marked this pull request as ready for review April 15, 2024 09:19

soyacz reviewed Apr 15, 2024

View reviewed changes

temichus assigned kbr-scylla Apr 24, 2024

aleksbykov force-pushed the raft-topology-exp-feature-upgrade branch from e8d533a to ad052de Compare April 26, 2024 12:49

kbr-scylla removed their assignment May 16, 2024

aleksbykov force-pushed the raft-topology-exp-feature-upgrade branch 5 times, most recently from 90b5607 to 8b75680 Compare June 9, 2024 09:37

aleksbykov force-pushed the raft-topology-exp-feature-upgrade branch from 8b75680 to e5f6f03 Compare June 9, 2024 09:51

aleksbykov added backport/6.0 Ready for review and removed backport/none Backport is not required labels Jun 9, 2024

temichus approved these changes Jun 9, 2024

View reviewed changes

temichus requested a review from soyacz June 9, 2024 13:53

aleksbykov requested review from yarongilor and roydahan June 9, 2024 15:39

soyacz reviewed Jun 10, 2024

View reviewed changes

yarongilor reviewed Jun 10, 2024

View reviewed changes

yarongilor mentioned this pull request Jun 16, 2024

fix(tablets): remove tablets from experimental features #7512

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(upgrade): upgrade with raft topology procedure #7295

fix(upgrade): upgrade with raft topology procedure #7295

aleksbykov commented Mar 25, 2024 •

edited

temichus commented Apr 24, 2024

kbr-scylla commented May 16, 2024

mykaul commented May 27, 2024

kbr-scylla commented May 27, 2024

soyacz left a comment

soyacz Jun 10, 2024

soyacz Jun 10, 2024

soyacz Jun 10, 2024

yarongilor Jun 10, 2024

yarongilor Jun 10, 2024

yarongilor Jun 10, 2024

fix(upgrade): upgrade with raft topology procedure #7295

Are you sure you want to change the base?

fix(upgrade): upgrade with raft topology procedure #7295

Conversation

aleksbykov commented Mar 25, 2024 • edited

Testing

PR pre-checks (self review)

Reminders

temichus commented Apr 24, 2024

kbr-scylla commented May 16, 2024

mykaul commented May 27, 2024

kbr-scylla commented May 27, 2024

soyacz left a comment

Choose a reason for hiding this comment

soyacz Jun 10, 2024

Choose a reason for hiding this comment

soyacz Jun 10, 2024

Choose a reason for hiding this comment

soyacz Jun 10, 2024

Choose a reason for hiding this comment

yarongilor Jun 10, 2024

Choose a reason for hiding this comment

yarongilor Jun 10, 2024

Choose a reason for hiding this comment

yarongilor Jun 10, 2024

Choose a reason for hiding this comment

aleksbykov commented Mar 25, 2024 •

edited