Wrong schema version in system.peers when using raft

[sylwiaszunejko]

I have started three node cluster with --experimental-features consistent-topology-changes --experimental-features tablets. I have used master version of Scylla (34c3688). Without any previous steps, I have connected to the empty cluster using cqlsh 127.0.0.2 and I have executed the following commands:

SELECT schema_version FROM system.local;
SELECT peer, host_id, schema_version FROM system.peers;

The output looks like that:

 schema_version
--------------------------------------
 c1f44a7f-c4ab-3022-b607-66f0b8bdd582

 peer      | host_id                              | schema_version
-----------+--------------------------------------+--------------------------------------
 127.0.0.3 | 5a4581a5-389b-4075-88d9-2dd533804f9c | 59adb24e-f3cd-3e02-97f0-5b395827453f
 127.0.0.4 | e5fa0b2c-de94-45f9-b9aa-1387b56be28b | 59adb24e-f3cd-3e02-97f0-5b395827453f

Accordingly, I have connected to 127.0.0.3 and 127.0.0.4, run the same commands, and got the outputs:

 schema_version
--------------------------------------
 c1f44a7f-c4ab-3022-b607-66f0b8bdd582

 peer      | host_id                              | schema_version
-----------+--------------------------------------+--------------------------------------
 127.0.0.2 | 749d1300-474e-46cb-8dfc-870ff058a20a | c1f44a7f-c4ab-3022-b607-66f0b8bdd582
 127.0.0.4 | e5fa0b2c-de94-45f9-b9aa-1387b56be28b | 59adb24e-f3cd-3e02-97f0-5b395827453f

 schema_version
--------------------------------------
 c1f44a7f-c4ab-3022-b607-66f0b8bdd582

 peer      | host_id                              | schema_version
-----------+--------------------------------------+--------------------------------------
 127.0.0.3 | 5a4581a5-389b-4075-88d9-2dd533804f9c | 59adb24e-f3cd-3e02-97f0-5b395827453f
 127.0.0.2 | 749d1300-474e-46cb-8dfc-870ff058a20a | c1f44a7f-c4ab-3022-b607-66f0b8bdd582

The schema versions in system.peers does not align with the schema versions in system.local.
This cause problems with for example running any command using gocql.

I have noticed that if I perform operation that changes schema (creating keyspace/table) it fixes system.peers.

I tried to reproduce this on cluster with nodes without --experimental-features consistent-topology-changes and the bug does not occurs.

I have also tested it with scylladb/scylla-nightly and the last properly working version is 5.4.0-dev-0.20230802.0239ba45272f-x86_64 (5.4.0-dev-0.20230803.39ca07c49b25-x86_64 has this bug).

I have performed a kind of bisect to find out from which commit this problem occurs, it happens to be 7c30954, the last commit without that bug is 3c1ca12.

I attach logs from all three nodes.

node3_logs.txt
node2_logs.txt
node1_logs.txt

[avelanarius]

cc @kbr-scylla @kostja

[kbr-scylla]

cc @gleb-cloudius

[gleb-cloudius]

Strange. Gossiper has different schema versions as well.

[kostja]

@kbr-scylla how do we test for this to stay in sync in the future? @gleb-cloudius can we make sure we don't update system.local through the observer on the raft command apply path?

[gleb-cloudius]

@gleb-cloudius can we make sure we don't update system.local through the observer on the raft command apply path?

We do not update system.local through the observer. We update gossiper state through it.

[kostja]

But how does the gossiper manage to get the wrong state in the first place? The node is just starting after all, it got to be taking it from the system table.

[gleb-cloudius]

The node was starting. Was receiving a schema update through raft, but not updating its local version in the gossiper.

[kbr-scylla]

@kbr-scylla how do we test for this to stay in sync in the future?

I don't know. We could compare versions from system.peers in our system tests -- similarly to how we check group 0 and token ring consistency.

But with the bug present, the tests would be flaky, sometimes they would pass because the problem didn't always reproduce (?) But maybe a flaky test is better than no test...

[gleb-cloudius]

This bug was 100% reproducible. But since peers table is updated as node learns about other nodes versions over the network a test may see different versions for some time.

[avikivity]

@gleb-cloudius please evaluate for backport

[gleb-cloudius]

All the version with non experimental schema over raft should get it.

[kbr-scylla]

Backported to 5.2