[Backport 5.2] Raft snapshot fixes

[kbr-scylla]

Backports required to fix #16683 in 5.2:

• when creating first group 0 server, create a snapshot with non-empty ID, and start it at index 1 instead of 0 to force snapshot transfer to servers that join group 0
• add an API to trigger Raft snapshot
• use the API when we restart and see that the existing snapshot is at index 0, to trigger a new one --- in order to fix broken deployments that already bootstrapped with index-0 snapshot.

[scylladb-promoter]

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-5.2/job/scylla-ci/91/

[gleb-cloudius]

Do we need to backport this if it is not used?

[kbr-scylla]

No, I'll remove

[kbr-scylla]

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-5.2/job/scylla-ci/91/

Damn. This is #12972, and this backport will cause more snapshot transfers, triggering the issue more often. Looks like we're not ignoring the error in 5.2 dtests :( (we only do it in SCT).

[mykaul]

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-5.2/job/scylla-ci/91/

Damn. This is #12972, and this backport will cause more snapshot transfers, triggering the issue more often. Looks like we're not ignoring the error in 5.2 dtests :( (we only do it in SCT).

But from that perspective, it's an artificial failure - it doesn't really mean the functionality is broken.
Or - do we need to backport some dtest fix to 5.2?

[kbr-scylla]

But from that perspective, it's an artificial failure - it doesn't really mean the functionality is broken.

The problem is that Scylla is closing connections after learning topology and this may happen in the middle of snapshot transfer. This is not a problem for Raft which retries the snapshot transfer indefinitely until it succeeds, so everything works. But there is an ERROR in the logs due to that single failed transfer. dtest infrastructure considers any ERRORs failures unless they are explicitly ignored by the test.

There was a fix that put some order into bootstrap so that the connection closing happens before we start snapshot transfers: #14354

But that fix was pretty complex and depended on other previous refactors, so we didn't backport it.

We have two options here:

• a change to dtest in 5.2 to ignore this pattern in the logs (in all dtests)
• change to Scylla to make it a WARN instead of ERROR

Actually I think I'll do the latter in this PR, the process will be smoother this way.

[kbr-scylla]

I prepared this workaround:

diff --git a/raft/server.cc b/raft/server.cc
index 48e15cf948..39a850c2c0 100644
--- a/raft/server.cc
+++ b/raft/server.cc
@@ -1064,6 +1064,18 @@ future<> server_impl::io_fiber(index_t last_stable) {
     co_return;
 }
 
+static bool is_closed_error(std::exception_ptr ep) {
+    try {
+        std::rethrow_exception(ep);
+    } catch (const seastar::rpc::remote_verb_error& e) {
+        return std::string_view{e.what()} == "connection is closed";
+    } catch (const seastar::rpc::closed_error&) {
+        return true;
+    } catch (...) {
+        return false;
+    }
+}
+
 void server_impl::send_snapshot(server_id dst, install_snapshot&& snp) {
     seastar::abort_source as;
     uint64_t id = _next_snapshot_transfer_id++;
@@ -1079,7 +1091,11 @@ void server_impl::send_snapshot(server_id dst, install_snapshot&& snp) {
             _snapshot_transfers.erase(dst);
             auto reply = raft::snapshot_reply{.current_term = _fsm->get_current_term(), .success = false};
             if (f.failed()) {
-                logger.error("[{}] Transferring snapshot to {} failed with: {}", _id, dst, f.get_exception());
+                auto ep = f.get_exception();
+                // Report our or remote's closed_error as WARNs instead of ERRORs.
+                // Workaround for scylladb/scylladb#12972 for ScyllaDB 5.2.
+                auto level = is_closed_error(ep) ? log_level::warn : log_level::error;
+                logger.log(level, "[{}] Transferring snapshot to {} failed with: {}", _id, dst, ep);
             } else {
                 logger.trace("[{}] Transferred snapshot to {}", _id, dst);
                 reply = f.get();

and tested manually that it works.

I'll also need to include #17106 in the backport so we need to get that into master first.

[kbr-scylla]

new version:

• included fixes for new test flakiness
• included 1640f83
• included workaround for raft - remote_verb_error (connection is closed) when transferring snapshot #12972 to make dtests pass
• in main.cc, register the API only if raft_group_registry is started (on master it's started unconditionally)
• removed unused error injection

[scylladb-promoter]

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-5.2/job/scylla-ci/92/

[scylladb-promoter]

CI state SUCCESS - https://jenkins.scylladb.com/job/scylla-5.2/job/scylla-ci/93/