Rework of bootstrap process

[mrForza]

Reviewers

• Main Reviewer: @Serpentian
• Second Reviewer: @Gerold103
• Team Lead: @sergepetrenko
• CTO: @sergos

Tickets

• Replica bootstrap / rebootstrap fails if role writes to sync space #11039

Changelog

• v1: Initial document.
• v2: Add a detailed explanation of the main solution (getting rid of final_join stage), add cons & pros and fix some minors in initial solution (Capturing vclock when limbo has no pending sync transactions).
• v3: Rework the 1st solution (changed bootstrap protocol), add a new solution (tracking stop_vclock right after sync _cluster write), change structure of document, add new reproducer of the bug.
• v4: Add a more detailed bug descriptions (master<->replica deadlock and master's inability for write load), split 1st solution into the two part for clarity, worked out the options of catch up mechanism, add "Using build_in_memory_checkpoint" solution which suggested Gerold103, get rid of a lot of code block in 1st solution for RFC's clarity.
• v5: Get rid of remaining implementation details, add 4th variant of catch up and describe all rejected alternatives, change "Restart between the checkpoint and register" section, make some minor style fixes.
• v6: Remove Harness raft 'learner' feature #6281 ticket from "Tickets", remove "master's inability for write load" bug from "Problem overview", reduce the main solution (leave only Part 1 which fixes master&replica deadlock), add more pros & cons for every solution, make some style fixex.

Problem overview

Master & Replica deadlock during bootstrap

Bug description:

• A master leads the synchronous limbo with quorum Q.
• A sync txn is prepared on the master; its data lands in WAL, but its CONFIRM is written a moment later (because some other replicas have not enough time to confirm it).
• A new replica starts bootstrapping right in that window.
• The master takes the snapshot, registers the replica, then captures stop_vclock while the CONFIRM is still unpersisted.
• During the FINAL_JOIN stage, the master sends the sync txn's data (≤ stop_vclock) but not its CONFIRM (> stop_vclock).
• The replica applies the sync txn → it sits in its limbo waiting for the CONFIRM.
• Bootstrap then runs the initial checkpoint, which waits for that sync txn to commit.
• The CONFIRM can only arrive via SUBSCRIBE, which starts only after the checkpoint - circular wait.
• The replica deadlocks until replication_synchro_timeout, then bootstrap fails.

The places in the code where the bug is reproduced:

Replica's side:

```
bootstrap_from_master
└── gc_checkpoint
    └── gc_do_checkpoint
        └── box_checkpoint_build_on_disk
            └── txn_checkpoint_build
                ├── txn_limbo_last_synchro_entry
                ├── txn_on_commit / txn_on_rollback
                ├── txn_journal_flush
                └── fiber_yield_deadline [LOOP]     ← wait until all sync txns in limbo will be confirmed or rollbacked
```

Master's side:

```
box_process_join
└── relay_final_join
    └── relay_final_join_f
        └── recover_remaining_wals
            └── recover_xlog
                └── if (signature >= stop_vclock->signature) ← filters out rows with signature ≥ stop_vclock, so CONFIRM is never sent
                    return;
```

Solutions

1. Changing the bootstrap protocol (getting rid of FINAL_JOIN stage)

In Tarantool there are 3 main ways how the bootstrap process can proceed:

1. Anonymous bootstrap

   
2. AnonToNamed bootstrap

   
3. Named bootstrap

   

The solution reorders the bootstrap so that registration (which increases the master's sync quorum) happens only after the replica is already an active, catching-up subscriber that can send ACKs. Concretely, a named replica no longer bootstraps via JOIN from a new master; instead it fetches the master's snapshot, builds its initial checkpoint, subscribes and catches up anonymously, and only then sends REGISTER.

1. Fix of Master & Replica deadlock during bootstrap Replica bootstrap / rebootstrap fails if role writes to sync space #11039

   We replace the JOIN-based bootstrap of a named replica with the sequence that anonymous replicas already use, plus a deferred registration: FETCH_SNAPSHOT → initial checkpoint → REGISTER → SUBSCRIBE. The key idea is a small reordering of bootstrap process — the replica's initial checkpoint is created before any WAL stream that can carry unconfirmed synchronous transactions:

   1. FETCH_SNAPSHOT instead of JOIN. The replica receives a consistent snapshot (with limbo/raft state in JOIN_META) via box_process_fetch_snapshot, which never sends unconfirmed rows. The replica recovers from it and creates its own initial checkpoint while its limbo is guaranteed clean, so nothing can be placed into the limbo during gc_checkpoint.

   2. REGISTER after the checkpoint. Once bootstrap has its checkpoint, the applier sends REGISTER and receives the "final-join" WAL stream via box_process_register. If this stream contains an unconfirmed sync transaction, it is now harmless: it just sits in the replica's limbo as a pending entry — the same supported steady state as any running replica receiving an in-flight sync transaction.

   3. SUBSCRIBE is unchanged from the old flow. The only difference is what it may find on arrival: if the REGISTER stage delivered synchronous transactions whose CONFIRM entries were not yet written on the master (they lay after stop_vclock), those transactions are now pending in the replica's limbo. The SUBSCRIBE resolves them: the replica ACKs the master through it, the master's pending sync transactions gather quorum and get confirmed, and the CONFIRM entries flow back to the replica through the relay, draining its limbo — just like for any running replica that received an in-flight sync transaction during normal replication.

   This part requires only replica's side changes, because our current master can serve any types of replication IPROTO requests in any way. On the replica's side we need to:

   1. Make applier_f always call applier_fetch_snapshot regardless of anon/named status of replica.
   2. Change the waiting of applier's statuses in box_bootstrap_from_master (from FINAL_JOIN/FETCHED_SNAPSHOT to APPLIER_FETCHED_SNAPSHOT).
   3. Remove applier_join entirely — named replicas no longer bootstrap via JOIN.
   4. Remove all unused applier's statuses — APPLIER_FINAL_JOIN and APPLIER_JOINED.

Pros & Cons:

• ❎ An architecturally more correct solution
• ❎ This solution needs only replica's side changes (no master)
• ❎ A lightweight solution (~50 lines of code)
• 📛 Many tests may stop working due to the deletion of the final_join
• 📛 All documentation notes should be updated due to deletion of final_join

Rejected alternatives

2. Using build_in_memory_checkpoint on the master side

The core idea of this solution: stop_vclock must not be a raw copy of instance_vclock, but a journal position V for which the master can prove that everything up to V is committed. That is exactly what box_checkpoint_build_in_memory produces: a pair of "journal vclock V" + "limbo state (confirmed_lsn, queue owner, term) captured at the exact moment the last pending sync txn of the build time got committed". The master uses the checkpoint's vclock as stop_vclock, streams (start_vclock, stop_vclock] as today, and then sends the checkpoint's limbo state as a "fake" PROMOTE meta row - the same mechanism send_join_meta already uses in the initial join. The replica applies it via txn_limbo_process and instantly commits all sync txns <= stop_vclock, so its initial checkpoint never waits for a CONFIRM that can only arrive via SUBSCRIBE.

The key insight (why this beats both the drain and the on-commit-trigger capture): we do not try to find a point in the xlog where every CONFIRM covering the data is also below it - Serpentian showed this is impossible without draining (T1 → T2 → CONFIRM(T1) → T3 → CONFIRM(T2) → … chases forever). Instead, we ship the limbo state next to the data, exactly like a snapshot does: snapshot recovery never replays CONFIRMs of the rows stored in it, it restores confirmed_lsn from the limbo state saved in the snapshot.

Pros & Cons:

• ❎ Preserves existing bootstrap protocol order.
• ❎ Zero limbo changes: no is_frozen, no drain, no new flags or states.
• 📛 Replica-side change + version/feature gate needed.
• 📛 Doesn't solve the underlying architectural issue.
• 📛 Testing matrix expands significantly (old/new master × old/new replica combinations).

3. Drain limbo during master freeze

The core idea of this solution: during bootstrap process the master should capture stop_vclock only when the limbo is "completed" (every sync txn with data in WAL also has its terminal record in WAL - CONFIRM or ROLLBACK).

High-level algorithm:

1. Block new sync-txn submissions until all sync entries are resolved.
2. Ensure that all existing entries in limbo are materialized in WAL (i.e. has a valid lsn).
3. Wait for the last sync txn to be resolved (COMMIT or ROLLBACK); loop until the limbo is empty of submitted sync txns.
4. If some sync txns can't be resolved - fail bootstrap process and start over.
5. Capture stop_vclock and then replay all final_join rows in the interval (start_vclock; stop_vclock] to newly added replica.

Implementation details:

1. Introduce ER_SYNC_LIMBO_DRAIN_TIMEOUT - an error which sends from master to replica if some pending sync txns don't have time to be resolved during bootstrap process in replicaset.

2. Introduce two new fields in struct limbo - bool is_frozen (when true, new sync-txn submissions must wait) and struct fiber_cond frozen_cond (signaled on unfreeze).

3. Define a new limbo function which captures vclock only when all sync txns entries are confirmed or rollbacked. It is important to mention that if some sync txn don't have time to be resolved we raise a new error to new replica.

   txn_limbo.cc

   int
   txn_limbo_snapshot_vclock_after_drain(struct txn_limbo *limbo,
   			      				      struct vclock *vclock)
   {
       /*
        * Only the leader persists CONFIRMs/ROLLBACKs and drives the queue.
        * A non-leader has nothing to drain.
        */
       if (limbo->state != TXN_LIMBO_STATE_LEADER) {
   	    vclock_copy(vclock, instance_vclock);
   	    return 0;
       }
       /*
        * Freeze new submissions so that the drained set is finite and no
        * new sync rows enter the WAL range being captured. Note, that
        * txn_limbo_flush() bypasses the freeze gate.
        */
       limbo->is_frozen = true;
       int rc = -1;
       if (txn_limbo_flush(limbo) != 0)
   	    goto unfreeze;
       /*
        * Wait for all the pending sync txns to reach a terminal state, so
        * that their CONFIRMs and ROLLBACKs appear within the captured
        * range.
        */
       double deadline = fiber_clock() + replication_synchro_timeout * 2;
       while (txn_limbo_last_synchro_entry(limbo) != NULL) {
   	    bool is_rollback;
   	    if (fiber_is_cancelled()) {
   		    diag_set(FiberIsCancelled);
   		    goto unfreeze;
   	    }
   	    if (txn_limbo_wait_last_txn(limbo, &is_rollback,
   				        deadline - fiber_clock()) != 0)
   		    break;
       }
       if (txn_limbo_last_synchro_entry(limbo) == NULL) {
   	    vclock_copy(vclock, instance_vclock);
   	    rc = 0;
       } else {
   	    diag_set(ClientError, ER_SYNC_LIMBO_DRAIN_TIMEOUT);
       }
   unfreeze:
       limbo->is_frozen = false;
       fiber_cond_broadcast(&limbo->frozen_cond);
       return rc;
   }

4. In txn_limbo_submit add a waiting loop in order to not allow sync txns appear on master:

   txn_limbo.cc

   int
   txn_limbo_submit(struct txn_limbo *limbo, uint32_t id, struct txn *txn,
           size_t approx_len)
   {
       // Only after all sync txns are resolved (during bootstrap) we can accept new sync txns.
       while (txn_limbo->is_frozen && !fiber_is_cancelled())
           fiber_cond_wait(&txn_limbo->frozen_cond);
       ...
       return txn_limbo_queue_submit(&limbo->queue, id == 0 ? instance_id : id,
                       txn, approx_len);
   }

5. In box_process_register and box_process_join insert a txn_limbo_capture_vclock_after_all_confirms waiting function

   box.cc

   void
   box_process_register(struct iostream *io, const struct xrow_header *header)
   {
       ...
       box_register_replica(&req.instance_uuid, req.instance_name);
       ...
       /* Remember master's vclock after the last request */
       struct vclock stop_vclock;
   if (txn_limbo_snapshot_vclock_after_drain(&txn_limbo, &stop_vclock) != 0)
   	diag_raise();
       ...
       relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
       say_info("final data sent.");
   }

   void
   box_process_join(struct iostream *io, const struct xrow_header *header)
   {
       ...
       struct vclock start_vclock;
       relay_initial_join(io, header->sync, &start_vclock, req.version_id,
               NULL);
       ...
       /* Remember master's vclock after the last request */
       struct vclock stop_vclock;
   if (txn_limbo_snapshot_vclock_after_drain(&txn_limbo, &stop_vclock) != 0)
   	diag_raise();
       ...
       relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
       say_info("final data sent.");
   }

Pros & Cons:

• ❎ No iproto protocol changes
• ❎ A draft patch is ready and all replication tests are passed
• 📛 We freeze master until all pending transactions are completed (confirm or rollback)
• 📛 There are can be problems with deferred quorum recalculation
• 📛 If master can't resolve all pending sync txns in limbo, the bootstrap process fails

[sergepetrenko]

Thanks for the design!

There's one extra issue with synchronous join you haven't mentioned: when an entry with replica's ID is committed on master, master re-calculates its replication_synchro_quorum.
So the following is possible:

1. Master works as usual, its quorum is Q and it has enough live replicas to commit transactions with this quorum
2. Master processes a JOIN request: assigns an ID to a new replica, which in turn results in quorum recalculation. Quorum is now Q + 1 (note, the ID assignment itself is processed with the old quorum Q)
3. It happens so that there are not enough live replicas for the master to commit anything with quorum Q + 1.
4. The "no quorum" situation persists until the replica's final join step is complete, because, as we know, there are no heartbeats during the final join stage, so the new replica cannot ack the incoming transactions anyhow.

That's a serious problem. On large installations with high write load it's quite easy to achive situations where the final join stage will run for minutes. And for that whole time the cluster will be unable to serve writes given an unlucky enough replica topology.

For example:
There's a cluster with 3 nodes, quorum = 2, one of the nodes is down. As soon as the fourth node is joined, quorum = 3, but its unachievable unless the new replica finishes join.
Or: there's a single node, so quorum = 1. As soon as the new replica is joined, quorum becomes 2. Same problem as above.

The alternative proposal (with bootstrap protocol rework) handles this issue in the following way:

1. Only send the registration request as soon as the replica sees it has a small enough lag with the master, thus limiting the potential downtime
2. The registration flow allows heartbeats, so replica starts acking fresh master's transactions as soon as it receives its own registration.

Speaking of your proposal, I like that it turned out so concise and easy to implement, but I don't see how this issue can be handled there. Maybe you have any ideas?

[mrForza]

Now, the quorum recalculation (replication_synchro_quorum) is deferred until the join is over. A freeze counter (box_quorum_freeze_count) and a pending flag (box_quorum_update_pending) are introduced in box.cc. While the freeze is active, box_update_replication_synchro_quorum does not recalculate the quorum and only marks the update as pending. box_process_join and box_process_register freeze the recalculation right before box_register_replica and release it via a scoped guard at the end of the handler, i.e. after the final join data has been sent. On the last release the deferred recalculation is executed, so the quorum grows only when the new replica is about to subscribe and ACK

High-level code of box.cc

static int box_quorum_freeze_count;

static bool box_quorum_update_pending;

static void
box_quorum_freeze(void)
{
	box_quorum_freeze_count++;
}

static void
box_quorum_unfreeze(void)
{
	assert(box_quorum_freeze_count > 0);
	box_quorum_freeze_count--;
	if (box_quorum_freeze_count == 0 && box_quorum_update_pending) {
		box_quorum_update_pending = false;
		box_update_replication_synchro_quorum();
	}
}

void
box_update_replication_synchro_quorum(void)
{
	if (box_quorum_freeze_count > 0) {
		box_quorum_update_pending = true;
		return;
	}
	...
}

void
box_process_join(struct iostream *io, const struct xrow_header *header)
{
	...
	relay_initial_join(io, header->sync, &start_vclock, req.version_id, NULL);
	say_info("initial data sent.");

	box_quorum_freeze();
	auto quorum_guard = make_scoped_guard(box_quorum_unfreeze);

    ...
	box_register_replica(&req.instance_uuid, req.instance_name);
    ...
    relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
	say_info("final data sent.");
    ...
}

void
box_process_register(struct iostream *io, const struct xrow_header *header)
{
    ...
	say_info("registering replica %s at %s", ...);

	box_quorum_freeze();
	auto quorum_guard = make_scoped_guard(box_quorum_unfreeze);

	box_register_replica(&req.instance_uuid, req.instance_name);
    ...
	relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
	say_info("final data sent.");
    ...
}

After this mini-patch all replication tests are passed!

[Gerold103]

About the problem of the quorum being not achievable during join - I like the Sergey's proposal to delay the registration until the replica is caught up. This looks elegant and simple. Adding new "frozen" states only increases the complexity. We had such "frozen" and "fenced" states already in the limbo, and their removal only made things simpler.

Regarding the problem of CONFIRM mentioned in the RFC itself - I still don't see why box_checkpoint_build_on_disk() doesn't help 🤔.

[mrForza]

I still don't see why box_checkpoint_build_on_disk() doesn't help 🤔.

Your suggestion is in progress) I will reply soon.

[Serpentian]

Good RFC, we still have several major comments to discuss, but in general it's already in very good shape.

---

applier_subscribe gained a stop_on_follow parameter that makes the loop return as soon as the applier transitions into the APPLIER_FOLLOW state
add a waiting of APPLIER_REGISTERED then APPLIER_READY for named replicas in the end of this function.

In the scope of the RFC review I won't review the code descriptions you have there, I suppose it should be done in the PR when it's ready. Basically, in that specific case I'm ok with anon bounded subscribe, but I'm not sure introducing the flag stop_on_follow is the best way to implement that and propose to decide it later. And speaking of waiting for REGISTERED, I'm not sure this will work, this state is skipped, if replica is already registered.

So, if I were you I'd avoid using such specific wording of solutions in code, we're not discussing the implementation, only ideas. But up to you.

During this phase the replica is an anonymous subscriber: it receives the master's rows (including the confirm/rollback of synchronous transactions) and is therefore able to send ACKs, which lets the master drain its pending sync queue.
Because the replica is already caught up and ACK-capable, registering it no longer risks stalling the master.

I find this part of description a little bit confusing and just wanna make sure, we understand that: Replica is always "able to send ACKSs", there's only two questions:

1. Which ACKs does it send after the quorum is increased (registration happens) - the transactions, which were done 10 minutes ago or 0.1 second ago. We're basically trying to minimize the tail in the box_process_register -> relay_final_join, so that we send as small amount of data there as possible in order to start ACKing the non confirmed transactions already (and not some old confirmed ones).
2. Whether master cares about our ACKs: master ignores ACKs from anon replicas, which is ok, but the first sentense is not correct
3. Whether we send these ACKs.

Speaking of the last one: can we consider sending ACKs as early, as possible? Now we start the ACK machinery on replica only in SUBSCRIBE, but we can do that even earlier: in the register stage. But only because of the changes, proposed in the current RFC. There is an important durability condition: ACKs must represent persisted data. The proposed flow creates the initial checkpoint and enables WAL before registration, which supports this. Legacy JOIN applies its final stream through a recovery journal that performs no actual writes, so we cannot give that path the same quorum (see the recovery_journal_write, now we just accumulate data in memory and then write the snap with this data).

Catch up mechanism's implementation. ii. Bounded wait + error raising.
the whole long flow again — fetch_snapshot → initial checkpoint creation → anon subscribe → catch-up

If only it was just that flow) The problem here is that recovery happens a lot earlier than we subscribe: the connections are created, we check ballots, then appliers are on pause and we recover snaps/xlogs (which can take hours) and only after that appliers are resumed. So, we'll just exit with error after user waited for 5 hours for tarantool to start. I agree, this is not an option for sure.

Catch up mechanism's implementation. iii. Bounded wait + background completion.

And this solution looks like the breaking change to me: it changes more than box.info reporting: startup currently requires a registered identity, and its subsequent Raft initialization calls a function which requires a nonzero ID. Applications may also use the ID immediately after box.cfg() returns. And since the replication_sync_timeout is 0 by default, almost any replica startup will end up in that new anon state. So, we either should introduce the compat and make sure user knows, that from now on it's possible that replica can return with anon state from box.cfg, without any id, or we shoulldn't introduce the breaking cahnge at all.

And in order not to introduce the breaking change we should use the solution, Vladislav proposed: register after bounded catch-up wait. But using the replication_sync_timeout will leave the absolute majority of users in the same state as before the patch: possible inavailability on register of new replica, which I don't like. And I'm not sure it's good idea to use the replication_sync_lag/timeout in two places. Maybe we should consider introducing ad new independent catch-up lag, whiich could inherit replication_sync_lag when unspecified (we already have the replication_reconnect_timeout working like that with replication_timeout). And separate non-zero catchup timeout for it.

Restart between the checkpoint and register stage and box.cfg

It seeems I'm ok with automatic register if no replication_anon is set, but we then should also describe the case, when master has commited registration, but replica didn't receive it and crashed. The now proposed anon SUBSCRIBE will fail. And the 4th part also doesn't decscribe the ER_INSTANCE_NAME_MISMATCH error (compare empty name with configured one).

---

---

@Gerold103, please, resolve the comments, if they're not useful anymore, way too many text in the comments

[mrForza]

I'm not sure this will work, this state is skipped, if replica is already registered.

Right, applier_register() is skipped when the own _cluster row comes with the snapshot. Since this is again an implementation details, I will remove this from my RFC, but I will take it into account when I start writing PR (the same is for stop_on_follow parameter).

So, if I were you I'd avoid using such specific wording of solutions in code, we're not discussing the implementation, only ideas. But up to you.

Yes, agree. I'm still learning how to write short and meaningful RFCs without implementation details.

Can we consider sending ACKs as early, as possible? ... in the register stage.

I think we need master's side changes: relay_final_join has no relay_reader_f, so the master wouldn't read those ACKs. Since the register tail is small in the new flow, I'd skip it.

And this solution looks like the breaking change to me.

Sadness, I'll write it down in rejected alternatives of "catch up mechanism". After our Monday's discussion I will stop at the first option (infinite wait).

we then should also describe the case, when master has commited registration, but replica didn't receive it and crashed

It will be done!

[mrForza]

Part 1, the 11039 fix

Thanks for the explanation, I'll keep that in mind when I do PR. In the meantime, I will remove all implementation details from my RFC.

Part 2, 6281 fix

As for anon acks, agree, my bad. As for acks in register stage I also think it will hard to implement because it requires master's side changes (we don't want to do it).

Part 4, the restart in the window

OK, I'll make these changes in the last part of my RFC.

Raft learner

I didn't quite understand why we should move away from the original decision towards Learners. What is the motivation?

[Gerold103]

I will tick the box. But the document still seems not complete. The comments about part 1 and related points from part 2 doesn't look all addressed.

Below is my summary, and in the end of the message is a more detailed analysis from Fable.

---

Meat

1. The changelist "On the replica's side we need to:" has 4 points, which are missing the REGISTER itself. You also can delete APPLIER_JOINED, I think. But this is an implementation detail already.

2. The restart of the new replica after getting the snapshot and its registration in _cluster, but before it receives the _cluster row, is still going to hurt. box.cfg{} on such restart will panic.

3. Since we fetch the snapshot while being anon, we will have to fetch it in replication_anon_ttl time. Otherwise, once we are done, but before we send REGISTER, the master's GC might move on, delete xlogs, and on SUBSCRIBE the replica will realize that it can't subscribe - the needed xlogs are gone. At least this means users would need to raise replication_anon_ttl at or above their max expected fetch-snapshot time, to be on the safe side. If they don't, then it might still work if lucky (the relay_initial_join() connection, while alive, will hold GC. The problem is the window between the snapshot being fetched and the REGISTER being sent.)

---

[AI]

v6 is the split from my previous comment, part 1 alone as the #11039 fix, good. Below is part 1 reviewed as a standalone change. The text is right where it matters: the snapshot comes from an in-memory checkpoint, which waits for the last pending synchronous transaction to be resolved before the read view is frozen, so the replica's limbo is empty when it writes its own checkpoint. And the _cluster row is written on the master behind whatever was pending at that moment, so what arrives unconfirmed in the REGISTER stream was written after the row, which is the steady state of any replica. Four things are missing from the text.

The register step is not in the change list. Items 1 to 4 fetch the snapshot and remove JOIN, but nothing sends REGISTER. After the snapshot the applier pauses in READY, bootstrap_from_master() writes the checkpoint and returns, and the next thing box_cfg_xc() does about the registration is the check which panics with "'replication_anon' cfg didn't work". The missing piece is the pair of waits which box_set_replication_anon() does today for the runtime anon-to-named switch: after the checkpoint, resume the applier to REGISTERED, then to READY. The applier sends REGISTER on its own then, need_id is true for it, on the connection it fetched the snapshot from, the way SUBSCRIBE reuses the JOIN connection today. Nothing else in the applier changes. Item 2 should also drop the APPLIER_JOINED wait, not only change the snapshot one.

The restart in the window did not leave with part 2. The window is between the initial checkpoint and the arrival of the own _cluster row, and part 1 has it exactly as part 2 did, only without the catch-up inside. It is not short either: on the master the row waits in the limbo behind the pending synchronous transactions like any other write, and then the stream delivers everything written since the snapshot, which is as much as the checkpoint took. A crash in there leaves a checkpoint, a named configuration and no own row, and box_cfg_xc() has one reaction to that state today, the panic above. Both cases of the former point 4 are still there: the master has not registered us (the same two waits once more, this time from box_cfg_xc() after the local recovery), and the master has committed the row but we crashed before receiving it (REGISTER is refused with ER_REPLICA_NOT_ANON, so the replica subscribes as named and waits for its own row, detecting the case through the ballot). So point 4 belongs to part 1, minus the catch-up; the code block in my round-4 comment is that, minus its two catch-up lines.

The late registration failure from my previous comment applies to part 1 unchanged, the price being the snapshot transfer instead of hours. The two asks stay: a failed registration keeps the transfer, which is the restart-in-window path above, and configuration errors from REGISTER fail box.cfg{} instead of retrying in LOADING forever.

One con to add. Until REGISTER the master knows the joining replica as an anonymous one, and keeps its xlogs the anonymous way: replication_anon_ttl, one hour by default, counted on the master from the moment the replica object was created, which is the start of the snapshot transfer, and not refreshed by the transfer itself. A bootstrap whose transfer plus initial checkpoint takes longer than that gets its WAL consumer dropped, and if the master collects the xlogs after that, REGISTER fails with a gap error which the applier retries forever. A JOIN today registers before the final stream and is never exposed to this. It deserves a line in the cons, with the TTL as the knob.

Smaller things. Changelog: "fixex". The pros say "only replica's side changes"; true, and worth keeping true through the points above, they all are.