Ensure all producers from other workers have registered during recovery #3057
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds two new phases to the recovery protocol to guarantee that all Producers on other workers have registered with a recovering worker before we proceed to initiate the rollback barrier. Fixes #3018.
jtfmumm
force-pushed
the
eo_bug_exploration
branch
from
1661ac2
to
741915d
Compare
Run via: `./master-crasher.sh 2 run_custom_tcp_crash0`. Note that `initializer` and `worker` give very different answers to the cluster status query. Output: ``` WARNING: all useful state files are deleted by this script! Worker initializer: port = 7103 Worker worker1: port = 7113 Success RUN: run_custom_tcp_crash0 Done, yay ... waiting ,,,c0s0,,,,, Query initializer Connected... Cluster Status: Cluster not yet initialized Processing messages: false Query worker1 Connected... Cluster Status: Processing messages: true, Worker count: 2, Workers: |initializer,worker1,| Query initializer again Connected... Cluster Status: Cluster not yet initialized Processing messages: false ```
slfritchie
suggested changes
Nov 13, 2019
There was a problem hiding this comment.
I've added commit 4bc7b87 that shows what happens when the initializer worker crashes & restarts a single time in a 2-worker cluster. (Full output logs are at http://wallaroolabs-dev.s3.amazonaws.com/logs/logs.1573679554.tar.gz)
Initializer says:
Cluster Status:
Cluster not yet initialized
Processing messages: false
Worker1 says:
Cluster Status:
Processing messages: true, Worker count: 2, Workers: |initializer,worker1,|
Initializer's restart does not progress beyond the point of Reconnect Phase 2: Wait for Reconnections.
We shouldn't register BarrierSource as a source with RouterRegistry because it is only a source for the purposes of the barrier protocol. As a result, unlike other sources, it does not use its own OutgoingBoundaries, but instead uses the canonical ones. This is because it only rarely sends messages (those related to barriers and to register/unregister itself as a Producer).
Even if we have not written any bytes to disk for a step, we might have acquired keys after the last complete checkpoint. These need to be cleared during rollback.
We currently support two reasons for rollback: (1) crash recovery and (2) checkpoint abort. When two workers concurrently initiate rollback we were only using the rollback id to settle priority. However, crash recovery should always trump abort checkpoint since the former must guarantee stronger conditions before rollback can commence (e.g. boundary reconnect and producer registration). With this commit, we use the rollback reason to help determine priority.
slfritchie
approved these changes
Nov 18, 2019
This was referenced Nov 18, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds two new phases to the recovery protocol to guarantee that
all Producers on other workers have registered with a recovering
worker before we proceed to initiate the rollback barrier.
Fixes #3018.