Fix topology replay during bootstrap and startup, decouple Accord from TCM#3781
Closed
ifesdjeen wants to merge 28 commits intoapache:cep-15-accordfrom
ifesdjeen:CASSANDRA-20142
Closed
Fix topology replay during bootstrap and startup, decouple Accord from TCM#3781ifesdjeen wants to merge 28 commits intoapache:cep-15-accordfrom ifesdjeen:CASSANDRA-20142
ifesdjeen wants to merge 28 commits intoapache:cep-15-accordfrom
ifesdjeen:CASSANDRA-20142
Conversation
…m TCM
Includes multiple changes, primary ones:
* Make HarrySimulatorTest work with Accord
* Removed nodes now live in TCM, no need to discover historic epochs in order to find removed nodes
* CommandStore <-> RangesForEpochs mappings required for startup are now stored in journal, and CS can be set up _without_ topology replay
* Topology replay is fully done via journal (where we store topologies themselves), and topology metadata table (where we store redundant/closed information)
* Fixed various bugs related to propagation and staleness
* TCM was previously relied on for "fetching" epoch: we can not rely on it as there's no guarantee we will see a consecutive epoch when grabbing Metadata#current
* Redundant / closed during replay was set with incorrect ranges in 1 of the code paths
* TCM was contacted multiple times for historical epochs, which made startup much longer under some circumstances
| PAXOS_REPAIR (false, "PaxosRepairStage", "internal", FBUtilities::getAvailableProcessors, null, Stage::multiThreadedStage), | ||
| INTERNAL_METADATA (false, "InternalMetadataStage", "internal", FBUtilities::getAvailableProcessors, null, Stage::multiThreadedStage), | ||
| FETCH_LOG (false, "MetadataFetchLogStage", "internal", () -> 1, null, Stage::singleThreadedStage), | ||
| FETCH_METADATA(false, "MetadataFetchLogStage", "internal", () -> 1, null, Stage::singleThreadedStage), |
| TCM_COMMIT_REQ (802, P0, rpcTimeout, INTERNAL_METADATA, MessageSerializers::commitSerializer, () -> commitRequestHandler(), TCM_COMMIT_RSP ), | ||
| TCM_FETCH_CMS_LOG_RSP (803, P0, rpcTimeout, FETCH_LOG, MessageSerializers::logStateSerializer, RESPONSE_HANDLER ), | ||
| TCM_FETCH_CMS_LOG_REQ (804, P0, rpcTimeout, FETCH_LOG, () -> FetchCMSLog.serializer, () -> fetchLogRequestHandler(), TCM_FETCH_CMS_LOG_RSP ), | ||
| TCM_FETCH_CMS_LOG_RSP (803, P0, rpcTimeout, FETCH_METADATA, MessageSerializers::logStateSerializer, RESPONSE_HANDLER ), |
| diskStateManager.loadLocalTopologyState((epoch, syncStatus, pendingSyncNotify, remoteSyncComplete, closed, redundant) -> { | ||
| getOrCreateEpochState(epoch).setSyncStatus(syncStatus); | ||
| if (syncStatus == SyncStatus.NOTIFYING) | ||
| switch (syncStatus) |
Contributor
There was a problem hiding this comment.
Either handle all cases and throw exception on unhandled, or simply test syncStatus == NOTIFYING?
Should we be doing anything in the NOT_STARTED case? We might need to fetch information from peers about what has been synced, though this will naturally start arriving from later epochs, so perhaps not a problem.
Contributor
Author
There was a problem hiding this comment.
You are right; should have only been == NOTIFYING here.
For now, we can do nothing with NOT_STARTED case. But I will add this area of code to the list of things that need audit.
| toComplete[i] = it.nextLong(); | ||
| Arrays.sort(toComplete); | ||
| for (long epoch : toComplete) | ||
| listener.onEndpointAck(removed, epoch); |
Contributor
There was a problem hiding this comment.
might be nice to move this and the below listener notifications outside of the synchronised block?
belliottsmith
approved these changes
Jan 9, 2025
Contributor
belliottsmith
left a comment
belliottsmith
left a comment
There was a problem hiding this comment.
+1 once feedback addressed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix topology replay during bootstrap and startup, decouple Accord from TCM
Includes multiple changes, primary ones: