[14.0.x] ISPN-15357 Unable to enable rebalancing after cluster scale up #11683
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Utilize the global state manager to persist the global rebalance status. This allows the rebalance disabled to tolerate complete cluster restarts and still avoid rebalancing. Otherwise, only rolling upgrades could do this.
This PR has two sides. First, by "stateful" we refer to nodes with information in the persistent state about the cache cluster before restart, and "stateless" a node without any information. Originally, a stateful cluster only accepts stateful joiners until the old cluster is complete. Stateless nodes can only join *after* that, but they have no clue about that, on what stage the recovery is. If they try joining too early, the receive an exception, which could lead to the node failing to start. With the changes, a stateless node can issue a join request at any point. It will *not* join the cluster imediatelly, as the cluster needs to recover, but now the node receive a specific response, so it can send a join request again after the recovery is complete. This is automatic now, and shouldn't need any intervention. The tricky case, starting a stateless coordinator with a stateful joiner. This is very likely a problem during initialization, or a misconfiguration during updates. In such a case, the stateful node trying to join will receive an exception, and likely fail to start. This would possible identify issues elsewhere and the user would need to manually fix it. If the user is manually adding nodes, it needs to make sure to start the new node *after* all the previous nodes are up and running. Any concurrent cluster start, or concurrent membership changes (i.e., a node shutdown while adding another one) could trigger this behavior.
|
Operator test failure seems unrelated. |
|
I've seen the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.redhat.com/browse/ISPN-15357
Backport of #11633.