SOLR-18301 overseer election does not converge#4625
Draft
kotman12 wants to merge 5 commits into
Draft
Conversation
Curator's RECONNECTED event is different from the previous RECONNECTED event. Before Solr10, the OnReconnect is only triggered after a reconnection from a session expiration. Check the following https://github.com/apache/solr/blob/fdb5314279657f7895a90123436d834e81ea3157/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L165 https://github.com/apache/solr/blob/fdb5314279657f7895a90123436d834e81ea3157/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L199 But Curator's RECONNECTED event is triggered every time a Solr node is disconnected from a ZooKeeper instance and reconnected to another ZooKeeper instance. Therefore, currently ZkController.onReconnect is invoked every time a Solr node reconnects to a ZooKeeper instance without a session expiration, which is a huge overhead, especially when we need to rolling-restart a ZooKeeper Cluster. It can take more than 10 minutes for a small Solr cluster to level out. Similiarly, now ZkController.onDisconnect is triggered just after a disconnection from a Zookeeper instance. It should only be triggered after a session expiration.
kotman12
commented
Jul 8, 2026
| * session possible and to give fine timing granularity around the boundary. | ||
| */ | ||
| @Test | ||
| public void testOverseerWedgesOnExpiryRacingReconnect() throws Exception { |
Contributor
Author
There was a problem hiding this comment.
I've added this check to show that this bug pre-dates the curator migration although it would be much harder to hit. Basically you'd need a strong coincidence of a reconnect at the same time as the session expiry. However, if you do a leader re-election on a small network blip that can easily trigger both stop and start in a very tight time window.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reproducing the stuck-overseer bug caused by close vs start data race. Previously overseer would enter leader election only on session expiry. By triggering election on every reconnect we uncovered certain latent bugs that were relatively rare to hit before but are now a lot more common.
Solution
Returning to previous behavior may be a good place to start, see #4577. I think it is still possible to hit this condition even if you run leader election on session expiry with a really ill-timed reconnect that happens exactly on session boundary. I will try to see what we can do here keeping in mind the uncertainty around the future of overseer.
Tests
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.