[RECOVERY] Make recovery delay configurable #8739
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
s1monw
added
>test
| @@ -303,10 +303,12 @@ public InternalTestCluster(long clusterSeed, | |||
| builder.put(RecoverySettings.INDICES_RECOVERY_CONCURRENT_STREAMS, RandomInts.randomIntBetween(random, 10, 15)); | |||
| builder.put(RecoverySettings.INDICES_RECOVERY_CONCURRENT_SMALL_FILE_STREAMS, RandomInts.randomIntBetween(random, 10, 15)); | |||
| builder.put(ThrottlingAllocationDecider.CLUSTER_ROUTING_ALLOCATION_NODE_CONCURRENT_RECOVERIES, RandomInts.randomIntBetween(random, 5, 10)); | |||
| builder.put(RecoverySettings.INDICES_RECOVERY_RETRY_DELAY, TimeValue.timeValueMillis(RandomInts.randomIntBetween(random, 10, 50))); // more shareds - we need to retry more often | |||
|
LGTM - note that I had added a similar settings in https://github.com/elasticsearch/elasticsearch/pull/8720/files#diff-d1e7cc297312b15bab61e9467d3d0cd7R78 , but i needed more control over different reasons to retry. I like the fact that is' in RecoverySettings now and dynamically updatable. I'll update my PR when rebasing. |
Today we wait 500ms before we retry a recovery if the target node is not ready. This happens if the source starts the recovery before the target has processed the clusterstate moving the target shard into the right state. This can cause a 500ms delay each time it happens while the shard is ready way earlier on the target node. This commit makes this delay configurable to mainly speed up test processing and shard allocation in tests.
clintongormley
added
:Distributed/Recovery
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today we wait 500ms before we retry a recovery if the target node is not ready.
This happens if the source starts the recovery before the target has
processed the clusterstate moving the target shard into the right state.
This can cause a 500ms delay each time it happens while the shard is ready
way earlier on the target node. This commit makes this delay configurable
to mainly speed up test processing and shard allocation in tests.