[CORE] Wait for pending shard removal on IndexService close #8608
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When an index service gets closed due to a delete index event it's possible that shards are already removed from the service that are currently in recovery. The recovery source applies the delete index first and causes the target to fail its shard. This shard gets removed from the service but it's content doesn't get deleted if the actual delete index is faster applied. This commit makes sure we wait for the pending removals that are in-flight before we close the index an delete it's content too once the lock is released.
|
@bleskes if you have time I'd love you to review this - assinging you |
|
I looked at it and my main concern is complexity. @s1monw had some ideas about simplifying it and is working on it. |
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Dec 19, 2014
elastic#8436 has introduced shard level locks in order to prevent directory reuse during fast deletion & creation of indices. As part for the change, close listeners were introduced to delete the folders once all out standing references were released. The new change has created race conditions causing shard folders not to be deleted (causing test failures due to left over corruption markers). This commit removes the listeners in favor of a simple timeout based solution to be use until a better listener based solution is ready ( elastic#8608 ).
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Dec 19, 2014
elastic#8436 has introduced shard level locks in order to prevent directory reuse during fast deletion & creation of indices. As part for the change, close listeners were introduced to delete the folders once all out standing references were released. The new change has created race conditions causing shard folders not to be deleted (causing test failures due to left over corruption markers). This commit removes the listeners in favour of a simple timeout based solution to be use until a better listener based solution is ready ( elastic#8608 ). Closes elastic#9009
|
closing in favor of #9083 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When an index service gets closed due to a delete index event it's possible
that shards are already removed from the service that are currently in
recovery. The recovery source applies the delete index first and causes
the target to fail its shard. This shard gets removed from the service
but it's content doesn't get deleted if the actual delete index is faster
applied. This commit makes sure we wait for the pending removals that
are in-flight before we close the index an delete it's content too once
the lock is released.