Ensure shards are deleted under lock on close #8579
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -357,7 +358,8 @@ public void onAllShardsClosed(Index index, List<Throwable> failures) { | |||
| @Override | |||
| public void onShardClosed(ShardId shardId) { | |||
| try { | |||
| nodeEnv.deleteShardDirectorySafe(shardId); | |||
| // this is called under the shard lock - we can safely delete it | |||
There was a problem hiding this comment.
Can you add that this method is called while the shard lock is held to the javadocs for IndexCloseListener.onShardClosed? Currently it says "Invoked once the last resource using the given shard ID is released" which isn't accurate since the lock is still held
|
@s1monw left a couple of comments |
|
@dakrone updated based on your comments |
|
LGTM, thanks for adding the documentation |
Today there is a race condition between the actual deletion of the shard and the release of the lock in the store. This race can cause rare imports of dangeling indices if the cluster state update loop tires to import the dangeling index in that particular windonw. This commit adds more safety to the import of dangeling indices and removes the race condition by holding on to the lock on store closing while the listener is notified.
s1monw
force-pushed
the
fix_lock_dangel_foo_bar
branch
from
fcc9966
to
a6e6c4e
Compare
clintongormley
added
:Core/Infra/Core
clintongormley
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today there is a race condition between the actual deletion of
the shard and the release of the lock in the store. This race can cause
rare imports of dangeling indices if the cluster state update loop
tires to import the dangeling index in that particular windonw. This commit
adds more safety to the import of dangeling indices and removes the race
condition by holding on to the lock on store closing while the listener
is notified.