Do not ignore shard not-available exceptions in replication #28571
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The shard not-available exceptions are currently ignored in the replication as the best effort avoids failing not-yet-ready shards. However these exceptions can also happen from fully active shards. If this is the case, we may have skipped important failures from replicas. Since elastic#28049, only fully initialized shards are received write requests. This restriction allows us to handle all exceptions in the replication. There is a side-effect with this change. If a replica retries its peer recovery second time after being tracked in the replication group, it can receive replication requests even though it's not-yet-ready. That shard may be failed and allocated to another node even though it has a good lucene index on that node. This PR does not change the way we report replication errors to users, hence the shard not-available exceptions won't be reported as before.
dnhatn
added
>enhancement
:Distributed/Recovery
|
Thanks @ywelsch for your quick review. |
dnhatn
added a commit
that referenced
this pull request
Feb 17, 2018
The shard not-available exceptions are currently ignored in the replication as the best effort avoids failing not-yet-ready shards. However these exceptions can also happen from fully active shards. If this is the case, we may have skipped important failures from replicas. Since #28049, only fully initialized shards are received write requests. This restriction allows us to handle all exceptions in the replication. There is a side-effect with this change. If a replica retries its peer recovery second time after being tracked in the replication group, it can receive replication requests even though it's not-yet-ready. That shard may be failed and allocated to another node even though it has a good lucene index on that node. This PR does not change the way we report replication errors to users, hence the shard not-available exceptions won't be reported as before. Relates #28049 Relates #28534
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Apr 27, 2018
Since elastic#28049, only fully initialized shards are received write requests. This enhancement allows us to handle all exceptions. In elastic#28571, we started strictly handling shard not-available exceptions and tried to keep the way we report replication errors to users by only reporting if the error is not shard-not-available exceptions. However, since then we unintentionally always log warn for all exception. This change restores to the previous behavior to log warn only if an exception is not a shard not-available exception.
dnhatn
added a commit
that referenced
this pull request
Apr 27, 2018
Since #28049, only fully initialized shards are received write requests. This enhancement allows us to handle all exceptions. In #28571, we started strictly handling shard-not-available exceptions and tried to keep the way we report replication errors to users by only reporting if the error is not shard-not-available exceptions. However, since then we unintentionally always log warn for all exception. This change restores to the previous behavior which logs warn only if an exception is not a shard-not-available exception. Relates #28049 Relates #28571
dnhatn
added a commit
that referenced
this pull request
Apr 27, 2018
Since #28049, only fully initialized shards are received write requests. This enhancement allows us to handle all exceptions. In #28571, we started strictly handling shard-not-available exceptions and tried to keep the way we report replication errors to users by only reporting if the error is not shard-not-available exceptions. However, since then we unintentionally always log warn for all exception. This change restores to the previous behavior which logs warn only if an exception is not a shard-not-available exception. Relates #28049 Relates #28571
dnhatn
added a commit
that referenced
this pull request
Apr 27, 2018
Since #28049, only fully initialized shards are received write requests. This enhancement allows us to handle all exceptions. In #28571, we started strictly handling shard-not-available exceptions and tried to keep the way we report replication errors to users by only reporting if the error is not shard-not-available exceptions. However, since then we unintentionally always log warn for all exception. This change restores to the previous behavior which logs warn only if an exception is not a shard-not-available exception. Relates #28049 Relates #28571
ywelsch
added a commit
that referenced
this pull request
Jun 11, 2018
Swallowing these exceptions is dangerous as they can result in replicas going out-of-sync with the primary. Follow-up to #28571
ywelsch
added a commit
that referenced
this pull request
Jun 11, 2018
Swallowing these exceptions is dangerous as they can result in replicas going out-of-sync with the primary. Follow-up to #28571
ywelsch
added a commit
that referenced
this pull request
Jun 11, 2018
Swallowing these exceptions is dangerous as they can result in replicas going out-of-sync with the primary. Follow-up to #28571
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Sep 11, 2018
We fail to notify the resync listener if the resync replication hits a shard unavailable exception. Moreover, we no longer need to swallow these unavailable exceptions. Relates elastic#28571 Closes elastic#33613
dnhatn
added a commit
that referenced
this pull request
Sep 13, 2018
dnhatn
added a commit
that referenced
this pull request
Sep 13, 2018
dnhatn
added a commit
that referenced
this pull request
Sep 13, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The shard not-available exceptions are currently ignored in the
replication as the best effort avoids failing not-yet-ready shards.
However these exceptions can also happen from fully active shards. If
this is the case, we may have skipped important failures from replicas.
Since #28049, only fully initialized shards are received write requests.
This restriction allows us to handle all exceptions in the replication.
There is a side-effect with this change. If a replica retries its peer
recovery second time after being tracked in the replication group, it
can receive replication requests even though it's not-yet-ready. That
shard may be failed and allocated to another node even though it has a
good lucene index on that node.
This PR does not change the way we report replication errors to users,
hence the shard not-available exceptions won't be reported as before.
Relates #28049
Relates #28534