Bulk operation fail to replicate operations when a mapping update times out #30379
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
we may fail to properly replicate operation when a mapping update on master fails. If a bulk operations needs a mapping update half way, it will send a request to the master before continuing to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is a problem because all operations that were processed so far are not replicated any more to the replicas. Although these operations were never "acked" to the user (we threw an error) it cause the local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. This PR changes the logic to treat *any* mapping update failure as a document level failure, meaning only the relevant indexing operation will fail. Backport of elastic#30244
bleskes
added
>non-issue
:Distributed/CRUD
|
Pinging @elastic/es-distributed |
ywelsch
approved these changes
May 4, 2018
| import java.util.concurrent.CountDownLatch; | ||
| import java.util.concurrent.atomic.AtomicReference; | ||
|
|
||
| public class BlockMasterServiceOnMaster extends SingleNodeDisruption { |
There was a problem hiding this comment.
5.x does not have a separate MasterService. Maybe just use BlockClusterStateProcessing instead of introducing this class?
There was a problem hiding this comment.
Good suggestion. Will do.
|
Thanks @ywelsch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Starting with the refactoring in #22778 (released in 5.3)
we may fail to properly replicate operation when a mapping update on master fails.
If a bulk operations needs a mapping update half way, it will send a request to the
master before continuing to index the operations. If that request times out or isn't
acked (i.e., even one node in the cluster didn't process it within 30s), we end up
throwing the exception and aborting the entire bulk. This is a problem because all
operations that were processed so far are not replicated any more to the replicas.
Although these operations were never "acked" to the user (we threw an error) it
cause the local checkpoint on the replicas to lag (on 6.x) and the primary and replica
to diverge.
This PR changes the logic to treat any mapping update failure as a document level
failure, meaning only the relevant indexing operation will fail.
Back port of #30244