Use local checkpoint to calculate min translog gen for recovery #51905
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
added
>enhancement
:Distributed/Engine
|
Pinging @elastic/es-distributed (:Distributed/Engine) |
ywelsch
reviewed
Feb 7, 2020
There was a problem hiding this comment.
I think that the uncommittedSizeInBytes and uncommittedOperations metrics are pretty useless today, as they are not a measure of how much data needs to be recovered/replayed after a crash. I would rather base these metrics on the local checkpoint of the safe commit in all cases, and completely disregard the last commit (which is irrelevant for recovery).
server/src/main/java/org/elasticsearch/index/engine/CombinedDeletionPolicy.java
Show resolved
Hide resolved
++. I pushed c4bccea. |
|
There are relevant test failures here |
henningandersen
approved these changes
Feb 7, 2020
server/src/main/java/org/elasticsearch/index/engine/NoOpEngine.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/index/engine/NoOpEngineTests.java
Show resolved
Hide resolved
|
@ywelsch @henningandersen Thanks for reviewing. |
This was referenced Feb 10, 2020
dnhatn
added a commit
that referenced
this pull request
Feb 11, 2020
We roll a new translog generation and trim operations that are above the global checkpoint during primary-replica resync. If the initOperations is empty, then the stale operation on the replica2 will be discarded as it is the only operation in a translog file (since #51905 where we started using the local checkpoint to calculate the minimum required translog generation for recovery). Otherwise, the stale op will be retained along with initOperations but will be skipped in snapshots. Relates #51905 Closes #52148
This was referenced Feb 11, 2020
dnhatn
added a commit
that referenced
this pull request
Feb 12, 2020
Since #51905, we use the local checkpoint of the safe commit to calculate the number of uncommitted operations of a translog stats. If a periodic flush triggered by afterWriteOperation completes before we sync translog, then the last commit is not safe. We also need to sync translog from Engine instead of the translog so that we can advance the safe commit. Relates #51905 Closes #52223
dnhatn
added a commit
that referenced
this pull request
Feb 12, 2020
dnhatn
added a commit
that referenced
this pull request
Feb 12, 2020
We need to reduce the translog sync interval for indices with translog async setting so that we can have the safe commit in the assertBusy interval. This is needed since #51905, where we use the local checkpoint of the safe commit to calculate the number of uncommitted operations of a translog stats. Closes #52251 Relates #51905
dnhatn
added a commit
that referenced
this pull request
Feb 12, 2020
We need to reduce the translog sync interval for indices with translog async setting so that we can have the safe commit in the assertBusy interval. This is needed since #51905, where we use the local checkpoint of the safe commit to calculate the number of uncommitted operations of a translog stats. Closes #52251 Relates #51905
dnhatn
added a commit
that referenced
this pull request
Feb 18, 2020
sbourke
pushed a commit
to sbourke/elasticsearch
that referenced
this pull request
Feb 19, 2020
Asserts that no new operations are made into the translog since we re-opened the engine. Relates elastic#51905 Closes elastic#52410
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Feb 26, 2020
…tic#51905) Today we use the translog_generation of the safe commit as the minimum required translog generation for recovery. This approach has a limitation, where we won't be able to clean up translog unless we flush. Reopening an already recovered engine will create a new empty translog, and we leave it there until we force flush. This commit removes the translog_generation commit tag and uses the local checkpoint of the safe commit to calculate the minimum required translog generation for recovery instead. Closes elastic#49970
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Feb 26, 2020
Since elastic#51905, we skip translog recovery if the local checkpoint of the safe commit equals to the global checkpoint. This change adjusts the test not to create a new snapshot in that case. Closes elastic#52221 Relates elastic#51905
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Feb 26, 2020
Since elastic#51905, we use the local checkpoint of the safe commit to calculate the number of uncommitted operations of a translog stats. If a periodic flush triggered by afterWriteOperation completes before we sync translog, then the last commit is not safe. We also need to sync translog from Engine instead of the translog so that we can advance the safe commit. Relates elastic#51905 Closes elastic#52223
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Feb 26, 2020
Asserts that no new operations are made into the translog since we re-opened the engine. Relates elastic#51905 Closes elastic#52410
dnhatn
added a commit
that referenced
this pull request
Feb 26, 2020
Today we use the translog_generation of the safe commit as the minimum required translog generation for recovery. This approach has a limitation, where we won't be able to clean up translog unless we flush. Reopening an already recovered engine will create a new empty translog, and we leave it there until we force flush. This commit removes the translog_generation commit tag and uses the local checkpoint of the safe commit to calculate the minimum required translog generation for recovery instead. Closes #49970
dnhatn
added a commit
that referenced
this pull request
Feb 26, 2020
dnhatn
added a commit
that referenced
this pull request
Feb 26, 2020
Since #51905, we use the local checkpoint of the safe commit to calculate the number of uncommitted operations of a translog stats. If a periodic flush triggered by afterWriteOperation completes before we sync translog, then the last commit is not safe. We also need to sync translog from Engine instead of the translog so that we can advance the safe commit. Relates #51905 Closes #52223
dnhatn
added a commit
that referenced
this pull request
Feb 26, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today we use the translog_generation of the safe commit as the minimum required translog generation for recovery. This approach has a limitation, where we won't be able to clean up translog unless we flush. Reopening an already recovered engine will create a new empty translog, and we leave it there until we force flush.
This commit removes the translog_generation commit tag and uses the local checkpoint of the safe commit to calculate the minimum required translog generation for recovery instead.
Closes #49970