Max Segment Count in Snapshot Index Details from Get Snapshots API can be Wrong #74249
Assignees
Labels
>bug
:Distributed/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
Team:Distributed
Meta label for distributed team
Comments
original-brownbear
added
>bug
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes elastic#74249
original-brownbear
added a commit
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes #74249
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes elastic#74249
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes elastic#74249
original-brownbear
added a commit
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes #74249
original-brownbear
added a commit
that referenced
this issue
Jun 18, 2021
If sequence numbers are equal across snapshots (and thus no files get written to the repository) the segment count in the index commit is not reliable as it may have changed due to background merges. => fixed by always using the segment count determined from the file names in the snapshot instead closes #74249
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There appears to be a bug in the logic that computes the segment count of a shard snapshot in some background merge scenarios.
The segment count is taken from the index commit that we are snapshotting via a call to
org.apache.lucene.index.IndexCommit#getSegmentCount. This is correct for all snapshots except for those that use the sequence number based shortcut introduced in #52182 that will prevent an updated set of files from getting uploaded when there already is a shard copy at the same sequence number that is in the current index commit.This means that if a snapshot is taken once before a background merge and once after without any indexing into the shard, then the segment count stored for the second snapshot will be wrong if segment counts for the shard changed. We can fix this by taking the segment count from the files list instead in this case.
The text was updated successfully, but these errors were encountered: