release-22.2: fix unstable sort in SSTSink #97062
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a forward port of #95446.
The focus of this patch is fixing the unstable sort described in #95445. Previously, we would sort only on the basis of the start key of two backup files returned as part of an ExportResponse. This could result in older revisions of a key getting sorted and flushed to the underlying SSTWriter before new revisions, resulting in out-of-order writes, causing the backup to fail. We saw this in the support issue cockroachlabs/support#1998 and have a test TestExportRevisionsWithTimestampPagination that failed under stress.
To fix this unstable sort we now use the following algorithm:
Sort on start key of the backup files.
If start key is the same, sort on end key of the backup files. If end key is the same, sort in descending order on the end key timestamp of the files as both files are guaranteed to only contain revisions of the same key.
Fixes: #95445
Release note (bug fix): Fixes a bug where a backup of keys with many revisions would fail with "pebble: keys must be added in order".
Release justification: high impact bug fix to prevent failing backups