[v23.3.x] cloud_storage: Fix timequery bug that triggers full scan #16518
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use timestamp to generate segments. (cherry picked from commit fff0d14)
Add new overload of the 'scan_remote_partition' function used in tests. This overload uses timequery. Also, add timestamps to the metadata. (cherry picked from commit 6d18fad)
Add extra fields without exposing them to the metrics endpoint. The fields are bytes_skipped and bytes_accepted. Also, add accessor methods so the probe values could be read programmatically. (cherry picked from commit 04ac537)
Update bytes_skip when the batches are skipped in the segment reader. Update bytes_accept when the batches are accepted. (cherry picked from commit adefd4c)
The new test uses remote_partition test code but performs timequery. It reproduces the issue from 16479 among other things. (cherry picked from commit f9a9afe)
The remote_partition_reader can reset to the begining of the partition in case if it didn't find the segment and both config.start_offset and config.max_offset are outside of the manifest. When this logic is applied to the timequery in case if timestamp overshoots last segment in the manifest we end up having a full partition scan. This commit fixes this by disabling this logic for timequeries. (cherry picked from commit f4f6743)
Lazin
approved these changes
Feb 7, 2024
|
CI failiure is #14655 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #16503
Fixes: #16517,