[v22.3.x] cloud_storage: fix timequery edge cases #9876
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The local GC test issues many requests in parallel, which can make it awkward to read logs. On failure, issue some queries sequentially to the offsets that failed, which also gives us a measurement of whether the failure was transient or permanent. Related: redpanda-data#9669 (cherry picked from commit 8ddab78)
_start_offset is only set once some background housekeeping happens: on a freshly created partition with a few records produced into it, it can be -1. In this case when we read the start timestamp, we should not try to find a segment containing this negative start offset. We may later refactor to avoid having non-empty logs carry a confusingly -1 start_offset. (cherry picked from commit f440c89)
... so that it may be executed against Apache Kafka to check the compatibility of the behavior. (cherry picked from commit 8693ad2)
This was incorrectly expecting a -1 when querying below the start of the log: the Kafka-compatible behavior is to return the first offset in the log with a timestamp greater than the query (i.e. the lwm of the log) (cherry picked from commit e2e4186)
...where the queried timestamp falls equal to the local log start_offset, or between the local log's start_timestamp() and start_offset. The existing code was incorrectly assuming that start_timestamp pointed to the same record as start_offset, whereas actually start_timestamp gives the timestamp at the start of the first segment, and start_offset may point to some location within a segment. This would result in timequery returning an offset too high if there was data in tiered storage that could have found the timestamp at a lower offset, or returning an offset when it should have returned null, if there was not tiered storage data available before start_offset. Fixes redpanda-data#9669 (cherry picked from commit f3f77f0)
|
Ping |
Methods 'rbegin()' and 'rend()' of the partition_manifest are only used to get last value in the manifest. The methods are replaced with 'last_segment' method that returns optional value which may contain a segment_meta value of the last uploaded segment in the partition. (cherry picked from commit 78a70b3)
|
Failure appears to be #10024 |
andrwng
approved these changes
Apr 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #9815
Fixes #9875,