Backport #3372 (Delay prefix truncation of offset translator until segments are deleted) #3541
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 2288708)
See redpanda-data#3377. Critical invariant for _start_offset is that it never decreases. Previously this invariant could be violated in a couple of ways. * Because _start_offset was updated only at the end of _truncate_prefix function, if all segments were deleted, external observer could observe _start_offset going backwards to the previous value after all segments have been deleted, but before it was updated in memory. * Because kvstore updates were not serialized, we could never be sure that we are not overwriting the bigger _start_offset value that gets written as a part of concurrent prefix_truncate. To mitigate that we serialize the kvstore updates with a mutex and update in-memory _start_offset at the beginning of prefix_truncate. (cherry picked from commit 038d6c8)
It doesn't make much sense for the max_collectible_offset to decrease, and making it non-decreasing guards against reordered calls to set_collectible_offset (cherry picked from commit a78a6c6)
Previously we prefix truncated offset translator immediately after writing the snapshot, but actual deletion of log segments happened later in the log gc loop. After this commit deletion of segments happens immediately and offset translator is prefix truncated after it. This allows waiting for readers that were reading before the prefix truncation point before prefix truncating the offset translator. (cherry picked from commit 2c9e003)
Because we need offset translation information to get offset ranges of aborted transactions, we extend the reader lifetime to the part.aborted_transactions call. This way reader will hold on to the segment locks and delay possible prefix truncation that could discard needed offset translation info. (cherry picked from commit bf9af3f)
ztlpn
requested review from
dotnwat,
ivotron,
jcsp,
LenaAn,
mmaslankaprv,
NyaliaLui,
rystsov and
VadimPlh
as code owners
jcsp
approved these changes
Jan 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
Backport #3372 to v21.11.x.