KAFKA-20876: Windowed restore optimisation gives up when endOffset-1 is a transaction control record (4.3) - #23087
Open
alanlau28 wants to merge 1 commit into
Open
KAFKA-20876: Windowed restore optimisation gives up when endOffset-1 is a transaction control record (4.3)#23087alanlau28 wants to merge 1 commit into
endOffset-1 is a transaction control record (4.3)#23087alanlau28 wants to merge 1 commit into
Conversation
… to log-start Backports KAFKA-13499 to 4.3 with the two defects found on a soak already fixed, rather than backporting the original and following up twice. A windowed store discards data older than its retention on write, so a checkpointless restore only needs the recent window. Restoring from log-start instead leaves zero margin against a delete-retention changelog: log-start is the offset the broker is deleting from, so the restore is lapped into OffsetOutOfRangeException whenever it cannot out-read the writer. StoreChangelogReader now resolves each windowed store's retention and seeks to latestTimestamp - retentionPeriod, falling back to seekToBeginning when no data record can be found. Two things the original backport got wrong, both measured on a 4.3 EOS soak: - The head-timestamp probe read endOffset - 1 and gave up on an empty poll. Under EOS that offset is almost always a transaction control record, which occupies an offset but is never delivered to a consumer: endOffset - 1 returned a record on 0 of 195 probes, and a segment dump shows the last batch is isControl=true with the pattern recurring at every transaction boundary. The probe now starts at endOffset - 32 and polls repeatedly at each position before stepping back, bounded by PROBE_MAX_ATTEMPTS and the log's beginning. It takes the newest record of the returned batch, so a deeper start costs nothing in accuracy. - One poll returns as soon as any fetch lands, so a partition could come back empty because another was served first, or because its own fetch had not arrived. Re-seeking on that basis also cancels the fetch that was about to answer. Polling again at the same position resolves it: on the soak this cut probe attempts from 14-20 to 1-2 and median probe time from ~1.5s to 4ms with no change in where the restore was seeked to. PlainToHeadersWindowStoreAdapter reports the retention of the store it adapts. It holds its delegate in a private field rather than as a WrappedStateStore, so extractRetentionPeriod's unwrap walk terminated on the adapter and resolved -1, silently skipping the optimisation for every stream-stream join store. shouldRetryProbePollBeforeFallingBackToLogStart fails without the probe change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://issues.apache.org/jira/browse/KAFKA-20876
4.3 restores a checkpointless windowed store from log-start unconditionally. Log-start is the offset a delete-retention changelog is being trimmed from, so the restore has zero margin: it survives only by out-reading the writer, and raises
OffsetOutOfRangeExceptionotherwise, marking the task corrupted.This backports KAFKA-13499 to 4.3 with #23086.
Also includes the
PlainToHeadersWindowStoreAdapterfix (#23037), which trunk carries in a separate PR but which 4.3 needs here: the adapter holds its delegate in a private field rather than as aWrappedStateStore, soextractRetentionPeriod's unwrap walk terminates on it and resolves-1, silently skipping the optimisation for every stream-stream join store.