KAFKA-20876: Windowed restore optimisation gives up when endOffset-1 is a transaction control record - #23086
Open
alanlau28 wants to merge 7 commits into
Open
KAFKA-20876: Windowed restore optimisation gives up when endOffset-1 is a transaction control record#23086alanlau28 wants to merge 7 commits into
endOffset-1 is a transaction control record#23086alanlau28 wants to merge 7 commits into
Conversation
Three defects stopped the KAFKA-13499 windowed restore optimisation from ever executing under exactly-once. All are reachable on trunk with enable.transactional.statestores at its default of false, since the EOS unclean close still wipes the task directory and forces a from-scratch restore. 1. PlainToHeadersWindowStoreAdapter holds its delegate in a private field rather than as a WrappedStateStore, so extractRetentionPeriod's unwrap walk ends on the adapter and resolves -1. That fails the gate and silently skips the optimisation for every stream-stream join store. Measured on a 4.3 soak carrying a backport: 441 and 379 non-finite retentions, 0 optimised seeks. 2. The head-timestamp probe seeked endOffset-1 and gave up on an empty poll. Under EOS that offset is a transaction control record -- confirmed by dumping a live segment: the last batch is isControl=true, count=1, size=78, and the pattern recurs at every transaction boundary. Markers occupy an offset but are never delivered to a consumer, so the poll returns nothing. Replaced with a bounded backward probe. It starts at endOffset-32 rather than -1, which costs nothing in accuracy because the probe takes the newest record of the returned batch. 3. The probe shared one poll across all unresolved partitions, so an empty result could merely mean another partition's fetch landed first -- doubling that partition's step-back on false evidence and spending a shared attempt budget on its behalf. Partitions starved into log-start seeks with zero margin and were lapped into OffsetOutOfRangeException. Each partition is now probed alone, with the others paused. Note poll() updates fetch positions for the whole assignment, so every probed partition is seeked up front; without that the first poll throws NoOffsetForPartitionException under auto.offset.reset=none. Tests: StoreChangelogReaderTest 47, SeekFallback 5, adapter 2, unwrap probe 1, soak topology probe 1, TxnStoreWipeAB 3, and a 7-test EOS-v2 integration suite against a real cluster. Each fix was verified to fail its test when reverted.
…e back Every probe attempt re-seeks before polling, which cancels a fetch that was in flight. An empty poll therefore does not distinguish "no record at this offset" from "the fetch has not landed yet", and the step-back doubles on that false evidence. Soak evidence that this is happening: with the probe starting at endOffset-32, ~46% of probes still walk out to backUsed 512-2048, and probeMs medians sit at 1.0-1.5s against ~0.5s for the previous build. Long runs of control records would explain that, but so would cancelled fetches, and the two are indistinguishable from the current instrumentation. Polling a second time at the same position separates them. If the deep step-backs are self-inflicted, backUsed should collapse to 32 and probeMs should fall; if they persist, the runs of control records are real. Note the existing tests pass either way -- this changes timing behaviour that neither MockConsumer nor a loopback EmbeddedKafkaCluster reproduces, so the result has to come from the soak.
… probing Individual polling removed the starvation but lost amortisation: 5 partitions cost 22 attempts and 2013ms against 1 attempt and 3ms for a shared poll that is simply polled more than once. The shared poll always could serve every partition; the defect was giving up after the first empty result, which only means a fetch has not landed. Also fixes the regression test's observable: a partition's position after restore reflects records it has since consumed, not where it was seeked, so the test now records seekToBeginning calls directly.
… one empty poll When restoring a windowed store with no checkpoint, StoreChangelogReader seeks to latestTimestamp - retentionPeriod rather than log-start, since data older than the retention is discarded on write. It learned latestTimestamp by seeking to endOffset - 1 and polling once; an empty poll fell back to seekToBeginning. Two reasons that poll comes back empty without the offset being at fault: - Under EOS the last offset is almost always a transaction control record, which occupies an offset but is never delivered to a consumer. Measured on a soak, endOffset - 1 returned a record on 0 of 195 probes, and dumping a segment shows the last batch is isControl=true with the pattern recurring at every transaction boundary. - One poll returns as soon as any fetch lands, so a partition can be empty because another was served first or because its own fetch has not arrived. The probe now starts at endOffset - 32 and polls repeatedly at each position before stepping back, bounded by PROBE_MAX_ATTEMPTS and the log's beginning. It takes the newest record of the returned batch, so a deeper start costs nothing in accuracy. Falling back to seekToBeginning remains the behaviour when no data record can be found. PlainToHeadersWindowStoreAdapter also now reports the retention of the store it adapts. It holds its delegate in a private field rather than as a WrappedStateStore, so extractRetentionPeriod's unwrap walk terminated on the adapter and resolved -1, silently skipping the optimisation for every stream-stream join store. shouldRetryProbePollBeforeFallingBackToLogStart fails without the change.
The probe stepped back from 32, re-seeking after three polls whether or not those polls had waited on a fetch, so a partition could be widened away from the answer it was about to give and end up restoring from log start. Start at 128, which answers the large majority outright, and give each window until it stops resolving anyone before widening to 512 and 2048.
alanlau28
force-pushed
the
KAFKA-20876-probe-retry
branch
from
August 9, 2026 15:03
f527149 to
bc2e02f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://issues.apache.org/jira/browse/KAFKA-20876
When restoring a windowed store with no checkpoint,
StoreChangelogReaderseeks tolatestTimestamp - retentionPeriodrather than log-start, since data older than the retention is discarded
on write. It learns
latestTimestampby seeking toendOffset - 1andpolling once in
seekNewPartitions; an empty poll falls back toseekToBeginning.The probe now starts at
endOffset - 32and polls repeatedly at eachposition before stepping back, bounded by
PROBE_MAX_ATTEMPTSand thelog's beginning. It takes the newest record of the returned batch, so a
deeper start costs nothing in accuracy — measured, the resolved record
sat 2 offsets behind the end.
seekToBeginningremains the behaviourwhen no data record can be found.