KAFKA-7401: Fix inconsistent range exception on segment recovery #6220
Conversation
| val fetchDataInfo = segment.read(startOffset, None, Int.MaxValue) | ||
| if (fetchDataInfo != null) | ||
| loadProducersFromLog(stateManager, fetchDataInfo.records) | ||
| if (stateManager.mapEndOffset < segment.baseOffset) { |
There was a problem hiding this comment.
The basic intent of this logic is to ensure that we have consistent producer state up to the base offset of the segment we're recovering before we begin the actual recovery. In this case, the log start offset is somewhere above the base offset of this segment, which makes the call to truncateAndReload above a little strange (i.e. the end offset is smaller than the start offset). This seems to work, but feels brittle. Since we do not actually do partial segment recovery, I wonder if we could do something like this:
val recoveryStartOffset = math.min(logStartOffset, segment.baseOffset)
stateManager.truncateAndReload(recoveryStartOffset, segment.baseOffset, time.milliseconds)Then the map end offset is always no greater than the segment base offset. What do you think?
There was a problem hiding this comment.
Thanks for the context. Given we want to have consistent producer state up to the base offset of the segment we are recovering, and truncateAndReload will set the map end offset properly in this case, this approach seems better. I also tested it just now, and it works (fixes the original bug). I will update the PR with your suggestion.
|
Was this released in Kafka 1.1.2? I cannot find the download here https://kafka.apache.org/downloads or here https://archive.apache.org/dist/kafka/ This ticket, https://issues.apache.org/jira/browse/KAFKA-7401, claims it was released as part of 1.1.2 If it hasn't been released, what effort would it be to get it released? We are currently unable to start one of our brokers due to this. Thank you! |
This PR fixes "java.lang.IllegalArgumentException: inconsistent range" which happens on broker startup after unclean shutdown during log cleaning phase that creates swap files (in case where base offset < log start offset). Added
testRecoveryAfterCrashAndIncrementedLogStartOffsetthat reproduces Kafka-7401.Committer Checklist (excluded from commit message)