KAFKA-19821: Duplicated batches should be logged #20740
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
storage
chia7712
reviewed
Oct 21, 2025
|
|
||
| Optional<BatchMetadata> duplicateBatch = maybeLastEntry.flatMap(e -> e.findDuplicateBatch(batch)); | ||
| if (duplicateBatch.isPresent()) { | ||
| logger.info("Found duplicate batch from client, duplicateBatchMetadata={}", duplicateBatch.get()); |
Member
There was a problem hiding this comment.
Perhaps we should move the log statement to UnifiedLog#append. Since another branch already has the log logger.trace("Appended message set with ..., it would be straightforward to add a similar log for duplicate batches
Member
Author
There was a problem hiding this comment.
@chia7712, thanks for the suggestion! Indeed, move the log.info to append method seems better.
chia7712
reviewed
Oct 22, 2025
| appendInfo.setLastOffset(duplicate.lastOffset()); | ||
| appendInfo.setLogAppendTime(duplicate.timestamp()); | ||
| appendInfo.setLogStartOffset(logStartOffset); | ||
| logger.info("Duplicate batch detected, returning AppendInfo from duplicate batch with last offset: {}, first offset: {}, next offset: {}, skipped messages: {}", |
Member
There was a problem hiding this comment.
I'm a bit concerned about the log flooding if the producer has network problems. Wouldn't the trace level be adequate?
Member
Author
There was a problem hiding this comment.
I’m good with either info or trace. As long as we leave a debug message behind, that’s fine 😃
brandboat
changed the title
chia7712
approved these changes
Oct 22, 2025
joshua2519
pushed a commit
to joshua2519/kafka
that referenced
this pull request
Oct 27, 2025
When a duplicate batch is detected, the entire MemoryRecords instance is skipped to prevent appending duplicate data to the log. This operation is silent, adding a log.info message here to provide better observability. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
eduwercamacaro
pushed a commit
to littlehorse-enterprises/kafka
that referenced
this pull request
Nov 12, 2025
When a duplicate batch is detected, the entire MemoryRecords instance is skipped to prevent appending duplicate data to the log. This operation is silent, adding a log.info message here to provide better observability. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a duplicate batch is detected, the entire MemoryRecords instance is
skipped to prevent appending duplicate data to the log. This operation
is silent, adding a log.info message here to provide better
observability.
Reviewers: Chia-Ping Tsai chia7712@gmail.com