Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Kafka source reads duplicated messages #1438

Merged
merged 1 commit into from
Jan 5, 2024

Conversation

jy4096
Copy link
Contributor

@jy4096 jy4096 commented Jan 5, 2024

fix: #1408

https://github.com/IBM/sarama/blob/4b9e8f6681955d56471eafa1b43b62085a58e1c6/consumer_group.go#L687-L689

// To follow upstream conventions, you are expected to mark the offset of the
// next message to read, not the last message read. Thus, when calling `MarkOffset`
// you should typically add one to the offset of the last consumed message.

https://github.com/IBM/sarama/blob/4b9e8f6681955d56471eafa1b43b62085a58e1c6/consumer_group.go#L823-L825

func (s *consumerGroupSession) MarkMessage(msg *ConsumerMessage, metadata string) {
	s.MarkOffset(msg.Topic, msg.Partition, msg.Offset+1, metadata)
}

Signed-off-by: jyu6 <juanlu_yu@intuit.com>
@jy4096 jy4096 marked this pull request as ready for review January 5, 2024 06:22
@whynowy
Copy link
Member

whynowy commented Jan 5, 2024

A bug introduced by #949 :)

@jy4096 jy4096 merged commit 795bef6 into numaproj:main Jan 5, 2024
19 checks passed
@jy4096 jy4096 deleted the kafka-source-duplicate-fix branch January 5, 2024 22:04
@yhl25
Copy link
Contributor

yhl25 commented Jan 6, 2024

Thank you for the fix @jy4096!

whynowy pushed a commit that referenced this pull request Jan 15, 2024
Signed-off-by: jyu6 <juanlu_yu@intuit.com>
(cherry picked from commit 795bef6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Duplicate Kafka messages
4 participants