Skip to content

KAFKA-10369: Add TimestampAndOffset value type of deduplication store (1/N)#22820

Open
AyoubOm wants to merge 1 commit into
apache:trunkfrom
AyoubOm:deduplication_timestamp_and_offset
Open

KAFKA-10369: Add TimestampAndOffset value type of deduplication store (1/N)#22820
AyoubOm wants to merge 1 commit into
apache:trunkfrom
AyoubOm:deduplication_timestamp_and_offset

Conversation

@AyoubOm

@AyoubOm AyoubOm commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

This is a first PR for adding deduplication operator to Streams DSL. See this global PR draft for reference #22189

This PR adds the TimestampAndOffset class which will be used as a the value type of the deduplication store. The store keeps track of the records seen so far, and will be used by the processor to deduplicate records. See wiki and mail thread discussion for the reason we need the timestamp and offset.

In brief:

  • Deduplication is bounded in time and the store doesn't keep records forever - old records should be evicted from the store => Hence Timestamp

  • Records may be processed twice in case of a failure - we need to track records that have been saved in the store and still not committed => Hence Offset

JIRA

KAFKA-10369

@github-actions github-actions Bot added triage PRs from the community streams small Small PRs labels Jul 13, 2026
@github-actions

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-attention small Small PRs streams triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant