Skip to content

KAFKA-20538: Make RocksDBTimeOrderedKeyValueBuffer headers-aware#22166

Draft
aliehsaeedii wants to merge 3 commits intoapache:trunkfrom
aliehsaeedii:rocksdb_buffer_headers_aware
Draft

KAFKA-20538: Make RocksDBTimeOrderedKeyValueBuffer headers-aware#22166
aliehsaeedii wants to merge 3 commits intoapache:trunkfrom
aliehsaeedii:rocksdb_buffer_headers_aware

Conversation

@aliehsaeedii
Copy link
Copy Markdown
Contributor

@aliehsaeedii aliehsaeedii commented Apr 28, 2026

Problem: RocksDBTimeOrderedKeyValueBuffer (the persistent buffer used by stream-table joins with grace periods) was missed when headers were propagated through Streams serdes. Two bugs:

  1. evictWhile() deserializes with the wrong headers. Both key and value deserializers are called with internalContext.headers() — the headers of whatever record the processor is currently handling, not the record being evicted. Since records can sit in the buffer for the entire grace period, these almost never match.
  2. put() does not persist the record's headers. It stores the caller-supplied recordContext directly in BufferValue, so record.headers() is never copied into the persisted ProcessorRecordContext and is unrecoverable at eviction time.

This PR fixes the followings:

  • put() builds a fresh ProcessorRecordContext carrying record.headers() and stores that in BufferValue, so the headers are persisted with the record
  • evictWhile() uses bufferValue.context().headers() for both key and value deserialization, so each evicted record is deserialized with the headers it was buffered with.

Testing: Unit tests

  • New tests assert headers propagate through eviction, that the record's headers (not the surrounding processor context's) are what surface, and that mutating the live processor context between put and evict does not affect the evicted record's headers.

@github-actions github-actions Bot added streams small Small PRs triage PRs from the community labels Apr 28, 2026
@aliehsaeedii aliehsaeedii changed the title KAFKA-20413: Make RocksDBTimeOrderedKeyValueBuffer headers-aware KAFKA-20538: Make RocksDBTimeOrderedKeyValueBuffer headers-aware Apr 28, 2026
@aliehsaeedii aliehsaeedii marked this pull request as draft April 28, 2026 10:35
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-attention small Small PRs streams triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant