Skip to content

KAFKA-20413: Add missing headers-aware ListValueStore #22859

Open
aliehsaeedii wants to merge 11 commits into
apache:trunkfrom
aliehsaeedii:headers-list-store-upgrade-migration
Open

KAFKA-20413: Add missing headers-aware ListValueStore #22859
aliehsaeedii wants to merge 11 commits into
apache:trunkfrom
aliehsaeedii:headers-list-store-upgrade-migration

Conversation

@aliehsaeedii

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #22156 (headers-aware outer-join list-value store). That PR makes the stream-stream outer-join spurious-results store read its values as AggregationWithHeaders<LeftOrRightValue> when dsl.store.format=headers, instead of the plain LeftOrRightValue. Because that store is persistent (RocksDB) and changelogged, flipping the config on an existing application leaves old PLAIN records that the new headers serde mis-parses:

  • right-side values are silently corrupted — the leading 0x00 LeftOrRightValue flag is read as an empty-headers headersSize=0 varint, dropping the first value byte (rightight);
  • left-side values throw SerializationException: negative length -1 (the 0x01 flag is read as headersSize=1).

This PR migrates old data safely instead of corrupting it.

Approach

Mirrors the KIP-1271 dual-column-family upgrade pattern (RocksDBTimestampedStoreWithHeaders / DualColumnFamilyAccessor), adapted to the list store. The join processor, OuterJoinStoreWrapper, and ListValueStore are unchanged; the work is in the store factory/builder wiring plus a new bytes store.

Note this store can't reuse RocksDBTimestampedStoreWithHeaders directly: its values are ListSerde blobs with headers stored per element inline, so the whole-value [0x00][ts=-1] migration converter would corrupt the list encoding. Instead:

  • RocksDBListValueStoreWithHeaders — a non-timestamped, headers-aware, dual-CF store. Legacy PLAIN blobs stay in the DEFAULT column family and are lifted to the headers format on read/write (prepend 0x00 per list element) via DualColumnFamilyAccessor, migrating into a new listValueWithHeaders column family. Selected in HEADERS mode by OuterStreamJoinStoreFactory for persistent stores.
  • Changelog restore path (in-memory stores and RocksDB rebuilds after state loss have no local CF to migrate): ChangeLoggingListValueBytesStoreWithHeaders stamps a reserved marker header on every HEADERS-format record, and a new rawListValueToHeadersListValue RecordConverter converts only unmarked (legacy) records on restore. converterForStore selects it via the HeadersAwareListValueStore marker interface — deliberately not HeadersBytesStore, which would pick the single-value converter and corrupt the list blob.
  • RocksDBStore reports a clear error on an unsupported HEADERS→PLAIN downgrade of the new store.

Upgrade scenarios covered

Scenario Mechanism
Rolling upgrade, RocksDB, retained local state dual-CF migrate-on-read/write
RocksDB, lost/wiped state → full restore changelog marker + RecordConverter
In-memory store (no CF) → restore changelog marker + RecordConverter
Fresh HEADERS store records marked, converter is a no-op
Staying PLAIN unchanged

Testing

  • New unit tests: ListValueStoreUpgradeUtilsTest (blob converter + marker, incl. the previously-corrupted right value and previously-crashing left value), additions to RecordConvertersTest (marker branching / tombstone passthrough) and StateManagerUtilConverterTest (converter selection).
  • New end-to-end store test RocksDBListValueStoreWithHeadersTest: write with the pre-headers PLAIN store, reopen as the HEADERS dual-CF store, assert correct reads for left/right/multi-element values and append-across-the-upgrade-boundary.
  • Regression: KStreamKStreamOuterJoinTest, KStreamKStreamLeftJoinTest, RocksDBStoreTest, RocksDBTimestampedStoreWithHeadersTest, ListValueStoreTest. Checkstyle and spotbugs pass.

No public API change; ListValueStore is internal, so no KIP is required.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

aliehsaeedii and others added 11 commits June 24, 2026 19:13
…ormat

When dsl.store.format=headers, the stream-stream outer-join spurious-results
store now reads its values as AggregationWithHeaders<LeftOrRightValue> instead
of the plain LeftOrRightValue. Because that store is persistent (RocksDB) and
changelogged, flipping the config on an existing application left old PLAIN
records that the new headers serde mis-parsed: right-side values were silently
truncated (leading 0x00 flag read as an empty-headers varint) and left-side
values threw SerializationException.

Migrate old data safely instead, mirroring the KIP-1271 dual-column-family
pattern used by RocksDBTimestampedStoreWithHeaders, adapted to the list store
whose values are ListSerde blobs of per-element encodings:

- RocksDBListValueStoreWithHeaders keeps legacy PLAIN blobs in the DEFAULT
  column family and lifts each element to the empty-headers format on read/write
  (prepend 0x00 per list element) via DualColumnFamilyAccessor, migrating them
  into a new listValueWithHeaders column family. Selected in HEADERS mode by
  OuterStreamJoinStoreFactory for persistent RocksDB stores.
- The changelog path (in-memory stores and RocksDB rebuilds have no local CF to
  migrate) is covered by a version marker: ChangeLoggingListValueBytesStoreWithHeaders
  stamps a reserved header on every HEADERS-format record, and a new
  rawListValueToHeadersListValue RecordConverter converts only unmarked (legacy)
  records on restore. converterForStore selects it via the HeadersAwareListValueStore
  marker interface (deliberately not HeadersBytesStore, which would pick the
  single-value headers converter and corrupt the list blob).
- RocksDBStore reports a friendly error on an unsupported HEADERS-to-PLAIN
  downgrade of the new store.

Adds unit tests for the blob converter and marker, the RecordConverter branching,
converterForStore selection, and end-to-end RocksDB dual-CF upgrade tests
(including append-across-the-upgrade-boundary).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added triage PRs from the community streams labels Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

streams triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant