Skip to content

[runtime] send Kafka tombstones on state pruning for durable log compaction#885

Open
rob-9 wants to merge 2 commits into
apache:mainfrom
rob-9:feat/691-kafka-prune-tombstones
Open

[runtime] send Kafka tombstones on state pruning for durable log compaction#885
rob-9 wants to merge 2 commits into
apache:mainfrom
rob-9:feat/691-kafka-prune-tombstones

Conversation

@rob-9

@rob-9 rob-9 commented Jul 8, 2026

Copy link
Copy Markdown

Addresses #691

Purpose of change

KafkaActionStateStore.pruneState() only removed entries from the in-memory cache. the underlying Kafka topic was never updated, so:

  1. the topic grew unbounded, meaning Kafka log compaction had no tombstones to trigger key deletion
  2. after a restart, rebuildState() replayed pruned records back into memory, resurrecting state that should have been discarded

This PR sends null-valued records to the state topic during pruning, and updates rebuildState() to skip tombstoned keys during replay instead of inserting them into the cache. Also fixes an NPE in the error-logging path that called record.value().toString() on a null value.

Tests

mvn --batch-mode --no-transfer-progress -pl runtime -am -Dtest=KafkaActionStateStoreTest test

  • testPruneState verifies tombstones are produced and in-memory entries are evicted
  • testPruneStateSendsTombstonesWithCorrectKeys asserts exact composite keys on produced tombstones
  • testPruneStateNoMatchingKeys verifies no tombstones sent when no keys match
  • testPruneStateWithNullProducer verifies graceful degradation when producer is null

API

No public API changes.

Documentation

  • doc-not-needed

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant