Skip to content

feat: add Kafka exactly-once consume-process-produce#134

Merged
joshua-temple merged 1 commit into
mainfrom
feat/source-eos
Jun 4, 2026
Merged

feat: add Kafka exactly-once consume-process-produce#134
joshua-temple merged 1 commit into
mainfrom
feat/source-eos

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

What this change does

Fleshes out the source Transactional capability into a working Kafka exactly-once path (one of the four roadmap items from #131), so a consume-process-produce cycle commits the produced records and the consumed offset as one atomic unit.

  • Core seam (source): Transactional.Begin now fences a consumed Message together with records produced through a new backend-neutral Tx handle (Tx.Produce, ProducedRecord) — emit and ack become one atomic unit. No vendor type crosses the seam.
  • source/kafka: WithTransactional(id) builds a GroupTransactSession (read-committed, no auto-commit); Begin produces through the session, marks the consumed offset on success only, and ends with TryCommit/TryAbort. A narrow transactor seam keeps the begin/produce/end choreography unit-testable with a fake session.
  • source/statemachine: DriveTx runs Fire + emit-as-produce + persist + offset-commit inside one transaction via a TxSink, returning Manual so the engine does not re-settle. Drive remains the at-least-once fallback; a broker abort after a durable save is covered by state-version dedup.

Why

Exactly-once consume-transition-emit is the headline guarantee of the source/statemachine bridge. On Kafka that requires the transactional producer fencing the offset commit, which only the adapter can wire — so the core seam stays vendor-neutral and the Kafka adapter owns the choreography. The capability is additive: existing at-least-once consumers are untouched.

Relates to: follows up the source Tier-1 PR (#131).

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • Tests added (fake-session ordering unit tests + RedPanda integration EOS round-trip asserting no duplicates across an abort); source 94.5% / kafka 89.9% / statemachine 88.7%
  • Public API documented (godoc + adapters & with-state docs + changelog notes)
  • mage check runs in CI across the workspace and the GOWORK=off source destinations

Flesh out the source Transactional capability into a working EOS path so a
consume-process-produce cycle commits the produced records and the consumed
offset atomically on Kafka.

Core seam (source): Transactional.Begin now fences a consumed Message together
with the records produced through a new backend-neutral Tx handle
(Tx.Produce, ProducedRecord), so emit and ack are one atomic unit. No vendor
type crosses the seam.

source/kafka: WithTransactional(id) builds a GroupTransactSession with
read-committed isolation and no auto-commit; Begin produces through the session,
marks the consumed offset on success only, and ends with TryCommit/TryAbort. A
narrow transactor seam keeps the begin/produce/end choreography unit-testable
with a fake session.

source/statemachine: DriveTx runs Fire + emit-as-produce + persist + offset
commit inside one transaction via a TxSink, returning Manual so the engine does
not re-settle; Drive remains the at-least-once fallback. A broker abort after a
durable save is covered by state-version dedup.

Tests: fake transact-session unit tests asserting begin/produce/commit/abort
ordering and the bridge's transactional path; an integration test (RedPanda)
doing a consume-produce EOS round-trip and asserting no duplicate output across
an aborted transaction. Docs: an Exactly-once (Kafka EOS) section in source
adapters and with-state, plus changelog notes.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple joshua-temple merged commit 9700788 into main Jun 4, 2026
117 checks passed
@joshua-temple joshua-temple deleted the feat/source-eos branch June 4, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant