KAFKA-20785: Return PRODUCER_FENCED when EndTxn commit races with coordinator-side abort - #23204
Conversation
…ith coordinator-side abort Under transactions V2, when the transaction coordinator aborts an open transaction on its own (e.g. after transaction.timeout.ms elapses), the abort bumps the producer epoch. A commit that was already in flight when the abort completed arrives with the pre-abort epoch, which matches the V2 EndTxn retry condition (stored epoch == request epoch + 1), so it passes the epoch check and fails the state check instead: the coordinator returns INVALID_TXN_STATE, which producer clients treat as unconditionally fatal. Kafka Streams cannot recover from it and the affected StreamThread dies, even though the transaction was fully rolled back server-side and the commit never took effect. Under transactions V1 the same race fails the strict epoch check and returns PRODUCER_FENCED, which applications such as Kafka Streams handle by rebalancing; the fatal outcome is V2-only. Return the recoverable INVALID_PRODUCER_EPOCH instead of INVALID_TXN_STATE for a commit that arrives in COMPLETE_ABORT at a retry epoch (epoch-bump or producer-id-overflow variant). Only a coordinator-initiated abort can produce this combination, since a client retrying its own EndTxn always carries the operation it originally sent. Producer clients map INVALID_PRODUCER_EPOCH on the EndTxn response to ProducerFencedException, which Kafka Streams recovers from by rebalancing, restoring the V1 behavior for this race. A commit in COMPLETE_ABORT at the current epoch still returns INVALID_TXN_STATE, as that indicates a client-side bug rather than this race. This is the EndTxn-path analogue of KAFKA-19690, which made the same correction on the Produce path.
github-actions
Bot
added
triage
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
jolshan
added
transactions
| transaction on its own (e.g. when it exceeds transaction.timeout.ms), bumping the epoch without the | ||
| producer's knowledge. A commit that was already in flight when such an abort completed arrives with the | ||
| pre-abort epoch and is indistinguishable from a retry. The commit is guaranteed not to have taken effect, | ||
| so the recoverable INVALID_PRODUCER_EPOCH is returned, matching the transaction V1 behavior for this race, |
There was a problem hiding this comment.
Just wanted to confirm the reasoning of IPE vs PF here -- it's because in TV1 we return IPE?
Is the error returned from a different part of the code (ie, in the log code right before the write happens)? I didn't see IPE used in this file for TV1
There was a problem hiding this comment.
Thanks. That's my bad wording, V1 does not return IPE, "matching the transaction V1 behavior" is supposed to mean V1 returns PF, which gets handled the same as IPE by client.
I picked IPE here to (a) tell the same story as the produce path for the same event, and (b) keep the semantic distinction: PF implies a newer producer took the transactional.id, which isn't what happened in a timeout abort. On the EndTxn response path the client already folds IPE into ProducerFencedException (EndTxnHandler), so the two codes behave identically for all TV2 clients. I can change it to PF if you'd prefer that.
There was a problem hiding this comment.
Yeah, I think there was an informal standard that only produce requests returned IPE, and all transactional requests return PF. I don't think it matters a ton in the grand scheme of things, but maybe its easier to keep things consistent.
My understanding for PF, is just that the producer was fenced, which in the server-side abort scenario, it is 😄 , but I see how the comment for the error could be confusing.
There was a problem hiding this comment.
I see, that makes sense. I've changed returned error from IPE to PF
…ommit/abort race The previous comments said the error choice matches transaction V1 behavior, which misread as V1 returning INVALID_PRODUCER_EPOCH here. V1 actually fails this race at its strict epoch check with PRODUCER_FENCED; what the fix restores is the recoverable outcome, not the error code. State that explicitly in the state-table note, along with why INVALID_PRODUCER_EPOCH is used instead of PRODUCER_FENCED: no newer producer exists (the epoch is merely stale), it mirrors KAFKA-19690's produce-path fix in UnifiedLog, and producer clients treat the two errors identically on EndTxn responses. The branch comment now defers to the state-table note instead of repeating it.
…NCED Transactional requests conventionally return PRODUCER_FENCED on epoch mismatches, and PRODUCER_FENCED is also what transactions V1's strict epoch check returns for this same race; INVALID_PRODUCER_EPOCH is the produce-path convention (KAFKA-19690). Client behavior is unchanged: the producer client already folds INVALID_PRODUCER_EPOCH into ProducerFencedException on EndTxn responses, so the two codes are handled identically.
suzhiking
changed the title
2 participants
Under transactions V2, when the transaction coordinator aborts an open
transaction on its own (e.g. after
transaction.timeout.mselapses),the abort bumps the producer epoch. A commit that was already in flight
when the abort completed arrives with the pre-abort epoch, which matches
the V2 EndTxn retry condition (stored epoch == request epoch + 1). It
therefore passes the epoch check and fails the state check instead: the
coordinator logs "state is COMPLETE_ABORT, but received transaction
marker result to send: COMMIT" and returns
INVALID_TXN_STATE, whichproducer clients treat as unconditionally fatal. Kafka Streams cannot
recover from it and the affected StreamThread dies, even though the
transaction was fully rolled back server-side and the commit never took
effect.
Under transactions V1 the same race fails the strict epoch check and
returns
PRODUCER_FENCED, which applications such as Kafka Streamshandle gracefully by rebalancing; the fatal outcome is V2-only.
This change returns the recoverable
PRODUCER_FENCEDinstead ofINVALID_TXN_STATEwhen a commit arrives inCOMPLETE_ABORTat a retryepoch (both the epoch-bump and the producer-id-overflow variants). Only
a coordinator-initiated abort can produce this combination: a client
retrying its own EndTxn always carries the operation it originally sent,
so a COMMIT at the pre-abort epoch means the producer never requested
the abort.
PRODUCER_FENCEDmatches what V1 returns for this race andfollows the existing convention that transactional requests return
PRODUCER_FENCEDon epoch mismatches while produce requests returnINVALID_PRODUCER_EPOCH; the producer client foldsINVALID_PRODUCER_EPOCHintoProducerFencedExceptionon EndTxnresponses anyway, so the two codes behave identically for clients —
Kafka Streams recovers by rebalancing, restoring the V1 behavior for
this race. A commit in
COMPLETE_ABORTat the current epoch stillreturns
INVALID_TXN_STATE, since that indicates a client-side bugrather than this race. Only transactions-V2 clients reach this code
path, so no protocol or version-gating changes are needed.
This is the EndTxn-path analogue of KAFKA-19690 (#20534), which made the
same correction on the Produce path for producers that were mid-produce
when the timeout abort landed, returning
INVALID_PRODUCER_EPOCHperthe produce-path convention.
Reviewers: Justine Olshan jolshan@confluent.io