[fix][client] Defer op cmd release to the write event loop on send timeout - #26456
[fix][client] Defer op cmd release to the write event loop on send timeout#26456nodece wants to merge 1 commit into
Conversation
…meout Motivation The send-timeout path (failPendingMessages with cnx() == null) released op.cmd and recycled the op inline on the timer thread while the message frame may still have been queued for writing on a connection event loop: the write callback sits behind a reconnect/disconnect storm, or the buffers are already in the channel outbound buffer. Releasing the buffers from the wrong thread could return them to the pool while the in-flight write is still reading them, letting new batches overwrite the frame content and corrupting the wire bytes (broken checksum / lost frame sync on the broker). The existing cnx != null branch already deferred this cleanup to the event loop; the null (reconnect-window) branch did not. Modifications - OpSendMsg tracks the event loop each cmd was last handed to for writing (writeEventLoop), and cmd is now volatile so stale callbacks can never observe a released cmd. - releaseOpCmdAndRecycle defers the cmd release and op recycle to that event loop (serialized after the in-flight write), with an inline fallback when the loop is shutting down. releaseOpCmd clears op.cmd before releasing so a stale callback always fails its guard. - WriteInEventLoopCallback now skips stale writes (op re-sent on another connection or already disposed) and only drops the reference it took, instead of writing a released buffer or mutating a recycled op. - failPendingMessages fails inline when the connection event loop rejects the deferred task (shutting down during reconnect churn) instead of leaving the pending messages queued. - ackReceived / recoverChecksumError / recoverNotAllowedError now go through the same write-loop-aware release.
b20890b to
6b94c81
Compare
|
Sharing the end-to-end validation I ran against the exact head of this PR ( Setup verification: built from a clean worktree at the PR head (bit-exact match with the PR ref, zero uncommitted changes, clean rebuild with reproducible jar hashes); bytecode-checked that the jar contains the Harness: standalone broker + proxy as separate docker containers, producer through the proxy with Results (freshly recreated containers for each phase):
No throughput difference vs. master under a normal direct-memory configuration. |
Motivation
The send-timeout path (
failPendingMessageswithcnx() == null) releasedop.cmdand recycled the op inline on the timer thread while the message frame may still have been queued for writing on a connection event loop: the write callback sits behind a reconnect/disconnect storm, or the buffers are already in the channel outbound buffer. Releasing the buffers from the wrong thread could return them to the pool while the in-flight write is still reading them, letting new batches overwrite the frame content and corrupting the wire bytes (broken checksum / lost frame sync on the broker). The existingcnx != nullbranch already deferred this cleanup to the event loop; the null (reconnect-window) branch did not.Modifications
OpSendMsgtracks the event loop each cmd was last handed to for writing (writeEventLoop), andcmdis nowvolatileso stale callbacks can never observe a released cmd.releaseOpCmdAndRecycledefers the cmd release and op recycle to that event loop (serialized after the in-flight write), with an inline fallback when the loop is shutting down.releaseOpCmdclearsop.cmdbefore releasing so a stale callback always fails its guard.WriteInEventLoopCallbacknow skips stale writes (op re-sent on another connection or already disposed) and only drops the reference it took, instead of writing a released buffer or mutating a recycled op.failPendingMessagesfails inline when the connection event loop rejects the deferred task (shutting down during reconnect churn) instead of leaving the pending messages queued.ackReceived/recoverChecksumError/recoverNotAllowedErrornow go through the same write-loop-aware release.Verifications
ProducerImplTest9/9 passing (includes 4 regression tests for the timeout + in-flight-write race).checkstyleMain/checkstyleTestpassing.