drpcstream: mutex-free sendWindow flow-control primitive#84
Open
suj-krishnan wants to merge 2 commits into
Open
drpcstream: mutex-free sendWindow flow-control primitive#84suj-krishnan wants to merge 2 commits into
suj-krishnan wants to merge 2 commits into
Conversation
Rework sendWindow into a lock-free credit balance in the style of grpc-go's writeQuota: an atomic balance, a cap-1 buffered channel for grant wakeups, and a caller-supplied done channel for termination. There is at most one acquirer (serialized by the stream write lock) and one granter (the connection reader), so no mutex is needed. acquire debits credit atomically and, if that leaves a deficit, waits on the channel; grant repays the balance and edge-signals the channel only when the deficit clears. A cap-1 non-blocking send makes the wakeup lost-wakeup-free without a condition variable. Termination is delegated to the done channel -- the stream's send signal -- so the window holds no error or closed state. terminate no longer calls into the window; setting sigs.send both wakes a blocked acquire and supplies its error (sigs.send.Err()), which matches the error a send parked in WriteFrame or a later send reads. acquire is a pure credit gate: with credit available it returns without consulting done, so a send racing termination can be admitted. That is harmless -- rawWriteLocked holds the stream write lock, so the frame is ordered before the terminal frame, and the write path drops it when the send signal is already set (companion change). Only a blocked acquire observes done, which is what wakes a parked sender on termination. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
WriteFrame consulted its cancel signal only when it had to park on a full buffer; on the common non-blocking path (buffer has room) it appended unconditionally. A data frame whose send was canceled or the stream terminated -- sigs.send set after the write path's own check but before WriteFrame -- would therefore still reach the wire. Check cancel at the top of the loop, before appending. IsSet is an atomic read and does not force the signal's lazy channel allocation, so the fast path stays allocation-free. Terminal frames pass a nil cancel and are never affected. This lets the send window stay a pure credit gate: acquire need not be absorbing on its fast path, because the write path now drops a data frame whose stream has terminated -- the drpc analogue of grpc-go's loopyWriter checking live stream membership before writing. It also closes the pre-existing race (independent of flow control) where a data frame could be appended just after the send signal was set. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
suj-krishnan
marked this pull request as ready for review
July 23, 2026 10:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the per-stream send-side flow-control primitive (
sendWindow) into a lock-free credit balance, and adds the write-path liveness check that lets it stay simple.sendWindowbecomes a lock-free credit balance: an atomic balance, a size 1 buffered channel for grant wakeups, and a caller-supplieddonechannel for termination. There is at most one acquirer (serialized by the stream write lock) and one granter (the connection reader).Termination is delegated to the
donechannel — the stream's send signal — so the window holds no error or closed state.terminateno longer calls into the window; settingsigs.sendboth wakes a blockedacquireand supplies its error viasigs.send.Err().WriteFramepreviously consulted its cancel signal only when parking on a full buffer; on the common non-blocking path it appended unconditionally. It now checkscancel.IsSet()before appending, so a data frame whose stream hasterminated is dropped at the write layer.
IsSetis a plain atomic read, so the fast path stays allocation-free; terminal frames pass a nil cancel and are unaffected.