Fix race when falling back from RDMA to TCP#3406
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a concurrency correctness issue in the RDMA transport handshake/fallback path by making the RDMA handshake state thread-safe and ensuring the TCP fallback state is published in a safe order, preventing TCP event handling from observing a partially published fallback.
Changes:
- Change
rdma::RdmaEndpoint::_statefrom a plain enum tobutil::atomic<State>to eliminate a C++ data race across handshake/TCP/completion paths. - Reorder fallback publication in
RdmaConnect::StartConnect()to setRdmaTransport::_rdma_state = RDMA_OFFbefore publishingRdmaEndpoint::_state = FALLBACK_TCP. - Use a relaxed atomic load for the completion-path invariant check in
RdmaEndpoint::HandleCompletion()to avoid adding unnecessary ordering overhead on a hot path.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/brpc/rdma/rdma_endpoint.h | Makes handshake state (_state) atomic to remove data races between concurrent RDMA/TCP/completion execution contexts. |
| src/brpc/rdma/rdma_endpoint.cpp | Fixes fallback publication ordering and updates the completion-path state check to use an explicit relaxed atomic load. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
chenBright
requested changes
Jul 25, 2026
Make the handshake state atomic and publish RDMA_OFF before FALLBACK_TCP. This prevents TCP event handling from observing a partially published fallback state. Use a relaxed load for the completion-path check because it does not consume related data. Signed-off-by: Lijin Xiong <legion.xiong@gmail.com>
legionxiong
force-pushed
the
fix-rdma-fallback-race
branch
from
July 27, 2026 01:56
735f4df to
6e5b15a
Compare
Contributor
Author
Done. |
Member
|
LGTM. Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #3405
Problem Summary:
RdmaEndpoint::_stateis accessed concurrently by the RDMA handshake,TCP event handling, and completion handling paths, but was not atomic.
This causes a C++ data race.
During TCP fallback,
RdmaTransport::_rdma_stateandRdmaEndpoint::_statealso form a logical state transition. A TCP eventmust not observe
FALLBACK_TCPbefore the precedingRDMA_OFFupdate isvisible, otherwise it may start normal TCP message processing with a
partially published transport state.
What is changed and the side effects?
Changed:
RdmaEndpoint::_statetobutil::atomic<State>.RdmaTransport::_rdma_statetoRDMA_OFFbefore publishingRdmaEndpoint::_stateasFALLBACK_TCP.FALLBACK_TCPwithmemory_order_release.memory_order_acquireinOnNewDataFromTcp(). When it observesFALLBACK_TCP, this acquire loadpairs with the release store and makes the preceding
RDMA_OFFupdatevisible before normal TCP message processing starts.
memory_order_relaxedfor all other handshake-state transitions andobservations because they do not publish or consume related data.
relaxed ordering is sufficient for the other accesses.
Side effects:
Performance effects:
The acquire load is limited to the TCP event path, while release stores
are only used when falling back to TCP. Other handshake-state accesses,
including the completion hot path, use relaxed ordering to avoid
unnecessary synchronization overhead.
Breaking backward compatibility:
None. This change does not modify any public API, wire protocol, or
externally visible configuration.