Fix RDMA server core dump on concurrent access to Socket::_read_buf - #3505
Open
chenBright wants to merge 1 commit into
Open
Fix RDMA server core dump on concurrent access to Socket::_read_buf#3505chenBright wants to merge 1 commit into
chenBright wants to merge 1 commit into
Conversation
chenBright
force-pushed
the
fix_rdma_read_buf
branch
from
August 30, 2026 06:55
5478b96 to
c4c0afb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: resolve #3479
Problem Summary:
What is changed and the side effects?
Changed:
One input stream, one buffer. New
InputMessengerProcessorholds thestate of a single input stream: its
butil::IOPortaland the message-size statisticsthat size the next read.
Socket::_read_buf,_last_msg_sizeand_avg_msg_sizemove into it, and
InputMessenger::CutInputMessage()/ProcessNewMessage()become its methods. A
Socketowns one for its fd, aRdmaEndpointowns one forits QP.
Socket::DoRead()now takes the destinationIOPortal*instead of alwaysfilling
_read_buf.The server stops parsing the TCP fd once RDMA is on (scenario 1).
RdmaTransport::Init()installsRdmaEndpoint::OnNewDataFromTcpfor both sidesrather than only for the client; it dispatches on
Socket::CreatedByConnect().In
ESTABLISHEDthe fd is only probed for EOF, never parsed. This is needed ontop of the buffer split because
preferred_indexandparsing_contextarestill per-Socket, so two streams must not parse at once.
CQ events start after the handshake, not during it (scenarios 2).
RdmaEndpoint::StartCqEvents()is split out ofDoAllocateResources(). Theserver calls it from
OnNewDataFromTcpAtServer()onceOnNewMessages()hasreturned, the client from
ProcessHandshakeAtClient(). No CQE is lost bydeferring:
BringUpQp()fills the RQ andDoAllocateResources()arms both CQsbefore the QP reaches RTS, and adding an already readable fd to an edge-triggered
epoll reports it immediately.
The handshake ACK no longer swallows what follows it (scenario 3). Phase
2 consumes exactly
HELLO_ACK_LENbytes and leaves the rest to the realprotocol when the connection falls back to TCP. When RDMA is on the fd is not an
RPC channel any more, so bytes on it stay a protocol error, and a guard at the
top of
ExecuteServerHandshake()turns them away once the endpoint has left thehandshake.
Side effects:
Performance effects:
Breaking backward compatibility:
Check List: