Skip to content

Fix RDMA server core dump on concurrent access to Socket::_read_buf - #3505

Open
chenBright wants to merge 1 commit into
apache:masterfrom
chenBright:fix_rdma_read_buf
Open

Fix RDMA server core dump on concurrent access to Socket::_read_buf#3505
chenBright wants to merge 1 commit into
apache:masterfrom
chenBright:fix_rdma_read_buf

Conversation

@chenBright

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: resolve #3479

Problem Summary:

What is changed and the side effects?

Changed:

  1. One input stream, one buffer. New InputMessengerProcessor holds the
    state of a single input stream: its butil::IOPortal and the message-size statistics
    that size the next read. Socket::_read_buf, _last_msg_size and _avg_msg_size
    move into it, and InputMessenger::CutInputMessage() / ProcessNewMessage()
    become its methods. A Socket owns one for its fd, a RdmaEndpoint owns one for
    its QP. Socket::DoRead() now takes the destination IOPortal* instead of always
    filling _read_buf.

  2. The server stops parsing the TCP fd once RDMA is on (scenario 1).
    RdmaTransport::Init() installs RdmaEndpoint::OnNewDataFromTcp for both sides
    rather than only for the client; it dispatches on Socket::CreatedByConnect().
    In ESTABLISHED the fd is only probed for EOF, never parsed. This is needed on
    top of the buffer split because preferred_index and parsing_context are
    still per-Socket, so two streams must not parse at once.

  3. CQ events start after the handshake, not during it (scenarios 2).
    RdmaEndpoint::StartCqEvents() is split out of DoAllocateResources(). The
    server calls it from OnNewDataFromTcpAtServer() once OnNewMessages() has
    returned, the client from ProcessHandshakeAtClient(). No CQE is lost by
    deferring: BringUpQp() fills the RQ and DoAllocateResources() arms both CQs
    before the QP reaches RTS, and adding an already readable fd to an edge-triggered
    epoll reports it immediately.

  4. The handshake ACK no longer swallows what follows it (scenario 3). Phase
    2 consumes exactly HELLO_ACK_LEN bytes and leaves the rest to the real
    protocol when the connection falls back to TCP. When RDMA is on the fd is not an
    RPC channel any more, so bytes on it stay a protocol error, and a guard at the
    top of ExecuteServerHandshake() turns them away once the endpoint has left the
    handshake.

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RDMA server: race condition on _read_buf between PollCq and OnNewMessages causes core dump

1 participant