Fix rdma handshake failing the socket instead of falling back to TCP

[chenBright]

What problem does this PR solve?

Issue Number: resolve #3416

Problem Summary:

During the RDMA handshake, RdmaEndpoint::AllocateResources() arms the send/recv
CQs through ReqNotifyCq(). If ibv_req_notify_cq() fails, ReqNotifyCq() calls
_socket->SetFailed() immediately. However, both handshake paths
(ProcessHandshakeAtClient() and ExecuteServerHandshake()) treat
AllocateResources() < 0 as a recoverable error: they only turn RDMA off
(RDMA_OFF) and move the endpoint to FALLBACK_TCP, expecting the connection to
keep working over plain TCP. Since the socket has already been failed, the
connection cannot carry TCP any more, so a single CQ arm failure turns a
"graceful degradation to TCP" into a "broken connection".

While fixing this, a second issue was found on the same path. RdmaConnect::Run()
reports the connect result with _done(errno, _data), i.e. it uses the current
errno as the connect error code. The client-side fallback branch returns right
after AllocateResources() fails, leaving errno set to the allocation error,
while every other path in ProcessHandshakeAtClient() ends with errno = 0. As a
result the connection is reported as failed to the upper layer even after the
endpoint has entered FALLBACK_TCP. Fixing only ReqNotifyCq() is therefore not
enough to make the fallback work.

What is changed and the side effects?

Changed:

Side effects:

• Performance effects:

• Breaking backward compatibility:

---

Check List:

• Please make sure your changes are compilable.
• When providing us with a new feature, it is best to add related tests.
• Please follow Contributor Covenant Code of Conduct.

[Copilot]

Pull request overview

This PR fixes RDMA handshake error handling so that certain RDMA initialization failures (notably CQ arming via ibv_req_notify_cq) correctly trigger a graceful fallback to plain TCP instead of prematurely failing the underlying socket, and ensures the connect callback reports success after fallback by clearing errno on the recoverable path.

Changes:

• Add a fatal_on_error switch to RdmaEndpoint::ReqNotifyCq so handshake-time CQ arming failures don’t fail the TCP socket, while runtime re-arm failures remain fatal.
• Clear errno on the client handshake’s recoverable AllocateResources() failure path to avoid reporting a failed connect after switching to TCP fallback.
• Add unit tests covering client/server fallback-to-TCP behavior when resource allocation is forced to fail in UT mode.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/brpc/rdma/rdma_endpoint.cpp	Adjusts handshake fallback behavior (no socket failure on CQ arm failure) and normalizes errno on recoverable fallback.
src/brpc/rdma/rdma_endpoint.h	Updates ReqNotifyCq signature to include fatal_on_error for handshake vs runtime behavior.
test/brpc_rdma_unittest.cpp	Adds UT coverage for client/server TCP fallback when RDMA resource allocation fails.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/brpc/rdma/rdma_endpoint.cpp:59

• g_fail_resource_alloc_for_test is a plain global bool that is written by tests while other server/client bthreads may already be running, and read from DoAllocateResources() on handshake threads. This introduces a C++ data race (no synchronization between the write and reads). Make this flag atomic (or otherwise synchronized) so the UT injection is thread-safe.

// Only for UT: force AllocateResources() to fail, so that the "fallback to TCP" path
// of the handshake can be tested without a real RDMA device.
bool g_fail_resource_alloc_for_test = false;

test/brpc_rdma_unittest.cpp:1931

• ResourceAllocFailGuard toggles rdma::g_fail_resource_alloc_for_test, which is read by handshake threads. If this flag remains a plain bool, the test introduces a data race (write in the test thread vs reads in server/client bthreads). Prefer making the flag atomic and using .load()/.store() here.

class ResourceAllocFailGuard {
public:
    explicit ResourceAllocFailGuard(bool v)
        : _saved(rdma::g_fail_resource_alloc_for_test) {
        rdma::g_fail_resource_alloc_for_test = v;
    }
    ~ResourceAllocFailGuard() {
        rdma::g_fail_resource_alloc_for_test = _saved;
    }

[yanglimingcn]

LGTM