[Bugfix][TE] Respect remote_accessible in RDMA verbs MR registration

[RuiqingFeng]

Description

RdmaTransport::registerLocalMemoryInternal() previously discarded the remote_accessible argument ((void)remote_accessible;) and unconditionally granted remote read/write permissions (IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE). This violated the API contract and exposed local-only memory to remote peers.

This PR fixes the issue by:

1. Setting IBV_ACCESS_REMOTE_* permissions only when remote_accessible is true. When false, registers with IBV_ACCESS_LOCAL_WRITE only and leaves the rkey vector empty.
2. Filtering out RDMA buffers with empty rkey vectors during metadata encoding so local-only memory is not advertised across the network.
3. Adding bounds checks in selectPeerDevice() for buffer_id and device_id against the rkey array size to prevent out-of-bounds indexing on local-only memory.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Added unit tests in rdma_context_reprobe_test.cpp and transfer_metadata_test.cpp:

1. RdmaMemoryRegistrationPolicyTest.LocalOnlyBufferHasNoPublishedRkey: Verifies that registering local memory with remote_accessible=false keeps its rkey empty.
2. TransferMetadataPublicationTest.OmitsBufferWithoutRkey: Verifies that buffers with empty rkeys are excluded when encoding P2P segment descriptions for network transfer.

Test commands:

cmake --build build-ci-local --target rdma_context_reprobe_test transfer_metadata_test
./build-ci-local/mooncake-transfer-engine/tests/rdma_context_reprobe_test --gtest_filter='*RegistrationPolicy*'
./build-ci-local/mooncake-transfer-engine/tests/transfer_metadata_test --gtest_filter='TransferMetadataPublicationTest.*'

[staryxchen]

1. Protocol coverage — Metadata filtering only skips empty rkey when protocol == "rdma", while barex / efa / cxi share the same encode path but are not filtered. EFA/CXI also still ignore remote_accessible. Is this PR intentionally RDMA-only?
2. PR title — The title looks concatenated (...registrationFix/rdma...); worth cleaning up before merge.

[RuiqingFeng]

1. Protocol coverage — Metadata filtering only skips empty rkey when protocol == "rdma", while barex / efa / cxi share the same encode path but are not filtered. EFA/CXI also still ignore remote_accessible. Is this PR intentionally RDMA-only?
2. PR title — The title looks concatenated (...registrationFix/rdma...); worth cleaning up before merge.

Thanks for the review!

Protocol coverage——Yes, this PR is intentionally scoped to RDMA verbs MR registration, as it addresses an immediate requirement in Broadcom RDMA NIC environment. I will follow up with EFA / CXI / barex support in a separate PR.
PR title——Updated. Cleaned up the title to [Bugfix][TE] Respect remote_accessible for RDMA verbs MR registration.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 71.42857% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ngine/src/transport/rdma_transport/worker_pool.cpp	0.00%	17 Missing ⚠️
...ne/src/transport/rdma_transport/rdma_transport.cpp	40.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[staryxchen]

LGTM
@alogfans pls double chck

[alogfans]

Overall LGTM. However, when the peer side rejects this memory region, it should print to log clearly about this to avoid slient bugs.

[RuiqingFeng]

Overall LGTM. However, when the peer side rejects this memory region, it should print to log clearly about this to avoid slient bugs.

Added an explicit error log [RDMA] No rkey for MR access: seg=... addr=... len=... in selectPeerDevice() when a peer memory region lacks remote access. Thanks!

[alogfans]

Instead of omitting local-only RDMA buffers from the segment metadata, could we keep their address ranges registered and explicitly mark them, while leaving the rkey unavailable? This allows the peer to reject such regions deterministically and distinguish them from missing or stale metadata, reducing the risk of accidental fallback or misuse.

[RuiqingFeng]

Instead of omitting local-only RDMA buffers from the segment metadata, could we keep their address ranges registered and explicitly mark them, while leaving the rkey unavailable? This allows the peer to reject such regions deterministically and distinguish them from missing or stale metadata, reducing the risk of accidental fallback or misuse.

Agreed! Updated to preserve local-only buffers in metadata with an empty rkey list ([]). Peers can now match the address range and fail with error No rkey for MR access without unnecessary reloads. Good idea, thanks!