You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Design Discussion: RDMA Library Selection and a Multi-Backend Transport Framework for Curvine
#2299
There are effectively only 3 API families among general-purpose user-space RDMA communication libraries: libibverbs (+ rdma_cm), libfabric (OFI), and UCX. Everything else is a socket compatibility layer, a language binding, or a specialized protocol stack.
Given Curvine's traffic characteristics (large, sequential, client→Worker-dominated block I/O), we recommend libfabric as the primary backend, UCX as an optional backend for GPU/HPC environments, and raw verbs as a performance baseline / private-deployment fallback; keep the control plane on TCP and land RDMA in the data plane first.
Zero abstraction overhead, full feature set (QP/CQ/SRQ/XRC/DC/atomics/DV), best possible latency
You must implement the CM event loop, QP state machine, GID index selection, and reconnect logic yourself; heavy C boilerplate; Rust bindings are essentially DIY
One fi_* API covering verbs/RoCE, rxm, efa, psm2/3, cxi, tcp/shm via providers; production-verified in storage (DAOS/CaRT, AWS EFA); tagged messaging (fi_tagged) maps naturally to RPC multiplexing
Many concepts (domain/av/ep/cq/mr/cntr) and a steep learning curve; large-message bandwidth close to raw verbs, but the rxm/tag path adds small-message latency; Rust bindings are immature
Multi-hardware, multi-generation NICs, storage systems
UCX
Transport × protocol framework
Performance ceiling (the foundation of NCCL/MPI); transports (rc/dc/ud/tcp/shm/gdr/cuda) separated from protocols (eager/rndv/tag/stream/am), mature rendezvous; GPUDirect RDMA; automatic multi-NIC selection
Heavy dependencies and configuration (dozens of UCX_TLS-style knobs), historical version-compatibility baggage; overkill for pure block storage; huge API surface
GPU/AI clusters, HPC environments already on the UCX stack
rsocket (librdmacm)
Socket compatibility layer
Cheapest integration; direct mapping of socket semantics
No raw one-sided semantics, limited performance, low maintenance activity
PoC only, never production
MPI / Mercury / Margo
High-level communication frameworks
Mature ecosystem
Symmetric/collective programming models do not fit client-server storage
Not applicable
NVMe-oF/RDMA, NFS-RDMA, SMB Direct
Specialized wire protocols
Standardized, with ready-made targets/initiators
Semantics do not match a filesystem; suitable only for specialized paths
Curvine's SPDK backend already uses NVMe-oF
In one sentence: general-purpose libraries ≈ 3 families (verbs, libfabric, UCX), plus 3 classes of wrappers/specialized protocols; only the first two-and-a-half are credible long-term backends for Curvine.
2. Curvine's Current State and Constraints
Curvine's network layer is a home-grown ORPC: a binary RPC format with a 22-byte fixed header + protobuf header + raw data segment (crates/core/rpc/src/message/rpc_message.rs:29). RpcFrame hard-codes TcpStream (crates/core/rpc/src/handler/rpc_frame.rs:38), and connections are established over tokio TCP (crates/core/rpc/src/client/raw_client.rs:33). Block I/O follows an Open→Running→Complete three-phase streaming RPC (crates/client/curvine-client-core/src/block/block_client.rs), with a 16 MiB per-frame limit. SPDK is used only for the local block storage backend (crates/adapters/curvine-storage-spdk/src/spdk_bdev.rs:688) and is fully decoupled from the network layer.
This has several key implications for the design:
The data plane is where RDMA pays off: block read/write is "large, sequential, client→Worker-dominated" traffic — the optimal shape for one-sided RDMA WRITE/READ. Metadata RPCs are small and irregular; TCP's tens-of-microseconds latency is not the bottleneck there.
DataSlice::MemSlice already exists (crates/core/curvine-io/src/block_io.rs:8; BlockIO already handles MemSlice), so data landed in registered memory by RDMA can be fed directly into the existing BlockIO::write_region — exactly the same integration pattern the SPDK path already uses.
A "split plane" allows incremental delivery: keep the control plane on TCP and add RDMA to the data plane, avoiding changes to the mature-but-TCP-coupled RpcFrame/splice/sendfile code.
Curvine already has a fault-injection runtime that can be reused to test RDMA failure paths such as QP errors and timeouts.
3. Recommendation
Use libfabric as the primary backend, UCX as an optional backend for GPU/HPC scenarios, and raw verbs as a performance baseline / private-deployment fallback. Keep rsocket out of production.
Rationale:
The "multi-library adaptation" problem is largely absorbed by libfabric's provider model: verbs (RoCE/IB/iWARP), rxm, efa (AWS cloud), psm (Intel), cxi (Slingshot) are all covered by one API. Adapting libfabric once gets you most vendor NICs, instead of maintaining one code path per vendor.
The programming model fits: the 64-bit tag of fi_tagged maps directly to ORPC's req_id; fi_write/fi_read provide one-sided RMA for zero-copy block data; the tcp/shm providers let the same code run its tests in RDMA-free environments.
Verified by the storage industry: DAOS's CaRT abstraction layer and AWS EFA storage traffic both run on libfabric, providing production maturity backing.
Lowest FFI maintenance cost: only one libfabric-sys needs to be maintained; UCX's API surface and configuration matrix is far larger, and binding maintenance costs significantly more.
When to flip: pure NVIDIA RoCE + GPU direct-memory access → UCX; single-vendor private deployment requiring microsecond-class small messages → raw verbs. The value of the framework is precisely that this choice becomes a configuration item, not a rewrite.
4. Framework Design
4.1 Layering and crate layout (following the existing adapters pattern)
crates/core/curvine-rdma: traits, registry, memory pools, the CQ↔tokio bridge, and the backend conformance test suite (always compiles; on macOS only the TCP passthrough backend is present).
crates/adapters/rdma-libfabric, rdma-verbs, rdma-ucx: feature-gated (rdma-libfabric / rdma-verbs / rdma-ucx), Linux-only, with build.rs probing via pkg-config. Missing dependencies fail at compile time only when the feature is enabled, and default builds are unaffected — the same pattern as the current spdk feature in curvine-storage-spdk.
4.2 Core abstraction: an operation-level API, not a WR-level API
Do not abstract at the "post_send/poll_cq" level (that is verbs dialect; libfabric and UCX have different semantics). Abstract operation semantics + capability declarations, and let each backend map them:
pubstructCapabilities{pubone_sided_write:bool,pubone_sided_read:bool,pubtagged_msg:bool,pubmax_msg_size:usize,pubmax_inline:usize,pubmulti_rail:bool,}pubtraitRdmaBackend:Send + Sync{fnname(&self) -> &'staticstr;fnprobe(&self) -> Result<Vec<DeviceInfo>>;// device / GID / RoCE-IB link layerfnopen(&self,dev:&DeviceInfo,cfg:&RdmaConfig) -> Result<Arc<dynRdmaContext>>;}pubtraitRdmaContext:Send + Sync{fncaps(&self) -> &Capabilities;fnalloc_mr(&self,len:usize,access:MrAccess) -> Result<Arc<dynMemoryRegion>>;fncreate_conn(&self,cfg:&ConnConfig) -> Result<Box<dynRdmaConn>>;fncq(&self) -> &dynCompletionQueue;// shared CQ, completions routed by wr_id}pubtraitRdmaConn:Send{/// Connection info is an opaque byte blob: verbs carries QPN/GID/PSN,/// libfabric carries its connect blob, UCX carries the worker address.fnlocal_handshake(&self) -> Vec<u8>;fnestablish(&mutself,peer:&[u8]) -> Result<()>;fnsend(&self,local:&dynMemoryRegion,range:Range<usize>,user:u64) -> Result<()>;fnrecv(&self,local:&dynMemoryRegion,range:Range<usize>,user:u64) -> Result<()>;fnwrite_remote(&self,local:&dynMemoryRegion,token:RemoteToken,user:u64) -> Result<()>;fnread_remote(&self,local:&dynMemoryRegion,token:RemoteToken,user:u64) -> Result<()>;}/// One-shot remote memory token: bound to a session nonce and invalidated after the transferpubstructRemoteToken{pubaddr:u64,pubkey:u64,publen:usize,pubnonce:u64}
Three design points:
Opaque handshake bytes (Vec<u8>): this is the core of "matching multiple libraries" — the abstraction layer does not care about connection semantics, only about transporting bytes. The control plane reuses the existing ORPC connection for bootstrap: add one RPC (additive messages in worker.proto, no changes to existing fields) that exchanges backend / caps / handshake blob / session_nonce, and the Worker replies with pool parameters (max_chunk, credits). Proto changes are additive, consistent with the repository's existing compatibility model.
Capability negotiation decides the data path: a backend with one_sided_write=false (e.g., the tcp provider or a future private SDK) automatically degrades to two-sided emulation; upper layers are unchanged.
Registry + compile-time features + optional dlopen: static registration by default; in a second phase, use libloading to provide a C-ABI vtable dynamic plugin mechanism so vendor SDKs (domestic NICs, smart NICs) can be integrated without code changes.
4.3 Data-plane rendezvous flow (matching the existing three-phase stream)
Write path (push; the client issues WRITEs):
BlockWriter BlockClient Worker
|-- WriteBlock RPC --->| |
| |-- OpenBlock ------------->| over the existing ORPC/TCP control plane
| |<-- AllocToken(rkey,addr)-| Worker carves a landing buffer from the MR pool
| |== RDMA WRITE_WITH_IMM ===>| zero-copy into MR; CQ completion -> persist queue
| |-- Complete -------------->| BlockIO::write_region(MemSlice)
| |<-- Ack -------------------|
Read path (push preferred; the Worker WRITEs into a client-registered receive buffer):
An alternative pull mode (client issues RDMA READ) suits very large sequential reads and receiver-paced flow; both modes share the same RemoteToken and are selected via read_mode = push|pull. One honest boundary: the disk→MR copy cannot be eliminated (unless going through NVMe-oF/GPU P2P). What RDMA eliminates is the TCP/IP stack, syscalls, and socket buffer copies — one copy on the client side and one on the Worker side, which is exactly the main benefit for block I/O.
Worker-to-worker replication (WriteReplicationBlock) reuses the same data channel, so replication traffic is also zero-copy into MR.
4.4 CQ ↔ tokio: a background completion pump
One shared CQ per io thread (no per-connection CQs, avoiding CQ explosion); wr_id = conn_id<<32 | slot routes completion events.
Adaptive polling: busy-poll while inflight > 0; when idle, sleep on eventfd + tokio::AsyncFd and wake on events. Completions notify async callers via tokio::sync::oneshot, coalescing wakes for batched completions.
Backpressure: inflight is capped by the number of buffers in the pool; when the pool is empty, callers await a semaphore — natural flow control.
4.5 Memory registration pool (MR pool)
Pre-register the whole pool with hugepages at startup (one MR covering the pool, carved by offset, with addr+offset carried in the token) to avoid per-operation reg_mr pinning and page-table setup costs.
Client side: LRU + refcount cache. FUSE read buffers allocated from the pool are WRITEd directly; external buffers are bounce-copied into the pool.
Worker side: landing buffers map directly to DataSlice::MemSlice and feed BlockIO::write_region.
Mind the memlock ulimit and madvise(MADV_HUGEPAGE); pool sizes are configurable.
The config structs live in crates/common/curvine-config/src/client_conf.rs / worker_conf.rs and map into ORPC-layer configuration, following the existing factory-function pattern of client_rpc_conf/worker_server_conf.
4.7 Failure handling, security, and observability
Degradation chain: capability mismatch during handshake → plain TCP; QP error → mark the connection dead, fail in-flight requests fast → exponential-backoff reconnect → circuit breaker falls back to TCP after repeated failures. Default fallback = tcp ensures RDMA is never a single point of availability.
Security is a hard requirement: an rkey is effectively a credential to write remote memory. Design: tokens are bound to a session nonce and invalidated after a single transfer (reclaimed via local invalidation where the backend supports it); the Worker only accepts RDMA CM connections from the management network segment; MRs use minimal permissions (REMOTE_WRITE for the write path, REMOTE_READ for the read path). On the RoCE side, isolation relies on PFC/ECN/ACL, and the operational prerequisites must be documented.
Observability: rdma_bytes_tx/rx, inflight, CQ depth, completion-latency histograms, and per-backend error counters, wired into the existing MetricsReport channel; add a CLI rdma probe for GID/MTU/provider preflight checks.
4.8 Test strategy
The conformance test suite lives in curvine-rdma: send/recv, one-sided write/read, token expiry, and error injection, with the same battery run against every backend — so "multi-library adaptation" is more than a claim.
Linux CI uses a soft-RoCE (rdma_rxe) loopback pair for end-to-end tests; feature builds run on dedicated runners with rdma-core/libfabric installed, leaving the default matrix unaffected.
Fault injection reuses Curvine's existing fault-injection runtime; the performance gate compares cv bench --transport rdma against the TCP baseline.
On macOS dev machines: features off; the abstraction layer + TCP passthrough still compile and pass tests.
At least one plugin-style backend integrated without recompilation
6. Risks and Boundaries
Rust bindings are a self-borne cost: libfabric-sys/ucx-sys must be maintained in-house; pin upstream versions and cover them with the CI matrix rather than chasing latest.
Small-message RPC is not worth it initially: the split-plane design keeps metadata on TCP; do not spread the complexity to low-yield paths just for "full-stack RDMA".
Memory cost: MRs are pinned physical pages; pooling + hugepages + explicit quotas are mandatory, not optional optimizations.
RoCE operational complexity is real: PFC deadlocks and ECN misconfiguration manifest as "intermittent hangs"; documentation and preflight tooling matter more than code.
Do not maintain both the libfabric verbs provider and a home-grown verbs backend simultaneously: before M4, keep only libfabric; start a raw-verbs backend only if profiling proves a gap.
7. Open Questions (feedback welcome)
What does the NIC/cloud-vendor distribution look like in your production environments? Is there an NVIDIA RoCE cluster or AWS EFA environment available to validate M1?
Should the control plane be upgraded to two-sided RDMA in a second phase (full-stack RDMA), or should the TCP control plane + RDMA data-plane split remain long-term?
Any field-tested recommendations for the default landing-buffer pool size, block size, and chunk_size defaults on the Worker side?
Should GPU direct access (GPUDirect, for LanceDB/vector-search scenarios) be pulled forward in priority to M2/M3?
Summary: Curvine should adopt libfabric as the primary RDMA backend (its provider mechanism absorbs most of the "multi-library adaptation" problem), leave UCX for GPU/HPC, and keep the control plane on TCP. The framework core is "operation-level traits + opaque handshake + capability negotiation + memory pools + CQ pump", delivered incrementally from M0 to M4, with every step able to fall back to TCP.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
RDMA Library Selection and a Multi-Backend Transport Framework for Curvine (Discussion Draft)
TL;DR
1. The RDMA Communication Library Landscape
fi_*API covering verbs/RoCE, rxm, efa, psm2/3, cxi, tcp/shm via providers; production-verified in storage (DAOS/CaRT, AWS EFA); tagged messaging (fi_tagged) maps naturally to RPC multiplexingUCX_TLS-style knobs), historical version-compatibility baggage; overkill for pure block storage; huge API surfaceIn one sentence: general-purpose libraries ≈ 3 families (verbs, libfabric, UCX), plus 3 classes of wrappers/specialized protocols; only the first two-and-a-half are credible long-term backends for Curvine.
2. Curvine's Current State and Constraints
Curvine's network layer is a home-grown ORPC: a binary RPC format with a 22-byte fixed header + protobuf header + raw data segment (
crates/core/rpc/src/message/rpc_message.rs:29).RpcFramehard-codesTcpStream(crates/core/rpc/src/handler/rpc_frame.rs:38), and connections are established over tokio TCP (crates/core/rpc/src/client/raw_client.rs:33). Block I/O follows an Open→Running→Complete three-phase streaming RPC (crates/client/curvine-client-core/src/block/block_client.rs), with a 16 MiB per-frame limit. SPDK is used only for the local block storage backend (crates/adapters/curvine-storage-spdk/src/spdk_bdev.rs:688) and is fully decoupled from the network layer.This has several key implications for the design:
DataSlice::MemSlicealready exists (crates/core/curvine-io/src/block_io.rs:8; BlockIO already handles MemSlice), so data landed in registered memory by RDMA can be fed directly into the existingBlockIO::write_region— exactly the same integration pattern the SPDK path already uses.RpcFrame/splice/sendfile code.3. Recommendation
Use libfabric as the primary backend, UCX as an optional backend for GPU/HPC scenarios, and raw verbs as a performance baseline / private-deployment fallback. Keep rsocket out of production.
Rationale:
fi_taggedmaps directly to ORPC'sreq_id;fi_write/fi_readprovide one-sided RMA for zero-copy block data; thetcp/shmproviders let the same code run its tests in RDMA-free environments.libfabric-sysneeds to be maintained; UCX's API surface and configuration matrix is far larger, and binding maintenance costs significantly more.When to flip: pure NVIDIA RoCE + GPU direct-memory access → UCX; single-vendor private deployment requiring microsecond-class small messages → raw verbs. The value of the framework is precisely that this choice becomes a configuration item, not a rewrite.
4. Framework Design
4.1 Layering and crate layout (following the existing adapters pattern)
crates/core/curvine-rdma: traits, registry, memory pools, the CQ↔tokio bridge, and the backend conformance test suite (always compiles; on macOS only the TCP passthrough backend is present).crates/adapters/rdma-libfabric,rdma-verbs,rdma-ucx: feature-gated (rdma-libfabric/rdma-verbs/rdma-ucx), Linux-only, with build.rs probing via pkg-config. Missing dependencies fail at compile time only when the feature is enabled, and default builds are unaffected — the same pattern as the currentspdkfeature incurvine-storage-spdk.4.2 Core abstraction: an operation-level API, not a WR-level API
Do not abstract at the "post_send/poll_cq" level (that is verbs dialect; libfabric and UCX have different semantics). Abstract operation semantics + capability declarations, and let each backend map them:
Three design points:
Vec<u8>): this is the core of "matching multiple libraries" — the abstraction layer does not care about connection semantics, only about transporting bytes. The control plane reuses the existing ORPC connection for bootstrap: add one RPC (additive messages in worker.proto, no changes to existing fields) that exchangesbackend / caps / handshake blob / session_nonce, and the Worker replies withpool parameters (max_chunk, credits). Proto changes are additive, consistent with the repository's existing compatibility model.one_sided_write=false(e.g., the tcp provider or a future private SDK) automatically degrades to two-sided emulation; upper layers are unchanged.libloadingto provide a C-ABI vtable dynamic plugin mechanism so vendor SDKs (domestic NICs, smart NICs) can be integrated without code changes.4.3 Data-plane rendezvous flow (matching the existing three-phase stream)
Write path (push; the client issues WRITEs):
Read path (push preferred; the Worker WRITEs into a client-registered receive buffer):
An alternative pull mode (client issues
RDMA READ) suits very large sequential reads and receiver-paced flow; both modes share the sameRemoteTokenand are selected viaread_mode = push|pull. One honest boundary: the disk→MR copy cannot be eliminated (unless going through NVMe-oF/GPU P2P). What RDMA eliminates is the TCP/IP stack, syscalls, and socket buffer copies — one copy on the client side and one on the Worker side, which is exactly the main benefit for block I/O.Worker-to-worker replication (
WriteReplicationBlock) reuses the same data channel, so replication traffic is also zero-copy into MR.4.4 CQ ↔ tokio: a background completion pump
wr_id = conn_id<<32 | slotroutes completion events.inflight > 0; when idle, sleep oneventfd+tokio::AsyncFdand wake on events. Completions notify async callers viatokio::sync::oneshot, coalescing wakes for batched completions.awaita semaphore — natural flow control.4.5 Memory registration pool (MR pool)
reg_mrpinning and page-table setup costs.DataSlice::MemSliceand feedBlockIO::write_region.memlockulimit andmadvise(MADV_HUGEPAGE); pool sizes are configurable.4.6 Configuration shape
The config structs live in
crates/common/curvine-config/src/client_conf.rs/worker_conf.rsand map into ORPC-layer configuration, following the existing factory-function pattern ofclient_rpc_conf/worker_server_conf.4.7 Failure handling, security, and observability
fallback = tcpensures RDMA is never a single point of availability.REMOTE_WRITEfor the write path,REMOTE_READfor the read path). On the RoCE side, isolation relies on PFC/ECN/ACL, and the operational prerequisites must be documented.rdma_bytes_tx/rx, inflight, CQ depth, completion-latency histograms, and per-backend error counters, wired into the existingMetricsReportchannel; add a CLIrdma probefor GID/MTU/provider preflight checks.4.8 Test strategy
curvine-rdma: send/recv, one-sided write/read, token expiry, and error injection, with the same battery run against every backend — so "multi-library adaptation" is more than a claim.cv bench --transport rdmaagainst the TCP baseline.5. Roadmap
curvine-rdmaabstraction layer + TCP passthrough backend + conformance tests; zero behavior changeMemSlicepersistence, degradation/circuit breakerrdma probe6. Risks and Boundaries
libfabric-sys/ucx-sysmust be maintained in-house; pin upstream versions and cover them with the CI matrix rather than chasing latest.7. Open Questions (feedback welcome)
chunk_sizedefaults on the Worker side?Summary: Curvine should adopt libfabric as the primary RDMA backend (its provider mechanism absorbs most of the "multi-library adaptation" problem), leave UCX for GPU/HPC, and keep the control plane on TCP. The framework core is "operation-level traits + opaque handshake + capability negotiation + memory pools + CQ pump", delivered incrementally from M0 to M4, with every step able to fall back to TCP.
All reactions