RDMA User

S3 RDMA Gateway (vgwrdma)

vgwrdma is a versitygw-based binary that exposes a standard S3 API but accelerates PUT/GET object bodies with RDMA, using NVIDIA's cuObject protocol instead of transferring data through the normal HTTP body. A cuObject-aware client (the NVIDIA libcuobjclient SDK, or this repo's own cuobjclient package used by cuobjtest) registers a GPU or host memory buffer, encodes it into an RDMA descriptor token, and sends a normal S3 request with the token in a header. The gateway RDMA READs (PUT) or RDMA WRITEs (GET) directly against that registered buffer, so object bytes never traverse the HTTP connection.

Hardware setup

• An RDMA-capable NIC on both the gateway and the client, connected over a fabric that supports RDMA (RoCE or InfiniBand). This has been validated with a Mellanox Technologies MT28908 Family [ConnectX-6], 100G, using RoCE.
• For the GPU client path, an NVIDIA GPU plus nvidia-peermem support so the NIC can RDMA directly against GPU memory (see kernel note below). The host-memory client path (cuobjclient_host build tag) needs no GPU.
• The gateway host needs the RDMA NIC reachable at the address/port passed to --rdma-ip/--rdma-port; this is a separate listener from the normal S3 --port.
• Both sides must agree on the InfiniBand Dynamic Connection key (--rdma-dc-key on the gateway, VGWRDMA_DC_KEY for the host-memory test client). Change the default (0xffeeddcc) in production.

Validated software environment

Role	OS/Kernel	RDMA/software stack
Client	Rocky Linux 9, kernel 5.14.0-687.15.1.el9_8.x86_64 (required for nvidia-peermem compatibility)	doca-ofed-3.4.0-085000.x86_64 + libcuobjclient-13-3-1.2.0.68-1.x86_64
Server	Rocky Linux 9, kernel 5.14.0-687.17.1.el9_8.x86_64	cuobjserver-1.2.0-1.x86_64

Both sides connected via Mellanox Technologies MT28908 Family [ConnectX-6], 100G, RDMA over Converged Ethernet (RoCE).

The specific kernel build on the client matters: nvidia-peermem (needed for the NIC to RDMA directly into GPU memory) was only compatible with the pinned 5.14.0-687.15.1.el9_8.x86_64 kernel at the time of validation — newer or older Rocky 9 kernel builds are not guaranteed to work with the installed NVIDIA driver/peermem module without re-validation.

Building

# gateway binary (requires libcuobjserver + headers, see Makefile RDMA vars)
make vgwrdma

# example client, GPU-backed (requires CUDA + libcuobjclient)
make cuobjtest-gpu

# example client, host-memory only (requires libibverbs + mlx5, no GPU/CUDA)
make cuobjtest-host

Docker-based builders that bundle the required toolchains are available as make vgwrdma-docker, make cuobjtest-gpu-docker, and make cuobjtest-host-docker (see build/vgwrdma-builder, build/cuobjtest-gpu-builder, build/cuobjtest-host-builder).

Running the gateway

./vgwrdma posix /path/to/storage \
  --rdma-ip 192.168.100.10 --rdma-port 19100 \
  --access admin --secret <secret>

--rdma-ip is required to start vgwrdma; it enables the cubackend.CuServer wrapper and the cumiddleware RDMA header parsing on the S3 listener. Key flags:

• --rdma-port (default 19100) — RDMA listener port.
• --pool-buf-size / --pool-buf-count (default 1GiB × 4) — size and count of the pre-registered RDMA buffer pool; bounds how much host memory is pinned up front and the largest single RDMA transfer (also capped by rdma.MaxTransferSize, 1 GiB).
• --rdma-dc-key, --rdma-num-dcis, --rdma-cq-depth, --rdma-retry-count — RDMA DC connection tunables; see rdma.RDMATunables for the full set and library defaults.

cuobjtest: example client

cuobjtest is a benchmark/validation client built on the cuobjclient SDK. It uploads a buffer of random data, downloads it back, and verifies a CRC32 checksum, reporting per-iteration and aggregate throughput.

./cuobjtest -endpoint http://<gateway-rdma-host>:7070 \
  -bucket cuobjtest -key obj -size 256MiB -n 5 \
  -access admin -secret <secret>

Useful flags:

• -size — transfer size per iteration (e.g. 4MiB, 256MiB, 1GiB), capped by cuobjclient.MaxTransferSize (1 GiB).
• -n — number of iterations to benchmark.
• -put-only / -get-only — restrict the run to one direction (-get-only assumes the object already exists from a prior PUT).
• -std-s3 — bypass the cuObject RDMA transfer and issue plain S3 PUT/GET requests (data in the HTTP body) against the same endpoint.

Which RDMA path is exercised depends on how the binary was built:

• make cuobjtest-gpu (default) links the real NVIDIA libcuobjclient and allocates CUDA device memory for the transfer buffer — this is the GPU Direct RDMA path and requires a CUDA-capable GPU plus nvidia-peermem.
• make cuobjtest-host builds with the cuobjclient_host tag, using rdma/hostclient instead: a pure libibverbs/mlx5 implementation that registers ordinary host memory and needs no GPU or CUDA runtime. Useful for validating the RDMA fabric/gateway path from a machine without a GPU. Configured via environment variables: VGWRDMA_RDMA_DEV, VGWRDMA_RDMA_PORT, VGWRDMA_GID_INDEX, VGWRDMA_DC_KEY.

The printed mode= line in the benchmark header indicates which path is active (real-cuda-cuobject-token, host-memory-cuobject-token, or standard-s3 when -std-s3 is set).

Finding the host-mode RDMA environment variables

VGWRDMA_RDMA_DEV/VGWRDMA_RDMA_PORT/VGWRDMA_GID_INDEX default to "first device", port 1, and "first non-link-local GID". Those defaults frequently pick the wrong device or GID table entry on multi-NIC or RoCE hosts, causing RDMA operations to fail (e.g. IBV_WC_RETRY_EXC_ERR completion errors on the gateway when the client's GID doesn't match the fabric it's actually reachable on). Determine the correct values explicitly instead of relying on the defaults:

1. VGWRDMA_RDMA_DEV — list the RDMA devices on the box and pick the one whose port is PORT_ACTIVE and physically connected to the fabric used by --rdma-ip on the gateway:

   ibv_devices                 # lists device names, e.g. mlx5_0, mlx5_1
   ibv_devinfo | grep -E "hca_id|state|link_layer"
   

   state: PORT_ACTIVE confirms the link is up; link_layer: Ethernet means RoCE (vs. InfiniBand). If a host has multiple HCAs, match the device to the interface on the same subnet as the gateway's --rdma-ip (ip addr / rdma link can map mlx5_* names to network interfaces).

2. VGWRDMA_RDMA_PORT — the HCA port number (not the S3/TCP port). Most NICs expose this as physical port 1; dual-port cards may need 2. Check with ibv_devinfo -d <dev>, which lists port: 1, port: 2, etc. — use the one showing state: PORT_ACTIVE.

3. VGWRDMA_GID_INDEX — run ibv_devinfo -v -d <dev> (or show_gids <dev> if available) and find the GID entry that matches:

   • the RoCE version the fabric/switch actually uses — RoCE v2 unless the network is specifically configured for RoCE v1.
   • the same IPv4 subnet/VLAN as the gateway's --rdma-ip.

   show_gids mlx5_0
   # DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
   # mlx5_0  1       0       fe80:0000:...                                           v1      enp202s0np0
   # mlx5_0  1       1       fe80:0000:...                                           v2      enp202s0np0
   # mlx5_0  1       3       0000:0000:0000:0000:0000:ffff:ac15:0012 192.168.100.10  v2      enp202s0np0
   

   Pick the v2 row (RoCEv2) whose IPv4 column matches the address you expect the client to be reachable on — that row's INDEX is VGWRDMA_GID_INDEX. The auto-selected "first non-link-local GID" default does not distinguish RoCE v1 from v2 or match against the gateway's subnet, so on hosts with more than one non-link-local GID entry it can pick the wrong one; setting it explicitly avoids that.

4. VGWRDMA_DC_KEY — must match the gateway's --rdma-dc-key (default 0xffeeddcc); only needs to be set if the gateway was started with a non-default key.