S3 RDMA
versitygw ships an optional RDMA-accelerated gateway binary, vgwrdma, that exposes the same S3 API as the standard gateway but moves object data over an RDMA fabric instead of through the HTTP body. The target worklaod is lower latency and higher throughput for large object transfers in HPC, AI/ML, and data-intensive workloads where the client and storage server are connected by a high-speed RDMA fabric such as RoCE or InfiniBand.
RDMA lets a NIC transfer data directly between the memory of two machines without involving the CPU or the OS kernel on either end. Eliminating the kernel networking stack removes the per-byte copy overhead and interrupt latency that cap ordinary TCP/IP throughput. RDMA fabrics (InfiniBand, RoCE) routinely sustain very high performace per port.
NVIDIA GPUDirect Storage (GDS) extends RDMA to GPU memory. Normally, data read from a storage device must first land in host (CPU) memory and then be copied to the GPU. With GDS, the NIC DMA-engines data directly into GPU DRAM, eliminating the host memory staging copy. This is the key enabler for applications that load training datasets or inference inputs straight into the GPU.
cuObject is the NVIDIA library protocol that ties GPUDirect Storage to an S3-style API. A cuObject-aware client registers a GPU (or host) memory buffer with the RDMA subsystem, encodes the resulting registration as a descriptor token, and includes that token in a standard S3 request header. The gateway reads the token, connects directly to the client's registered buffer, and performs the RDMA transfer there. The object bytes never appear in the HTTP body at all, instead are transferred in a separate RDMA session. vgwrdma implements the server side of this protocol via the libcuobjserver vendor library.
| Scenario | Benefit |
|---|---|
| AI/ML training data loading from object storage | Direct GPU memory ingestion; eliminates host-memory copy |
| HPC checkpoint and restart | High-bandwidth, low-latency bulk transfer to/from GPU or host buffers |
| Large-object streaming on a RoCE/IB fabric | Near line-rate throughput limited by the NIC, not the CPU |
| Standard S3 workloads on RDMA-connected nodes |
vgwrdma falls back to normal HTTP for non-RDMA clients; nothing changes for those clients |
The RDMA path is only activated when the client sends a cuObject descriptor header. Clients that do not pass RDMA descriptors communicate through the normal HTTP body unchanged, so vgwrdma can serve a mixed cluster without reconfiguration.
vgwrdma is built from the same codebase as the standard versitygw binary and supports all the same storage backends (POSIX, S3 proxy, ScoutFS, etc.) and S3 API operations. It adds a second listener port (--rdma-port) dedicated to the RDMA fabric and wraps the chosen backend with cubackend.CuServer, which intercepts PutObject/GetObject calls that carry a cuObject token and performs the RDMA transfer instead of reading from or writing to the HTTP connection.
The RDMA enabled gateway is designed to add an S3 RDMA service in front of any exisitng storage system. The RDMA transfers happen between client and vgwrdma memory. The gateway service accesses data in the backend storage system through the standard implementation for that system. This means that it is more likely to benefit client systems that bottleneck on CPU memory copy data transfers. Overall performance is still primarily determined by backend storage system capability.
A host-memory build (cuobjclient_host tag) uses ordinary libibverbs to register host RAM instead of GPU memory. This is useful for benchmarking the gateway and RDMA fabric on machines without a GPU, and for any workload that wants high-throughput object transfers without GPU involvement.
-
User Guide — hardware requirements, building, running the gateway, and the
cuobjtestbenchmark client - Developer Docs — package layout, internal architecture, and how to extend or integrate the RDMA path
- NVIDIA cuObject: Accelerated CUDA libraries for Object Storage
