RDMA Devel

Components

• cmd/vgwrdma: the gateway binary. Adds --rdma-* flags on top of the standard versitygw flags and wires cubackend.CuServer in front of the chosen storage backend when --rdma-ip is set.
• cubackend: CuServer, a backend.Backend decorator. Delegates every S3 operation to the wrapped backend except PutObject/GetObject, which use RDMA transfers when the request carries a cuObject descriptor.
• cumiddleware: Fiber middleware that parses the cuObject RDMA headers (X-Amz-Rdma-Token, or the legacy 3-header scheme used by this repo's own test tooling) and stashes the descriptor/size/remote address on the request context for cubackend to pick up.
• rdma: cgo bindings to libcuobjserver (the vendor library implementing the RDMA DC protocol server side). Builds only on linux/amd64 with cgo; a stub implementation on other platforms lets the rest of the tree build.
• rdma/bufferpool: a fixed pool of pre-allocated, pre-registered RDMA buffers so CuServer doesn't pay allocation/registration cost per request.
• cuwrapper: small C++ shims (cuobjserver_wrapper, cuobjclient_wrapper, rdma_host_client_wrapper) that present a plain C ABI over the vendor C++ libraries so cgo can link against them.
• cuobjclient: Go client SDK used by cuobjtest. Has two implementations selected by build tag: a CUDA/GPU-backed one (session_linux.go, default) using the real NVIDIA libcuobjclient, and a host-memory-only one (session_host_linux.go, cuobjclient_host build tag) using rdma/hostclient for RDMA-capable machines with no GPU.
• cmd/cuobjtest: example/benchmark client, see below.

cuobjclient (Go client SDK)

The SDK lives in cuobjclient/ and exposes a small session-based API:

• NewSession(size int) (*Session, error)
• (*Session).Upload(base *s3.Client, bucket, key string, src []byte) error
• (*Session).Download(base *s3.Client, bucket, key string, dst []byte) error
• (*Session).Close()

Platform/build behavior

• Default GPU build (session_linux.go):
  • build tags: linux && amd64 && cgo && !cuobjclient_host
  • uses NVIDIA libcuobjclient and CUDA memory registration
• Host-memory build (session_host_linux.go):
  • build tags: linux && amd64 && cgo && cuobjclient_host
  • no GPU dependency; uses rdma/hostclient
• Unsupported platform stub (session_stub.go):
  • selected when not linux/amd64/cgo
  • returns explicit "only supported on linux/amd64 with cgo" errors

Maximum transfer size is 1 << 30 bytes (1 GiB), defined in cuobjclient/limits.go.

Session lifecycle and constraints

1. Create one session for a fixed transfer size.
2. Reuse that session across multiple sequential transfers.
3. Each Upload/Download buffer length must exactly match session size.
4. Session methods are not goroutine-safe; do not call methods concurrently on the same session.
5. Call Close() to release GPU/host RDMA resources.

What Upload/Download do under the hood

Implementation is shared in cuobjclient/transport.go:

• SDK sends a zero-byte PutObject/GetObject request.
• It injects cuObject headers (X-Amz-Rdma-*) through Smithy build middleware so headers are included in SigV4 signed headers.
• Actual payload transfer happens via RDMA, not HTTP body streaming.
• SDK validates gateway response headers:
  • X-Amz-Rdma-Reply (offload success)
  • X-Amz-Rdma-Bytes-Transferred (must match expected size)
• If these headers are missing or mismatched, SDK returns an error rather than silently treating the transfer as success.

Host-memory mode environment variables

When built with -tags cuobjclient_host, NewSession reads:

• VGWRDMA_RDMA_DEV (default: first device)
• VGWRDMA_RDMA_PORT (default: 1)
• VGWRDMA_GID_INDEX (default: auto-select first non-link-local GID)
• VGWRDMA_DC_KEY (default: 0xffeeddcc via hostclient.DefaultDCKey)

Minimal SDK usage sketch

sess, err := cuobjclient.NewSession(size)
if err != nil {
    return err
}
defer sess.Close()

putBuf := make([]byte, size)
getBuf := make([]byte, size)

if err := sess.Upload(s3Client, bucket, key, putBuf); err != nil {
    return err
}
if err := sess.Download(s3Client, bucket, key, getBuf); err != nil {
    return err
}

cuobjtest (SDK example utility)

For a full, runnable reference using this SDK, see cmd/cuobjtest/main.go.

cuobjtest demonstrates:

• creating a normal AWS SDK v2 S3 client (UsePathStyle: true)
• creating and reusing one cuobjclient.Session
• PUT/GET data-path benchmarking with per-iteration latency + GB/s
• checksum verification for round-trip correctness
• baseline comparison mode with standard S3 only (-std-s3)

Example run:

make cuobjtest-gpu-docker

./cuobjtest \
  -access <access-key> \
  -secret <secret-key> \
  -endpoint http://<gw-host>:7070 \
  -bucket cuobjtest

If you are validating SDK integration in a new client, cuobjtest is the best starting point for expected control flow and error handling.