fix(rdma): align server/client depth, max_msg and MAX_BLOCK_BYTES defaults - #275
Merged
Merged
Conversation
…efaults Six default value changes to prevent silent hot-round degradation observed on GLM-5.2-NVFP4 B200 (-29.8% hot vs cold): 1. Server RDMA depth default 1->4 (rdma_server.cc) 2. Client RDMA depth default 1->4 (rdma_transport.cc) 3. Client DFKV_RDMA_MAX_BLOCK_BYTES default 64MiB->4MiB (rdma_transport.cc) 4. Server --max-msg default 64MiB->32MiB (dfkv_server_main.cc) 5. Server DFKV_RDMA_RECV_SEGMENT_SIZE default 2GiB->16GiB (rdma_server.cc) 6. Server DFKV_RAM_TIER_BYTES default 4GiB->16GiB (ram_tier.h) With new defaults: per-connection 16 MiB (depth 4 x 4 MiB slot), 16 GiB segment holds ~1024 connections; 16 GiB RAM tier matches production deployment sizes. Verification (xb01-0064, 8xB200, GLM-5.2-NVFP4, dfkv v2.6.3): Before (depth=1): Cold 37306 tok/s, Hot 26191 tok/s = -29.8% After (depth=4): Cold 38549 tok/s, Hot 37409 tok/s = -2.9% Prefetch failures: 532 -> 0, depth clamp warnings: >0 -> 0
ketor
force-pushed
the
fix/rdma-defaults-depth-maxmsg
branch
from
August 7, 2026 12:13
d8f5749 to
512c988
Compare
Update DEPLOY.md, CONNECTORS.md, datapath-perf-notes.md to reflect new defaults from PR dingodb#275: - DFKV_RDMA_DEPTH: 1 -> 4 - DFKV_RDMA_MAX_BLOCK_BYTES: 64 MiB -> 4 MiB - --max-msg: 64 MiB -> 32 MiB - DFKV_RDMA_RECV_SEGMENT_SIZE: 2 GiB -> 16 GiB - DFKV_RAM_TIER_BYTES: 4 GiB -> 16 GiB Connection capacity examples updated: 64 MiB/depth=1/2 GiB/31 QP -> 4 MiB/depth=4/16 GiB/1024 QP. systemd example updated. Performance measurements preserved. ARCHITECTURE.md unchanged (no default values to update).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On GLM-5.2-NVFP4 (DSA multi-pool model) with dfkv L3, the hot round throughput is -29.8% worse than cold when server/client RDMA defaults are used without explicit configuration:
Root Cause
Server and client RDMA pipeline depth default=1 while production uses depth=4. The mismatch causes batching window clamp, PUT batch failures during burst writes (128 pages per batch), and subsequent hot-round prefetch failures.
Additionally, client DFKV_RDMA_MAX_BLOCK_BYTES defaults to 64 MiB (inherited from max_payload), allocating ~256 MiB per connection and capping receive segment capacity at ~256 connections. GLM-5.2-NVFP4 per-SG-chunk payload is ~1 MiB, so 64 MiB is 64x oversized.
Fix
Four default value changes:
Verification (xb01-0064, 8xB200, GLM-5.2-NVFP4, dfkv v2.6.3)
Before fix (server depth=1, client depth=4)
After fix (server depth=4, client depth=4)
Improvement
Connection memory impact
Files Changed
src/cache/rdma_server.cc: ServerDepth() default 1 to 4src/cache/rdma_server.h: header commentsrc/cache/dfkv_server_main.cc: --max-msg default 64 to 32 MiB, --rdma-depth help, --max-msg helpsrc/transport/rdma_transport.cc: depth_ default 1 to 4, DFKV_RDMA_MAX_BLOCK_BYTES default 64 to 4 MiBsrc/transport/rdma_transport.h: header commentsScope