Large-value GET: copies after lookup vs registered-memory DMA (multi-MiB)

[serge-ouedraogo]

We are evaluating Valkey as an external KV-cache tier for LLM inference, where values are often multi-MiB, and we are trying to better understand the architectural ceiling of the large-value GET path.
On the same test environment, we measured:

Path	Throughput
CXL-backed RDMA Read (ib_read_bw, q=16)	~372.9 Gb/s
DRAM-backed RDMA Read	~378.0 Gb/s
CPU sequential read from the CXL NUMA node	~121.3 Gb/s
Local UDS GET, 16 MiB, c32	~36.6 Gb/s
Valkey Over RDMA GET, 16 MiB, c32	~29.5 Gb/s
The RDMA Read test is Valkey-free; the HCA reads directly from registered CXL-backed memory. The GET rows are RESP GET of pre-populated 16 MiB strings on a Valkey-based server. The UDS result is particularly interesting because it removes the network from the Valkey path.
We are not treating this as a Valkey bug. We are trying to understand what happens between locating a large value in memory and delivering it to the client. If GET is primarily memcpy-class CPU processing, the ~121 Gb/s CPU CXL read may be the fairer comparison than the ~373 Gb/s DMA result, but there is still a meaningful gap.
A few questions where your guidance would help:

1. For a multi-MiB GET, what are the major data movements after the value is located? How many CPU copies or memory traversals should we expect?
2. Does RESP reply construction copy the value into an intermediate buffer, or can it reference the existing value memory directly?
3. Which portions of large-value GET processing remain on the main event loop versus I/O threads?
4. If a value resides in externally managed or registered memory, can a module or transport consume that memory directly after lookup, or does it still need to pass through the conventional reply path?
   Any pointers to the relevant Valkey code paths—or corrections to our understanding of the large-value GET path—would be very helpful.
   cc @zuiderkwast @PingXie @madolson @enjoy-binbin @pizhenwei

[pizhenwei]

Hi,

1. Characteristics of the RESP Protocol
   The Redis Serialization Protocol (RESP) is a text-based, streaming protocol with the following key features that impact high-performance data access:

Streaming & Dynamic Length: RESP is stream-oriented, meaning there is no fixed message boundary. The length of a command or response is not known upfront until the entire payload is parsed.

Self-Descriptive Format: It uses prefixes to indicate data types and lengths, e.g., *1\r\n$4\r\nPING\r\n. This requires the server to parse the request and construct a dynamic response, often with variable lengths (especially for GET commands where the value size is unpredictable).

Fragmentation: A single logical command can be split across multiple network packets (e.g., *1\r\n, $4\r\n, PING\r\n are all valid fragments). This works well with TCP streams but poses challenges for protocols that expect deterministic, contiguous data blocks.

These inherent properties make RESP a "software-friendly" protocol that relies heavily on CPU parsing and dynamic memory management.

2. Characteristics of Valkey Over RDMA
   RDMA is designed for deterministic, bulk data transfers with fixed sizes and pre-registered memory regions. When adapting Valkey (which uses RESP) to RDMA, the following characteristics emerge:

Control/Data Plane Separation: To handle the dynamic nature of RESP, the Valkey-over-RDMA implementation typically splits the path:

Control Messages (short, e.g., GET command and metadata) are exchanged via RDMA Send/Recv.

Data Payload (large values) is transferred via RDMA Read/Write operations.

Memory Registration Overhead: Large values must be placed in contiguous, pre-registered memory regions before they can be exposed for RDMA access. This registration and management adds CPU and software overhead.

Multi-RTT Interaction: Due to the unpredictable response size, the server cannot initiate a direct RDMA Read for the value without first locating and constructing it. The process requires at least a round-trip of control messages (request → metadata reply → data transfer), incurring additional latency and CPU cycles for protocol orchestration.

Software-Orchestrated Data Movement: This data "orchestration" step is performed by the CPU, not the hardware.

3. Summary
   The measured throughput gap between pure hardware RDMA Read (~373 Gb/s) and Valkey Over RDMA GET (~29.5 Gb/s) is not a bug or a limitation of the network hardware, but rather a fundamental consequence of the protocol impedance mismatch:

RESP is dynamic and streaming, requiring CPU-intensive parsing and construction that cannot be offloaded to a simple DMA engine.

RDMA requires certainty and pre-defined parameters (address, length, and timing), which conflict with the on-the-fly nature of RESP responses.

Therefore, Valkey Over RDMA must insert a software translation layer that orchestrates the data transfer. The performance ceiling is set by the CPU's ability to parse protocols, manage memory, and coordinate the DMA engine, not by the raw bandwidth of the interconnect.

In essence, the GET path cannot be reduced to a single DMA operation because the software must first "understand" the request and "construct" the response before the hardware can be told where to find the data. The ~29.5 Gb/s result reflects the true end-to-end cost of this software-hardware co-design.

Please translate this page into EN by AI to understand more.

[serge-ouedraogo]

Thanks @pizhenwei. This is helpful, and it makes sense that a GET is not a single ib_read of the value: RESP is dynamic, and the RDMA path has to translate that into registered, sized transfers.
The local UDS result makes us think there may be an additional limitation earlier in the GET path though. On the same desk we see:

• CPU sequential read from the CXL NUMA node (Valkey-free, memcpy-class): ~121 Gb/s
• Local UDS GET, 16 MiB, c32: ~36.6 Gb/s
• Valkey Over RDMA GET, 16 MiB, c32: ~29.5 Gb/s
  Since UDS removes both the network and the RDMA translation layer, the drop from ~121 to ~36.6 Gb/s appears to happen largely within the Valkey/application path itself. That could include value access, reply construction, buffer handling/copies, and event-loop or I/O-thread work.
  The additional drop from ~36.6 to ~29.5 Gb/s may then be attributable to the extra RDMA transport and orchestration work you described.
  The part we are trying to understand is what happens after the key lookup has resolved the value and its length. At that point, does Valkey inherently need to construct/copy the payload into connection or transport buffers, or is there room to reference the existing value memory more directly and reduce some of that work?
  Ultimately, we are trying to understand which parts of the ~121 → ~36.6 → ~29.5 Gb/s progression are inherent to the large-value GET architecture and which may be optimizable. Any pointers into the GET, reply, or I/O-thread code paths would be very helpful.

[pizhenwei]

Thanks @pizhenwei. This is helpful, and it makes sense that a GET is not a single ib_read of the value: RESP is dynamic, and the RDMA path has to translate that into registered, sized transfers. The local UDS result makes us think there may be an additional limitation earlier in the GET path though. On the same desk we see:

• CPU sequential read from the CXL NUMA node (Valkey-free, memcpy-class): ~121 Gb/s
• Local UDS GET, 16 MiB, c32: ~36.6 Gb/s
• Valkey Over RDMA GET, 16 MiB, c32: ~29.5 Gb/s
  Since UDS removes both the network and the RDMA translation layer, the drop from ~121 to ~36.6 Gb/s appears to happen largely within the Valkey/application path itself. That could include value access, reply construction, buffer handling/copies, and event-loop or I/O-thread work.
  The additional drop from ~36.6 to ~29.5 Gb/s may then be attributable to the extra RDMA transport and orchestration work you described.
  The part we are trying to understand is what happens after the key lookup has resolved the value and its length. At that point, does Valkey inherently need to construct/copy the payload into connection or transport buffers, or is there room to reference the existing value memory more directly and reduce some of that work?
  Ultimately, we are trying to understand which parts of the ~121 → ~36.6 → ~29.5 Gb/s progression are inherent to the large-value GET architecture and which may be optimizable. Any pointers into the GET, reply, or I/O-thread code paths would be very helpful.

Here is a simplified English explanation of the GET key request/response flow, with a focus on the RDMA-related limitations you mentioned.

Simplified RESP Example for GET key

Client → Server (Request):
*2\r\n$3\r\nGET\r\n$3\r\nkey\r\n
(This is a simple byte string sent over the TCP stream.)

Server → Client (Response):
If key exists: $5\r\nvalue\r\n

Why RDMA Write Cannot Be Used Here

When the server sends back the response, it must serialize the reply into a plain string (a sequence of bytes) according to the RESP protocol. The server does not know or care about the client-side memory address where this string should eventually be stored. Therefore, the server cannot use RDMA write to directly place the response payload into the client's pre-registered memory buffer, because RDMA write requires the server to know the remote memory address and key (memory handle) of the destination buffer on the client side.

Why the Client Cannot Temporarily Register RDMA Memory for Streaming Data

On the client side, when parsing the response value, the RESP protocol requires the client to completely and correctly parse the entire value string from the incoming byte stream. However, the client reads the data streamingly — it does not know in advance how many bytes are already available in the TCP receive buffer/RDMA RX buffer, nor how many more bytes are still being sent by the server. The message may arrive in multiple chunks, and the client can only determine the full length of the value after reading the length prefix (e.g., $5) and then waiting for exactly that many bytes to arrive.

Because of this streaming nature:

The client cannot predict the exact arrival time or the complete size of the value before it fully arrives.

It cannot temporarily register an RDMA memory region and ask the server to write directly into that specific memory, because the server would need a fixed, known remote address and key at the moment of sending.

Moreover, the client does not know which portion of the incoming stream belongs to the value until the entire RESP frame is fully received and parsed.

Thus, the client must read the data from the socket/RDMA RX buffer into a local buffer incrementally, parse the RESP format step by step, and only after the full value is received and parsed can it copy or move the extracted value into its final application memory. This makes RDMA-based remote write infeasible for this kind of variable-length, stream-parsed protocol.

[serge-ouedraogo]

Thanks @pizhenwei — I think the distinction is clear now.

From your explanation, we understand that RESP and RDMA can work together, but RESP does not naturally expose the memory-placement semantics needed for one-sided RDMA placement. RESP GET still returns a framed byte stream, while direct placement would require the server to know the client destination address, rkey, and length.

For us, LMCache does know the expected KV-chunk size ahead of time and preallocates the pinned MemoryObj, so size is not really the issue. But RESP still has to be parsed to identify the reply/payload, and the server still does not know that specific MemoryObj is the destination. I think destination awareness is the main gap we would need to bridge for a placement-aware path.
Thanks again for your input.