Context
While evaluating prima.cpp's distributed server mode for integration with a Go cluster orchestrator, I did a source-level review of the ring transport's failure handling (at commit 6f9b7c4). The pipelined-ring design and the server-mode engineering (slot-scoped KV forwarding, server_mode seq-id alignment) look solid — this report is specifically about what happens when a worker rank dies mid-decode.
Observed behavior (3 related code paths)
1. llama_recv_tensors has no receive timeout — src/llama.cpp:18054:
static void llama_recv_tensors(zmq::socket_t & socket, struct llama_ubatch * ubatch, const bool is_out_embd=false) {
std::vector<zmq::message_t> recv_msgs;
if (!zmq::recv_multipart(socket, std::back_inserter(recv_msgs))) {Unlike llama_recv_meta (src/llama.cpp:17920), which sets rcvtimeo=1000 for its first poll, the tensor-phase receive blocks on a default (infinite) socket. If a peer rank dies while the ring is inside the tensor-forwarding phase, its neighbor blocks in recv_multipart forever. In server mode this wedges the decode loop — all slots hang, /health still answers, and only a process restart recovers.
2. When the meta-phase 1s timeout does catch the failure, the error is misattributed — examples/server/server.cpp:2434-2449:
if (ret != 0) {
if (n_batch == 1 || ret < 0) {
// if you get here, it means the KV cache is full - try increasing it via the context size
SRV_ERR("failed to decode the batch: KV cache is full - ...");
...
send_error(slot, "Input prompt is too big compared to KV size. Please try increasing KV size.");llama_recv_meta returning -1 (dead peer) propagates as a nonzero llama_decode return, which this handler interprets exclusively as KV pressure: the batch-halving retry loop runs pointlessly, and the client receives "Input prompt is too big compared to KV size" for what is actually a ring-communication failure.
3. llama_recv_tensors failure is void-logged rather than surfaced — on a failed recv_multipart it logs "Failed to receive tensor data." and returns void, so the caller proceeds with an unfilled ubatch rather than aborting the decode with a distinct error.
Reproduction sketch
Two-device ring in server mode (llama-server --world 2 --rank 0 ... + llama-cli --world 2 --rank 1 ...), start a long streaming completion, kill -9 the rank-1 process mid-generation. Expected observations per the code above: the request either hangs indefinitely (tensor phase) or errors with the KV-size message (meta phase); subsequent requests hang.
Suggested direction
- Apply a bounded
rcvtimeo to the tensor-phase receive, mirroring llama_recv_meta's pattern (perhaps a few seconds rather than 1s, given tensor sizes).
- Give ring-communication failures a distinct error return through
llama_decode so server.cpp can distinguish "peer unreachable" from "KV cache full" — both for the retry logic and the client-facing message.
- Have
llama_recv_tensors return an error status instead of logging + returning void.
Happy to test patches on a multi-node setup. Thanks for the project — the ICLR paper's piped-ring + prefetch design is genuinely compelling for heterogeneous home clusters.
Context
While evaluating prima.cpp's distributed server mode for integration with a Go cluster orchestrator, I did a source-level review of the ring transport's failure handling (at commit 6f9b7c4). The pipelined-ring design and the server-mode engineering (slot-scoped KV forwarding,
server_modeseq-id alignment) look solid — this report is specifically about what happens when a worker rank dies mid-decode.Observed behavior (3 related code paths)
1.
llama_recv_tensorshas no receive timeout —src/llama.cpp:18054:Unlike
llama_recv_meta(src/llama.cpp:17920), which setsrcvtimeo=1000for its first poll, the tensor-phase receive blocks on a default (infinite) socket. If a peer rank dies while the ring is inside the tensor-forwarding phase, its neighbor blocks inrecv_multipartforever. In server mode this wedges the decode loop — all slots hang,/healthstill answers, and only a process restart recovers.2. When the meta-phase 1s timeout does catch the failure, the error is misattributed —
examples/server/server.cpp:2434-2449:llama_recv_metareturning-1(dead peer) propagates as a nonzerollama_decodereturn, which this handler interprets exclusively as KV pressure: the batch-halving retry loop runs pointlessly, and the client receives "Input prompt is too big compared to KV size" for what is actually a ring-communication failure.3.
llama_recv_tensorsfailure is void-logged rather than surfaced — on a failedrecv_multipartit logs"Failed to receive tensor data."and returnsvoid, so the caller proceeds with an unfilledubatchrather than aborting the decode with a distinct error.Reproduction sketch
Two-device ring in server mode (
llama-server --world 2 --rank 0 ...+llama-cli --world 2 --rank 1 ...), start a long streaming completion,kill -9the rank-1 process mid-generation. Expected observations per the code above: the request either hangs indefinitely (tensor phase) or errors with the KV-size message (meta phase); subsequent requests hang.Suggested direction
rcvtimeoto the tensor-phase receive, mirroringllama_recv_meta's pattern (perhaps a few seconds rather than 1s, given tensor sizes).llama_decodesoserver.cppcan distinguish "peer unreachable" from "KV cache full" — both for the retry logic and the client-facing message.llama_recv_tensorsreturn an error status instead of logging + returning void.Happy to test patches on a multi-node setup. Thanks for the project — the ICLR paper's piped-ring + prefetch design is genuinely compelling for heterogeneous home clusters.