Distributed LLM Inference: llama.cpp RPC across Metal + CUDA over 10GbE

Real-world benchmarks of llama.cpp's RPC backend splitting inference across Apple Silicon (Metal) and NVIDIA Blackwell (CUDA) connected by a 10 Gbps direct Ethernet link.

Why This Matters

Most LLM users are limited to a single machine's memory. llama.cpp's RPC backend lets you combine GPUs across machines over TCP — no shared memory, no NVLink, just Ethernet. This opens up:

• Running 100GB+ models that don't fit on one machine
• Using mixed hardware (Apple Silicon + NVIDIA) in the same inference pipeline
• Leveraging tensor core acceleration for prompt processing on one machine while decoding on another

But what are the real-world tradeoffs? This repo documents actual benchmarks, not theoretical throughput.

Hardware

Machine	GPU	Memory	Backend	Role
Mac Studio	Apple M2 Ultra	128 GB unified	Metal	llama-server host + decode
DGX Spark	NVIDIA GB10 Blackwell	120 GB unified	CUDA 13	rpc-server (remote compute)

Network: 10 Gbps direct-attach Ethernet (no switch, point-to-point). Measured throughput: 9.41 Gbps via iperf3.

Results

Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) — fits one machine easily

Mode	Prompt (tok/s)	Generation (tok/s)	Notes
Local Metal only	76.1	91.8	Baseline — all on Mac Studio
RPC Metal + CUDA	317.7	52.7	4.2x prefill speedup, decode slower

Qwen2.5-72B-Instruct Q4_K_M (44.2 GB) — fits Mac Studio alone (tight)

Mode	Prompt (tok/s)	Generation (tok/s)	Model Split	Notes
Local Metal only	28.2	11.1	44 GB on Metal	All layers local
RPC Metal + CUDA	29.5	5.9	30.7 GB Metal + 13.8 GB CUDA	~50% decode penalty

Analysis

Prefill (prompt processing) benefits from RPC:

• Blackwell tensor cores accelerate matrix multiplications during prefill
• 4.2x speedup on 7B, slight improvement on 72B (memory bandwidth limited at 72B scale)
• Benefit increases with prompt length

Decode (token generation) is slower with RPC:

• Each token requires a network round-trip (~0.17ms on 10GbE)
• Latency compounds: ~2x slower decode on 7B, ~47% slower on 72B
• This is inherent to the architecture — decode is sequential and latency-bound

Key insight: RPC's real value isn't speed — it's capacity. For models that fit one machine, local is faster. For models that don't fit (100GB+), RPC is the only option short of buying bigger hardware.

Projected: Models That Need Both Machines

Model	Size (Q4_K_M)	Split	Expected Decode
Qwen3-235B-A22B	~132 GB	~82 GB Metal + ~50 GB CUDA	~5-8 tok/s
MiniMax M2.5 (230B/10B MoE)	~138 GB	~82 GB Metal + ~56 GB CUDA	~5-8 tok/s
DeepSeek-R1	~404 GB	Won't fit 248 GB combined	Need quantization

Architecture

┌─────────────────────┐         10 Gbps          ┌─────────────────────┐
│   Mac Studio        │    direct Ethernet link   │   DGX Spark         │
│   (Metal backend)   │◄────────────────────────► │   (CUDA backend)    │
│                     │                           │                     │
│   llama-server      │         TCP:50052         │   rpc-server        │
│   - loads GGUF      │ ────── RPC protocol ────► │   - provides CUDA   │
│   - serves API      │                           │     compute for     │
│   - runs Metal      │                           │     offloaded       │
│     layers          │                           │     layers          │
│   - port 9999       │                           │                     │
└─────────────────────┘                           └─────────────────────┘

• llama-server (Mac Studio) loads the model file, auto-splits layers between local Metal and remote CUDA based on available memory
• rpc-server (DGX) provides raw GPU compute — no model file needed on this machine
• Layer assignment is automatic: llama.cpp queries available memory on each device and distributes accordingly
• The GGUF file only needs to exist on the llama-server host

Setup

Prerequisites

• Two machines with GPUs (any combination of Metal/CUDA/ROCm)
• Network connection between them (10GbE recommended, 1GbE works but slower)
• Same llama.cpp build on both machines (same commit)

1. Build llama.cpp on both machines

Both machines must build from the same commit to ensure RPC protocol compatibility.

Machine A (Metal — macOS):

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build \
  -DGGML_METAL=ON \
  -DGGML_RPC=ON \
  -DGGML_BLAS=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(sysctl -n hw.ncpu)

Machine B (CUDA — Linux):

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build \
  -DGGML_CUDA=ON \
  -DGGML_RPC=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

2. Download a GGUF model (on Machine A only)

# Example: Qwen2.5-7B for quick testing
huggingface-cli download bartowski/Qwen2.5-7B-Instruct-GGUF \
  Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  --local-dir ./models/

# For serious benchmarking: Qwen2.5-72B
huggingface-cli download bartowski/Qwen2.5-72B-Instruct-GGUF \
  Qwen2.5-72B-Instruct-Q4_K_M.gguf \
  --local-dir ./models/

3. Start rpc-server on Machine B

# On Machine B (CUDA)
cd llama.cpp
LD_LIBRARY_PATH=build/bin build/bin/rpc-server \
  -H 0.0.0.0 \
  -p 50052

You should see:

ggml_rpc_server: listening on 0.0.0.0:50052

4. Start llama-server on Machine A with RPC

# On Machine A (Metal)
cd llama.cpp
build/bin/llama-server \
  -m models/Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  --rpc <MACHINE_B_IP>:50052 \
  -ngl 99 \
  --host 0.0.0.0 \
  --port 9999 \
  -c 4096

The server will log the layer split:

llm_load_tensors: offloading 28 layers to GPU (RPC)
llm_load_tensors: offloading 4 layers to GPU (Metal)

5. Test

curl -s http://localhost:9999/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5",
    "messages": [{"role": "user", "content": "Explain distributed inference in one paragraph."}],
    "max_tokens": 200
  }' | python3 -m json.tool

Benchmarking

Quick benchmark with llama-bench

# Local only (baseline)
build/bin/llama-bench \
  -m models/Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  -ngl 99 \
  -p 512 -n 128

# With RPC
build/bin/llama-bench \
  -m models/Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  --rpc <MACHINE_B_IP>:50052 \
  -ngl 99 \
  -p 512 -n 128

Benchmark script (automated)

See benchmark.sh — runs local vs. RPC for both prompt and generation, outputs a comparison table.

Reproducing with different hardware

This setup works with any combination:

Machine A	Machine B	Expected behavior
Metal (Apple Silicon)	CUDA (NVIDIA)	Tested in this repo
CUDA (NVIDIA)	CUDA (NVIDIA)	Both use tensor cores
Metal	Metal	Both use Apple GPU
Metal	ROCm (AMD)	Should work (untested)
CUDA	CPU (no GPU)	Machine B uses CPU for its layers

For 1GbE instead of 10GbE: expect ~10x more network overhead on decode. Prefill may still benefit if the remote GPU is significantly faster.

Known Issues

1. Ollama GGUFs may be incompatible — Ollama's custom GGUF format sometimes adds non-standard metadata (e.g., rope.dimension_sections with wrong element count). Use GGUFs from HuggingFace (bartowski, etc.) instead.

2. SSH background jobs — nohup rpc-server & over SSH can fail silently. Use ssh -t with < /dev/null or run in a tmux/screen session.

3. Firewall — ensure the RPC port (50052) is open on Machine B. On Ubuntu: sudo ufw allow 50052/tcp.

4. Memory pressure — if either machine runs low on memory, llama.cpp may crash or OOM-kill. Check available memory before loading large models: free -h (Linux) or memory_pressure (macOS).

Lessons Learned

1. RPC is for capacity, not speed. If your model fits on one machine, keep it local. RPC adds latency to every decode step.

2. Prefill benefits most from fast remote GPUs. Blackwell tensor cores made a real difference for prompt processing — this matters for long-context applications (RAG, document analysis).

3. 10GbE direct-attach is cheap and effective. A $30 DAC cable between two machines gives 9.4 Gbps with sub-millisecond latency. No switch needed.

4. Model files only need to exist on the host. The rpc-server receives tensor data over the network — no model download needed on remote machines.

5. Same commit matters. RPC protocol isn't versioned — mismatched builds between host and remote can cause silent failures or crashes.

Related

• llama.cpp RPC documentation
• GGUF model format
• bartowski's GGUF quantizations — reliable source for compatible GGUFs

License

MIT