Skip to content

Pipeline Parallelism

Jump to bottom
youngharold edited this page Feb 17, 2026 · 4 revisions

Pipeline Parallelism

Why Not True GGUF Layer Sharding?

We investigated splitting GGUF files by transformer block and distributing partial files to machines, so no single machine needs RAM for the full model. This turned out to be not feasible with the llama.cpp ecosystem:

Constraints

  1. llama-server requires the full GGUF file. It loads the file header, all tensor metadata, and builds the full computation graph at startup. There's no --layer-start/--layer-end flag — it expects embedding, all transformer blocks, and the output head.

  2. No hidden-state API. Even if you could load partial models, llama-server doesn't expose intermediate hidden states between layers. You can't chain two instances where one runs layers 0-31 and feeds hidden states to another running layers 32-63.

  3. Exo does pipeline parallelism by partitioning models across machines, but it only supports Apple Silicon and NVIDIA CUDA (no Windows, no ROCm). This rules out our 4070 desktop (Windows) and the incoming 7900 XTX rig (ROCm).

  4. llama.cpp's RPC backend already distributes layers. The rpc-server instances on worker machines receive individual GGML tensor operations over TCP — the coordinator handles the layer assignment via --tensor-split. This IS distributed inference, just not pipeline parallelism.

The Real Bottleneck

The only limitation of the RPC approach: the coordinator must mmap the full GGUF into system RAM. For a Qwen2.5-72B-Instruct Q8_0 model (~75GB), no workstation in our fleet has enough RAM.

Solution: Unraid as Coordinator

The Unraid server has 128GB DDR3 RAM and a Quadro P400 (2GB VRAM). Making it the coordinator:

Property Value
RAM 128 GB (enough for any model mmap)
GPU Quadro P400 (2GB) — gets zero layers
Network Gigabit Ethernet to all machines
OS Unraid (Linux-based)

How It Works

Unraid (coordinator, 128GB RAM)
    │ mmaps full 75GB GGUF → no swap
    │ P400 listed in tensor-split with vram_gb: 0 (gets zero layers)
    │ All compute delegated to RPC workers
    ▼
┌─────────────────┬───────────────────┬──────────────┬────────────────┐
│ Desktop          │ XPS               │ MacBook M2   │ 7900 XTX rig  │
│ 4070 Ti (16GB)  │ 2070 (8GB)        │ Metal (16GB) │ 2x XTX (48GB) │
│ 3060 (12GB)     │                   │              │ (coming soon)  │
│ rpc:50052,50053 │ rpc:50052         │ rpc:50052    │ rpc:50052,53   │
└─────────────────┴───────────────────┴──────────────┴────────────────┘

Why P400 must be listed: llama-server auto-detects the P400 as CUDA device 0. If it's not in the --tensor-split array, the array length doesn't match the device count and distribution breaks. Setting vram_gb: 0 gives it a 0.00 split ratio — zero layers assigned, zero VRAM used.

Config

See configs/cluster-unraid-coord.yaml for the full working config.

GGUF Inspection

Before distributing a model, inspect it to understand RAM requirements and layer distribution:

# Show model metadata, layer count, tensor sizes
tightwad inspect ~/models/Qwen2.5-72B-Instruct-Q8_0.gguf

# Show which layers go to which GPU in your cluster
tightwad inspect ~/models/Qwen2.5-72B-Instruct-Q8_0.gguf --plan

The --plan flag reads your cluster config and maps layers to GPUs based on tensor-split ratios, showing estimated VRAM per GPU.

Requires: pip install tightwad[inspect] (adds the gguf package)

Model Distribution

Copy models to worker machines for local access or caching:

# rsync to all workers that have model_dir configured
tightwad distribute qwen2.5-72b

# Specific target
tightwad distribute qwen2.5-72b -t 192.168.86.36:/models/

# Preview without transferring
tightwad distribute qwen2.5-72b --dry-run

Workers need ssh_user and model_dir in config. Transfers run in parallel via asyncio with Rich progress bars. Prefers rsync (delta transfer, resume) with scp fallback.

Speculation Over the Pool

Once the coordinator is running with RPC workers, layer the speculative decoding proxy on top for the full Combined Mode experience. See Architecture#Combined Mode and Speculative Decoding for details.

Benchmark Results (2026-02-17)

Llama 3.1 8B (M4 Metal draft) → Llama 3.3 70B (4-GPU RPC pool: 4070+3060+2070+M2, 52GB VRAM):

Mode Tokens Time Speed
RPC pool direct (autoregressive) 512 231s 2.2 tok/s
RPC pool + speculation 519 127s 4.1 tok/s
Speedup 1.86x

100% acceptance rate, 33 tokens/round, 6 verification rounds for 519 tokens.

Critical: Same-Family Drafting Required

Draft → Target Acceptance Result
Llama 3.2 3B → Llama 3.3 70B 1.6% 0.1 tok/s (10x slower)
Llama 3.1 8B → Llama 3.3 70B 100% 4.1 tok/s (1.86x faster)

Llama 3.2 (1B/3B) has a different architecture from Llama 3.1/3.3 despite sharing a tokenizer. Same tokenizer is necessary but not sufficient — the models must share the same architecture for high acceptance.

Next Test: Internet GPU Pooling

Pool GPUs across the internet (not just LAN) via Tailscale/WireGuard to load a model too large for any single location. The RPC backend and proxy are plain TCP/HTTP — they work over any network path.

Why speculation matters even more over the internet: Higher latency means autoregressive generation is even slower (each token = one internet round-trip). Speculation amortizes that latency over 32 tokens per round-trip. The worse the latency, the bigger the speculation win.

Goal: Load a 100B+ model across geographically distributed GPUs and benchmark speculative decoding over internet latency. Draft model runs locally (zero network cost), target verifies over the tunnel.

Clone this wiki locally