Skip to content

Pipeline Parallelism

Jump to bottom
youngharold edited this page Feb 17, 2026 · 4 revisions

Pipeline Parallelism

Why Not True GGUF Layer Sharding?

We investigated splitting GGUF files by transformer block and distributing partial files to machines, so no single machine needs RAM for the full model. This turned out to be not feasible with the llama.cpp ecosystem:

Constraints

  1. llama-server requires the full GGUF file. It loads the file header, all tensor metadata, and builds the full computation graph at startup. There's no --layer-start/--layer-end flag — it expects embedding, all transformer blocks, and the output head.

  2. No hidden-state API. Even if you could load partial models, llama-server doesn't expose intermediate hidden states between layers. You can't chain two instances where one runs layers 0-31 and feeds hidden states to another running layers 32-63.

  3. Exo does pipeline parallelism by partitioning models across machines, but it only supports Apple Silicon and NVIDIA CUDA (no Windows, no ROCm). This rules out our 4070 desktop (Windows) and the incoming 7900 XTX rig (ROCm).

  4. llama.cpp's RPC backend already distributes layers. The rpc-server instances on worker machines receive individual GGML tensor operations over TCP — the coordinator handles the layer assignment via --tensor-split. This IS distributed inference, just not pipeline parallelism.

The Real Bottleneck

The only limitation of the RPC approach: the coordinator must mmap the full GGUF into system RAM. For a Qwen2.5-72B-Instruct Q8_0 model (~75GB), no workstation in our fleet has enough RAM.

Solution: Unraid as Coordinator

The Unraid server has 128GB DDR3 RAM and a Quadro P400 (2GB VRAM). Making it the coordinator:

Property Value
RAM 128 GB (enough for any model mmap)
GPU Quadro P400 (2GB) — gets zero layers
Network Gigabit Ethernet to all machines
OS Unraid (Linux-based)

How It Works

Unraid (coordinator, 128GB RAM)
    │ mmaps full 75GB GGUF → no swap
    │ P400 listed in tensor-split with vram_gb: 0 (gets zero layers)
    │ All compute delegated to RPC workers
    ▼
┌─────────────────┬───────────────────┬──────────────┬────────────────┐
│ Desktop          │ XPS               │ MacBook M2   │ 7900 XTX rig  │
│ 4070 Ti (16GB)  │ 2070 (8GB)        │ Metal (16GB) │ 2x XTX (48GB) │
│ 3060 (12GB)     │                   │              │ (coming soon)  │
│ rpc:50052,50053 │ rpc:50052         │ rpc:50052    │ rpc:50052,53   │
└─────────────────┴───────────────────┴──────────────┴────────────────┘

Why P400 must be listed: llama-server auto-detects the P400 as CUDA device 0. If it's not in the --tensor-split array, the array length doesn't match the device count and distribution breaks. Setting vram_gb: 0 gives it a 0.00 split ratio — zero layers assigned, zero VRAM used.

Config

See configs/cluster-unraid-coord.yaml for the full working config.

GGUF Inspection

Before distributing a model, inspect it to understand RAM requirements and layer distribution:

# Show model metadata, layer count, tensor sizes
tightwad inspect ~/models/Qwen2.5-72B-Instruct-Q8_0.gguf

# Show which layers go to which GPU in your cluster
tightwad inspect ~/models/Qwen2.5-72B-Instruct-Q8_0.gguf --plan

The --plan flag reads your cluster config and maps layers to GPUs based on tensor-split ratios, showing estimated VRAM per GPU.

Requires: pip install tightwad[inspect] (adds the gguf package)

Model Distribution

Copy models to worker machines for local access or caching:

# rsync to all workers that have model_dir configured
tightwad distribute qwen2.5-72b

# Specific target
tightwad distribute qwen2.5-72b -t 192.168.86.36:/models/

# Preview without transferring
tightwad distribute qwen2.5-72b --dry-run

Workers need ssh_user and model_dir in config. Transfers run in parallel via asyncio with Rich progress bars. Prefers rsync (delta transfer, resume) with scp fallback.

Speculation Over the Pool

Once the Unraid coordinator is running with RPC workers, layer the speculative decoding proxy on top for the full Combined Mode experience. See Architecture#Combined Mode and Speculative Decoding for details.

The draft model runs on any junk hardware (even the P400 or a laptop CPU), while the target URL points to the Unraid coordinator. The proxy converts serial autoregressive generation into batch verification, amortizing RPC overhead over 32 tokens per round-trip.

Clone this wiki locally