Skip to content

Architecture

Jump to bottom
youngharold edited this page Feb 18, 2026 · 11 revisions

Architecture

Tightwad has three modes: Speculative Decoding Proxy (draft/verify across machines), RPC Cluster (tensor-parallel across machines), and Combined Mode (speculation over a pool — the killer feature). They can run independently or together.

Speculative Decoding Proxy

See Speculative Decoding for full details. The proxy sits between the client and inference servers, coordinating a draft→verify→accept loop that produces output identical to the target model but faster.

Single-drafter mode:

Client → Tightwad Proxy (:8088) → Draft Server (small, fast model)
                              → Target Server (large, accurate model)

Multi-drafter mode:

Client → Tightwad Proxy (:8088) → Drafter 1 (GPU, llamacpp) ──┐
                              → Drafter 2 (CPU, llamacpp) ──┼→ Pick best → Target verifies
                              → Drafter 3 (CPU, ollama)   ──┘
                              → Target Server (large model)

All drafters race in parallel via asyncio.wait(FIRST_COMPLETED) with a short grace period (0.3s for llamacpp, 1s for ollama). Once the first drafter returns, stragglers get a brief window to finish — then remaining tasks are cancelled. The proxy picks the winner (preferring llamacpp backends with per-token logprobs, then most tokens) and sends it to the target for verification. No tensor data crosses the network — only token IDs (bytes). This is the preferred mode when the target model fits on a single machine.

Combined Mode — Speculation Over RPC Pool

When a model is too large for any single machine, combine both modes: pool GPUs via RPC to run the large model, then use speculative decoding on top to overcome RPC's per-token latency.

ANY junk hardware (CPU, 2GB GPU)
    │ runs Qwen3-1.7B draft (local, fast, no network)
    │ sends token IDs (bytes)
    ▼
Tightwad Proxy (:8088)
    │ sends draft to pool for BATCH verification
    ▼
RPC GPU Pool (:8090) — 4070+3060+2070+M2
    │ verifies 32 tokens in ONE forward pass
    │ 1 RPC round-trip for 32 tokens (not 32 round-trips)
    ▼
5.4 tok/s vs 3.0 tok/s pool-only (1.8x speedup)

Why it works: RPC autoregressive generation pays 100-300 MB network cost per token. With speculation, the draft generates 32 tokens locally (zero network), then the pool verifies all 32 in one batch — amortizing the RPC overhead over 32 tokens instead of 1.

Critical requirements:

  • Draft must use llamacpp backend (not Ollama) — Ollama falls back to text-match which provides NO batch speedup
  • Draft must be same model family as target (e.g., Qwen3-1.7B → Qwen3-32B)
  • Target URL points to the RPC pool coordinator, not a single-machine server

Swarm Transfer — P2P Model Distribution

See Swarm Transfer for full details. The swarm protocol distributes GGUF models to workers using BitTorrent-style chunked P2P transfer instead of single-source rsync.

Source machine (seeder, :9080)
    │ serves 64 MB pieces over HTTP
    ├──► Worker 1 (puller) ──► becomes seeder too
    ├──► Worker 2 (puller) ──► becomes seeder too
    │                          │
    │                          └──► Worker 3 (pulls from source + worker 2)
    └──► Worker 4 (puller)

Key components: manifest.py (piece hashing, bitfield tracking), swarm_transfer.py (Starlette seeder + async httpx puller with rarest-first selection).

Docker Deployment

The Docker image packages the proxy as a lightweight container (~100 MB). It's designed for proxy-only mode — users already have Ollama/llama-server running on their machines, Tightwad just coordinates between them.

┌─────────────────────────────────────────────────┐
│  Docker container (tightwad)                    │
│  ┌─────────────────────────────────────────┐    │
│  │  tightwad proxy start                   │    │
│  │  Config from TIGHTWAD_* env vars        │    │
│  │  No YAML file needed                    │    │
│  └────────────┬──────────────┬─────────────┘    │
│               │              │                  │
└───────────────┼──────────────┼──────────────────┘
                │              │
    ┌───────────▼───┐   ┌─────▼──────────┐
    │ Draft server  │   │ Target server  │
    │ (LAN, any hw) │   │ (LAN, big GPU) │
    └───────────────┘   └────────────────┘

Key design decisions:

  • network_mode: host (Linux default) — the proxy needs to reach LAN servers. Bridge networking adds complexity for no benefit.
  • Env var config — zero files to manage. The load_proxy_from_env() fallback builds a ProxyConfig from TIGHTWAD_* env vars when no YAML exists.
  • Proxy-only image — no llama.cpp binaries, no model files, no RPC. Just the Python proxy (~10 lines Dockerfile).

See Configuration for the full env var reference and Mac/Docker Desktop notes.

RPC Cluster

Tightwad manages a llama.cpp inference cluster where one machine acts as coordinator (runs llama-server) and others act as workers (run rpc-server). The coordinator loads the model and distributes transformer layers across all GPUs — local and remote — based on VRAM proportions.

Data Flow

User/App → HTTP API (:8080) → llama-server (coordinator)
                                    │
                              ┌─────┴──────┐
                              │ Model Load  │
                              │ Layer Split │
                              └─────┬──────┘
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
              Local GPU 0     Local GPU 1     Remote GPUs
              (HIP/CUDA)      (HIP/CUDA)     (via RPC/TCP)
  1. The coordinator loads the GGUF model file
  2. Layers are assigned to GPUs according to --tensor-split ratios
  3. For local GPUs, compute happens directly via HIP or CUDA
  4. For remote GPUs, GGML operations are serialized over TCP to rpc-server instances
  5. Results flow back to the coordinator which assembles the final output

Coordinator Placement

There are two valid strategies for choosing the coordinator:

Strategy 1: Most VRAM (Desktop Coordinator)

The machine with the most VRAM coordinates, keeping more layers local with zero network overhead. Good when the model fits in available system RAM.

Example: Desktop with RTX 4070 Ti Super + RTX 3060 (28GB) coordinates; XPS (2070, 8GB) and MacBook M2 (16GB) are RPC workers.

Strategy 2: Most RAM (Unraid Coordinator)

When models are too large for any workstation's RAM to mmap, use the machine with the most system RAM as coordinator — even if its GPU is weak. The coordinator must mmap the full GGUF; the actual GPU compute goes to RPC workers.

Example: Unraid (128GB RAM, P400 2GB) coordinates with vram_gb: 0 — P400 gets zero layers but must be listed in tensor-split since llama-server detects it as CUDA device 0. All compute goes to RPC workers.

Unraid (128GB RAM, coordinator)
    │ mmaps 75GB GGUF, P400 gets 0 layers
    ▼
┌──────────────┬──────────────┬──────────────┐
│ Desktop      │ XPS          │ MacBook M2   │
│ 4070+3060    │ 2070 (8GB)   │ Metal (16GB) │
│ (28GB)       │              │              │
└──────────────┴──────────────┴──────────────┘

See configs/cluster-unraid-coord.yaml and Pipeline Parallelism for details on why this works and how to inspect/distribute models.

Tensor Split Calculation

Tightwad auto-calculates split ratios from VRAM:

Example: 4-GPU pool across 3 machines

GPU VRAM Ratio
RTX 4070 Ti Super (coordinator) 16 GB 0.34
RTX 3060 (coordinator) 12 GB 0.26
RTX 2070 (XPS, RPC) 8 GB 0.17
Apple M2 Metal (Mac, RPC) 11 GB 0.23
Total 47 GB 1.00

This means ~60% of layers stay on the coordinator (local) and ~40% go over the network.

Apple Silicon VRAM

For Mac rpc-servers, use recommendedMaxWorkingSetSize (shown at rpc-server startup) rather than total unified memory. An M2 with 16GB RAM reports ~11GB available for Metal. Using the full 16GB will cause allocation failures.

RPC Protocol

The llama.cpp RPC backend:

  • Serializes individual GGML tensor operations (matmul, softmax, etc.) over TCP
  • Each rpc-server instance manages one or more devices — use --device to restrict which are exposed
  • The protocol is vendor-agnostic — the coordinator doesn't care if a worker uses CUDA, HIP, or Metal
  • Memory allocation, data transfer, and compute are all handled transparently
  • Version matching is critical — coordinator and all rpc-servers must be the exact same llama.cpp build. Mismatched versions fail silently (no error, tensors just don't distribute)

Mac rpc-server Gotcha: Dual Device

By default, macOS rpc-server exposes two backends: Metal GPU and CPU. This registers as two RPC devices on the coordinator, throwing off --tensor-split calculations.

Fix: Use --device MTL0 to expose only the Metal GPU:

rpc-server --host 0.0.0.0 --port 50052 --device MTL0

Run rpc-server --help or start without --device to see the "No devices found" error which lists available device names (e.g., MTL0, BLAS, CPU).

Model Hot-Swap

tightwad swap <model> stops the coordinator and restarts with a different model. RPC workers are stateless (they just expose GPU compute) so they don't need to restart. This makes model switching fast — only the coordinator needs to reload.

PID Management

Component PID File
RPC Coordinator ~/.tightwad/coordinator.pid
Speculative Proxy ~/.tightwad/proxy.pid
Swarm Seeder ~/.tightwad/swarm-<model>.pid

All can run simultaneously. The CLI checks PID files for start/stop/status operations. If a process dies unexpectedly, stale PID files are cleaned up automatically on the next command.

Clone this wiki locally