Architecture

Architecture

Tightwad has three modes: Speculative Decoding Proxy (draft/verify across machines), RPC Cluster (tensor-parallel across machines), and Combined Mode (speculation over a pool — the killer feature). They can run independently or together.

Speculative Decoding Proxy

See Speculative Decoding for full details. The proxy sits between the client and inference servers, coordinating a draft→verify→accept loop that produces output identical to the target model but faster.

Single-drafter mode:

Client → Tightwad Proxy (:8088) → Draft Server (small, fast model)
                              → Target Server (large, accurate model)

Multi-drafter mode:

Client → Tightwad Proxy (:8088) → Drafter 1 (GPU, llamacpp) ──┐
                              → Drafter 2 (CPU, llamacpp) ──┼→ Pick best → Target verifies
                              → Drafter 3 (CPU, ollama)   ──┘
                              → Target Server (large model)

All drafters race in parallel via asyncio.wait(FIRST_COMPLETED) with a short grace period (0.3s for llamacpp, 1s for ollama). Once the first drafter returns, stragglers get a brief window to finish — then remaining tasks are cancelled. The proxy picks the winner (preferring llamacpp backends with per-token logprobs, then most tokens) and sends it to the target for verification. No tensor data crosses the network — only token IDs (bytes). This is the preferred mode when the target model fits on a single machine.

Combined Mode — Speculation Over RPC Pool

When a model is too large for any single machine, combine both modes: pool GPUs via RPC to run the large model, then use speculative decoding on top to overcome RPC's per-token latency.

ANY junk hardware (CPU, 2GB GPU)
    │ runs Qwen3-1.7B draft (local, fast, no network)
    │ sends token IDs (bytes)
    ▼
Tightwad Proxy (:8088)
    │ sends draft to pool for BATCH verification
    ▼
RPC GPU Pool (:8090) — 4070+3060+2070+M2
    │ verifies 32 tokens in ONE forward pass
    │ 1 RPC round-trip for 32 tokens (not 32 round-trips)
    ▼
5.4 tok/s vs 3.0 tok/s pool-only (1.8x speedup)

Why it works: RPC autoregressive generation pays 100-300 MB network cost per token. With speculation, the draft generates 32 tokens locally (zero network), then the pool verifies all 32 in one batch — amortizing the RPC overhead over 32 tokens instead of 1.

Critical requirements:

• Draft must use llamacpp backend (not Ollama) — Ollama falls back to text-match which provides NO batch speedup
• Draft must be same model family as target (e.g., Qwen3-1.7B → Qwen3-32B)
• Target URL points to the RPC pool coordinator, not a single-machine server

Swarm Transfer — P2P Model Distribution

See Swarm Transfer for full details. The swarm protocol distributes GGUF models to workers using BitTorrent-style chunked P2P transfer instead of single-source rsync.

Source machine (seeder, :9080)
    │ serves 64 MB pieces over HTTP
    ├──► Worker 1 (puller) ──► becomes seeder too
    ├──► Worker 2 (puller) ──► becomes seeder too
    │                          │
    │                          └──► Worker 3 (pulls from source + worker 2)
    └──► Worker 4 (puller)

Key components: manifest.py (piece hashing, bitfield tracking), swarm_transfer.py (Starlette seeder + async httpx puller with rarest-first selection).

Docker Deployment

The Docker image packages the proxy as a lightweight container (~100 MB). It's designed for proxy-only mode — users already have Ollama/llama-server running on their machines, Tightwad just coordinates between them.

┌─────────────────────────────────────────────────┐
│  Docker container (tightwad)                    │
│  ┌─────────────────────────────────────────┐    │
│  │  tightwad proxy start                   │    │
│  │  Config from TIGHTWAD_* env vars        │    │
│  │  No YAML file needed                    │    │
│  └────────────┬──────────────┬─────────────┘    │
│               │              │                  │
└───────────────┼──────────────┼──────────────────┘
                │              │
    ┌───────────▼───┐   ┌─────▼──────────┐
    │ Draft server  │   │ Target server  │
    │ (LAN, any hw) │   │ (LAN, big GPU) │
    └───────────────┘   └────────────────┘

Key design decisions:

• network_mode: host (Linux default) — the proxy needs to reach LAN servers. Bridge networking adds complexity for no benefit.
• Env var config — zero files to manage. The load_proxy_from_env() fallback builds a ProxyConfig from TIGHTWAD_* env vars when no YAML exists.
• Proxy-only image — no llama.cpp binaries, no model files, no RPC. Just the Python proxy (~10 lines Dockerfile).
• Healthcheck — python -c "import httpx; ..." hits /v1/models every 10s (curl not in python:3.12-slim, httpx is already installed). Compose marks the container unhealthy after 5 consecutive failures.
• Persistent logs — ./logs:/root/.tightwad/logs volume mount keeps proxy logs across container restarts. Without this, logs vanish when the container is recreated.

See Configuration for the full env var reference and Mac/Docker Desktop notes.

RPC Cluster

Tightwad manages a llama.cpp inference cluster where one machine acts as coordinator (runs llama-server) and others act as workers (run rpc-server). The coordinator loads the model and distributes transformer layers across all GPUs — local and remote — based on VRAM proportions.

Data Flow

User/App → HTTP API (:8080) → llama-server (coordinator)
                                    │
                              ┌─────┴──────┐
                              │ Model Load  │
                              │ Layer Split │
                              └─────┬──────┘
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
              Local GPU 0     Local GPU 1     Remote GPUs
              (HIP/CUDA)      (HIP/CUDA)     (via RPC/TCP)

1. The coordinator loads the GGUF model file
2. Layers are assigned to GPUs according to --tensor-split ratios
3. For local GPUs, compute happens directly via HIP or CUDA
4. For remote GPUs, GGML operations are serialized over TCP to rpc-server instances
5. Results flow back to the coordinator which assembles the final output

Coordinator Placement

There are two valid strategies for choosing the coordinator:

Strategy 1: Most VRAM (Desktop Coordinator)

The machine with the most VRAM coordinates, keeping more layers local with zero network overhead. Good when the model fits in available system RAM.

Example: Desktop with RTX 4070 Ti Super + RTX 3060 (28GB) coordinates; XPS (2070, 8GB) and MacBook M2 (16GB) are RPC workers.

Strategy 2: Most RAM (Unraid Coordinator)

When models are too large for any workstation's RAM to mmap, use the machine with the most system RAM as coordinator — even if its GPU is weak. The coordinator must mmap the full GGUF; the actual GPU compute goes to RPC workers.

Example: Unraid (128GB RAM, P400 2GB) coordinates with vram_gb: 0 — P400 gets zero layers but must be listed in tensor-split since llama-server detects it as CUDA device 0. All compute goes to RPC workers.

Unraid (128GB RAM, coordinator)
    │ mmaps 75GB GGUF, P400 gets 0 layers
    ▼
┌──────────────┬──────────────┬──────────────┐
│ Desktop      │ XPS          │ MacBook M2   │
│ 4070+3060    │ 2070 (8GB)   │ Metal (16GB) │
│ (28GB)       │              │              │
└──────────────┴──────────────┴──────────────┘

See configs/cluster-unraid-coord.yaml and Pipeline Parallelism for details on why this works and how to inspect/distribute models.

Tensor Split Calculation

Tightwad auto-calculates split ratios from VRAM:

Example: 4-GPU pool across 3 machines

GPU	VRAM	Ratio
RTX 4070 Ti Super (coordinator)	16 GB	0.34
RTX 3060 (coordinator)	12 GB	0.26
RTX 2070 (XPS, RPC)	8 GB	0.17
Apple M2 Metal (Mac, RPC)	11 GB	0.23
Total	47 GB	1.00

This means ~60% of layers stay on the coordinator (local) and ~40% go over the network.

Apple Silicon VRAM

For Mac rpc-servers, use recommendedMaxWorkingSetSize (shown at rpc-server startup) rather than total unified memory. An M2 with 16GB RAM reports ~11GB available for Metal. Using the full 16GB will cause allocation failures.

RPC Protocol

The llama.cpp RPC backend:

• Serializes individual GGML tensor operations (matmul, softmax, etc.) over TCP
• Each rpc-server instance manages one or more devices — use --device to restrict which are exposed
• The protocol is vendor-agnostic — the coordinator doesn't care if a worker uses CUDA, HIP, or Metal
• Memory allocation, data transfer, and compute are all handled transparently
• Version matching is critical — coordinator and all rpc-servers must be the exact same llama.cpp build. Mismatched versions fail silently (no error, tensors just don't distribute)

Mac rpc-server Gotcha: Dual Device

By default, macOS rpc-server exposes two backends: Metal GPU and CPU. This registers as two RPC devices on the coordinator, throwing off --tensor-split calculations.

Fix: Use --device MTL0 to expose only the Metal GPU:

rpc-server --host 0.0.0.0 --port 50052 --device MTL0

Run rpc-server --help or start without --device to see the "No devices found" error which lists available device names (e.g., MTL0, BLAS, CPU).

Model Hot-Swap

tightwad swap <model> stops the coordinator and restarts with a different model. RPC workers are stateless (they just expose GPU compute) so they don't need to restart. This makes model switching fast — only the coordinator needs to reload.

Live Dashboard

The proxy includes a built-in web dashboard at /dashboard — always available when the proxy is running, no extra process or port needed.

GET /dashboard          → Single-page HTML dashboard
GET /v1/tightwad/events → SSE stream (stats + health every 2s)
GET /v1/tightwad/history → JSON of recent request records (max 50)

Layout:

┌──────────────────────────────────────────────┐
│ TIGHTWAD DASHBOARD               uptime: 2h  │
├─────────────────────┬────────────────────────┤
│ Server Health       │ Speculation Stats      │
│ ● Target: alive     │ ╭─────────────────╮    │
│ ● Draft: alive      │ │  acceptance rate │    │
│   Wins: 42          │ │  SVG line chart  │    │
│                     │ ╰─────────────────╯    │
├─────────────────────┴────────────────────────┤
│ Request Log                                   │
│ Time │Rounds│Drafted│Accepted│Rate│Draft/Verify│
│ ...  │ ...  │ ...   │ ...    │... │ ...ms      │
└──────────────────────────────────────────────┘

Features:

• Server health cards — colored dots (green/red) for target and all drafters, model names, backend type, drafter win counts. Updated every 2s via SSE.
• Live SVG charts — acceptance rate (green, 0-100%) and tokens/round (blue) as rolling 2-minute polylines. Pure SVG, no CDN dependencies.
• Request log table — per-request timing breakdown (draft_ms, verify_ms, total_ms), color-coded acceptance rate cells (green >70%, yellow 40-70%, red <40%). Capped at 50 rows, newest first.
• SSE auto-reconnect — uses EventSource API with built-in reconnect. Dashboard shows "reconnecting..." on connection loss.

Zero new dependencies. Dashboard is served by the existing Starlette app using starlette.responses.HTMLResponse and StreamingResponse. The HTML/CSS/JS is a single inline constant (~250 lines).

Implementation: tightwad/dashboard.py provides three handlers registered in create_app(). Request timing is captured via RequestRecord dataclass (ring buffer of 50) with per-round draft_ms/verify_ms accumulated across the speculation loop.

PID Management

Component	PID File
RPC Coordinator	~/.tightwad/coordinator.pid
Speculative Proxy	~/.tightwad/proxy.pid
Swarm Seeder	~/.tightwad/swarm-<model>.pid

All can run simultaneously. The CLI checks PID files for start/stop/status operations. If a process dies unexpectedly, stale PID files are cleaned up automatically on the next command.