Skip to content

Architecture

Jump to bottom
youngharold edited this page Feb 17, 2026 · 11 revisions

Architecture

Tightwad has two modes: RPC Cluster (tensor-parallel across machines) and Speculative Decoding Proxy (draft/verify across machines). They can run independently or together.

Speculative Decoding Proxy

See Speculative Decoding for full details. The proxy sits between the client and inference servers, coordinating a draft→verify→accept loop that produces output identical to the target model but faster.

Single-drafter mode:

Client → Tightwad Proxy (:8088) → Draft Server (small, fast model)
                              → Target Server (large, accurate model)

Multi-drafter mode:

Client → Tightwad Proxy (:8088) → Drafter 1 (GPU, llamacpp) ──┐
                              → Drafter 2 (CPU, llamacpp) ──┼→ Pick best → Target verifies
                              → Drafter 3 (CPU, ollama)   ──┘
                              → Target Server (large model)

All drafters race in parallel via asyncio.gather. The proxy picks the winner (most tokens with logprobs) and sends it to the target for verification. No tensor data crosses the network — only token IDs (bytes). This is the preferred mode when the target model fits on a single machine.

RPC Cluster

Tightwad manages a llama.cpp inference cluster where one machine acts as coordinator (runs llama-server) and others act as workers (run rpc-server). The coordinator loads the model and distributes transformer layers across all GPUs — local and remote — based on VRAM proportions.

Data Flow

User/App → HTTP API (:8080) → llama-server (coordinator)
                                    │
                              ┌─────┴──────┐
                              │ Model Load  │
                              │ Layer Split │
                              └─────┬──────┘
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
              Local GPU 0     Local GPU 1     Remote GPUs
              (HIP/CUDA)      (HIP/CUDA)     (via RPC/TCP)
  1. The coordinator loads the GGUF model file
  2. Layers are assigned to GPUs according to --tensor-split ratios
  3. For local GPUs, compute happens directly via HIP or CUDA
  4. For remote GPUs, GGML operations are serialized over TCP to rpc-server instances
  5. Results flow back to the coordinator which assembles the final output

Why Coordinator Has Most VRAM

The coordinator should be the machine with the most VRAM because:

  • Local layers have zero network overhead — keeping more layers local means less data on the wire
  • Prompt processing is bandwidth-heavy — the coordinator processes prompts through local layers first, only sending intermediate tensors to remote GPUs for their assigned layers
  • Token generation is lighter on bandwidth — once the KV cache is populated, per-token network traffic is smaller

In our topology, the ROCm machine has 48GB (2x 7900 XTX) vs 28GB on the Windows desktop, so the ROCm machine coordinates.

Tensor Split Calculation

Tightwad auto-calculates split ratios from VRAM:

GPU VRAM Ratio
7900 XTX #0 24 GB 0.32
7900 XTX #1 24 GB 0.32
RTX 4070 Ti Super 16 GB 0.21
RTX 3060 12 GB 0.16
Total 76 GB 1.00

This means ~64% of layers stay on the coordinator (local) and ~36% go over the network.

RPC Protocol

The llama.cpp RPC backend:

  • Serializes individual GGML tensor operations (matmul, softmax, etc.) over TCP
  • Each rpc-server instance manages one GPU
  • The protocol is vendor-agnostic — the coordinator doesn't care if a worker uses CUDA, HIP, or Metal
  • Memory allocation, data transfer, and compute are all handled transparently

Model Hot-Swap

tightwad swap <model> stops the coordinator and restarts with a different model. RPC workers are stateless (they just expose GPU compute) so they don't need to restart. This makes model switching fast — only the coordinator needs to reload.

PID Management

Component PID File
RPC Coordinator ~/.tightwad/coordinator.pid
Speculative Proxy ~/.tightwad/proxy.pid

Both can run simultaneously. The CLI checks PID files for start/stop/status operations. If a process dies unexpectedly, stale PID files are cleaned up automatically on the next command.

Clone this wiki locally