Home

Tightwad-Inference Wiki

Mixed-vendor GPU inference cluster manager with speculative decoding proxy.

Getting Started

Fastest path: Docker with env vars (no config files):

docker run --rm --network host \
  -e TIGHTWAD_DRAFT_URL=http://192.168.1.10:11434 \
  -e TIGHTWAD_TARGET_URL=http://192.168.1.20:11434 \
  ghcr.io/akivasolutions/tightwad

Or install and auto-discover servers:

pip install -e .
tightwad init    # scans LAN, finds Ollama/llama-server, generates config
tightwad proxy start

Three Modes

1. RPC Cluster — Pool CUDA + ROCm GPUs across machines into a single inference endpoint using llama.cpp RPC
2. Speculative Decoding Proxy — A fast draft model proposes tokens, a large target model verifies them in batch. Output quality identical to the target model alone, 2-3x faster.
3. Swarm Transfer — BitTorrent-style P2P model distribution. Split GGUF files into 64 MB pieces with SHA256 hashes, pull from any peer that has pieces.

Pages

• Architecture — RPC cluster design, data flow, Docker deployment, and tensor split calculation
• Speculative Decoding — How the proxy works, verification algorithm, use cases
• Hardware Setup — Building llama.cpp for CUDA/ROCm workers and coordinator
• Configuration — cluster.yaml reference, env var config, Docker deployment
• CLI Reference — All commands and options (init, proxy, cluster, swarm)
• Swarm Transfer — P2P model distribution (manifest, seeder, puller)
• Benchmarking — Benchmark scripts, methodology, and published results
• Troubleshooting — Common issues and fixes (proxy, RPC, Docker, init wizard)
• Network Optimization — Bandwidth tuning and layer placement
• OpenClaw Integration — Registering Tightwad as an OpenClaw provider

Quick Links

• llama.cpp RPC docs
• Cross-vendor RPC confirmation
• Speculative decoding (Google Research)