Home

Tightwad-Inference Wiki

Mixed-vendor GPU inference cluster manager with speculative decoding proxy.

Three Modes

1. RPC Cluster — Pool CUDA + ROCm GPUs across machines into a single inference endpoint using llama.cpp RPC
2. Speculative Decoding Proxy — A fast draft model proposes tokens, a large target model verifies them in batch. Output quality identical to the target model alone, 2-3x faster.
3. Swarm Transfer — BitTorrent-style P2P model distribution. Split GGUF files into 64 MB pieces with SHA256 hashes, pull from any peer that has pieces.

Pages

• Architecture — RPC cluster design, data flow, and tensor split calculation
• Speculative Decoding — How the proxy works, verification algorithm, use cases
• Hardware Setup — Building llama.cpp for CUDA/ROCm workers and coordinator
• Configuration — cluster.yaml reference (RPC cluster + proxy config)
• CLI Reference — All commands and options (cluster + proxy)
• Swarm Transfer — P2P model distribution (manifest, seeder, puller)
• Troubleshooting — Common issues and fixes
• Network Optimization — Bandwidth tuning and layer placement
• OpenClaw Integration — Registering Tightwad as an OpenClaw provider

Quick Links

• llama.cpp RPC docs
• Cross-vendor RPC confirmation
• Speculative decoding (Google Research)