Home

Hydra-Inference Wiki

Mixed-vendor GPU inference cluster manager with speculative decoding proxy.

Two Modes

1. RPC Cluster — Pool CUDA + ROCm GPUs across machines into a single inference endpoint using llama.cpp RPC
2. Speculative Decoding Proxy — A fast draft model proposes tokens, a large target model verifies them in batch. Output quality identical to the target model alone, 2-3x faster.

Pages

• Architecture — RPC cluster design, data flow, and tensor split calculation
• Speculative Decoding — How the proxy works, verification algorithm, use cases
• Hardware Setup — Building llama.cpp for CUDA/ROCm workers and coordinator
• Configuration — cluster.yaml reference (RPC cluster + proxy config)
• CLI Reference — All commands and options (cluster + proxy)
• Troubleshooting — Common issues and fixes
• Network Optimization — Bandwidth tuning and layer placement
• OpenClaw Integration — Registering Hydra as an OpenClaw provider

Quick Links

• llama.cpp RPC docs
• Cross-vendor RPC confirmation
• Speculative decoding (Google Research)