Configuration

Configuration

Hydra reads its cluster topology and proxy settings from a YAML config file.

Config Resolution

1. hydra -c /path/to/cluster.yaml (CLI flag)
2. HYDRA_CONFIG environment variable
3. configs/cluster.yaml in the project directory (default)

Full Reference

# ── Speculative Decoding Proxy ──────────────────────────────────

proxy:
  host: 0.0.0.0          # Bind address for proxy API
  port: 8088              # Proxy API port
  max_draft_tokens: 8     # Tokens to draft per round (default: 8)
  fallback_on_draft_failure: true  # Forward to target if draft is down

  draft:
    url: http://192.168.86.250:11434   # Draft server URL
    model_name: qwen3:8b               # Model name (display + Ollama routing)
    backend: ollama                     # "ollama" or "llamacpp"

  target:
    url: http://192.168.86.36:11434    # Target server URL
    model_name: qwen3:32b
    backend: ollama

# ── RPC Cluster ─────────────────────────────────────────────────

# Coordinator — the machine running llama-server
coordinator:
  host: 0.0.0.0        # Bind address for the API
  port: 8080            # API port
  backend: hip          # "hip" (AMD/ROCm) or "cuda" (NVIDIA)
  gpus:                 # Local GPUs (no rpc_port)
    - name: "7900 XTX #0"
      vram_gb: 24
    - name: "7900 XTX #1"
      vram_gb: 24

# Workers — remote machines running rpc-server
workers:
  - host: 192.168.86.36     # Worker IP address
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052      # Port for this GPU's rpc-server
      - name: "RTX 3060"
        vram_gb: 12
        rpc_port: 50053

# Models available for loading (RPC cluster)
models:
  qwen3-72b:                 # Model identifier (used with hydra start -m)
    path: /models/Qwen3-72B-Q4_K_M.gguf
    ctx_size: 8192           # Context window (default: 8192)
    predict: 4096            # Max tokens to predict (default: 4096)
    flash_attn: true         # Enable flash attention (default: true)
    default: true            # Load this model by default

  deepseek-r1-70b:
    path: /models/DeepSeek-R1-Distill-Qwen-70B-Q4_K_M.gguf
    ctx_size: 8192
    predict: 4096
    flash_attn: true

# Binary paths (RPC cluster)
binaries:
  coordinator: /usr/local/bin/llama-server
  rpc_server: rpc-server.exe

Proxy Config

The proxy section is optional. If omitted, hydra proxy commands will show an error and the RPC cluster commands work as before.

proxy.draft / proxy.target

Each server endpoint has:

Field	Required	Description
url	Yes	Full URL including port (e.g., http://192.168.86.250:11434)
model_name	Yes	Model identifier. For Ollama, use the exact tag (qwen3:8b, not qwen3-8b)
backend	No	"ollama" or "llamacpp" (default: "llamacpp")

proxy.max_draft_tokens

Number of tokens the draft model generates per speculation round. Higher values mean more potential speedup but lower acceptance rates if the models diverge. Default: 8.

proxy.fallback_on_draft_failure

When true (default), the proxy forwards requests directly to the target if the draft server is unreachable. The proxy stays up but runs at target-only speed.

RPC Cluster Config

coordinator.gpus[].vram_gb

Used to calculate tensor split ratios. Doesn't need to be exact — Hydra uses these as proportions, not absolute limits. Round to the nearest GB.

workers[].gpus[].rpc_port

Each GPU on a worker needs its own rpc-server instance on a unique port. Standard convention: start at 50052 and increment.

models[].path

Absolute path to the GGUF file on the coordinator machine. Workers don't need the model file — they only receive tensor data over RPC.

models[].default

Mark one model as default: true. This is loaded when running hydra start without -m.

Multiple Workers

workers:
  - host: 192.168.86.36
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052
  - host: 192.168.86.50
    gpus:
      - name: "RTX 3090"
        vram_gb: 24
        rpc_port: 50052

Tensor split is calculated across all GPUs in order: coordinator locals first, then workers top-to-bottom.

Environment Variables

Variable	Description
HYDRA_CONFIG	Path to cluster.yaml