Configuration

Configuration

Hydra reads its cluster topology from a YAML config file.

Config Resolution

1. hydra -c /path/to/cluster.yaml (CLI flag)
2. HYDRA_CONFIG environment variable
3. configs/cluster.yaml in the project directory (default)

Full Reference

# Coordinator — the machine running llama-server
coordinator:
  host: 0.0.0.0        # Bind address for the API
  port: 8080            # API port
  backend: hip          # "hip" (AMD/ROCm) or "cuda" (NVIDIA)
  gpus:                 # Local GPUs (no rpc_port)
    - name: "7900 XTX #0"
      vram_gb: 24
    - name: "7900 XTX #1"
      vram_gb: 24

# Workers — remote machines running rpc-server
workers:
  - host: 192.168.86.36     # Worker IP address
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052      # Port for this GPU's rpc-server
      - name: "RTX 3060"
        vram_gb: 12
        rpc_port: 50053

# Models available for loading
models:
  qwen3-72b:                 # Model identifier (used with hydra start -m)
    path: /models/Qwen3-72B-Q4_K_M.gguf
    ctx_size: 8192           # Context window (default: 8192)
    predict: 4096            # Max tokens to predict (default: 4096)
    flash_attn: true         # Enable flash attention (default: true)
    default: true            # Load this model by default

  deepseek-r1-70b:
    path: /models/DeepSeek-R1-Distill-Qwen-70B-Q4_K_M.gguf
    ctx_size: 8192
    predict: 4096
    flash_attn: true

# Binary paths
binaries:
  coordinator: /usr/local/bin/llama-server
  rpc_server: rpc-server.exe

Key Fields

coordinator.gpus[].vram_gb

Used to calculate tensor split ratios. Doesn't need to be exact — Hydra uses these as proportions, not absolute limits. Round to the nearest GB.

workers[].gpus[].rpc_port

Each GPU on a worker needs its own rpc-server instance on a unique port. Standard convention: start at 50052 and increment.

models[].path

Absolute path to the GGUF file on the coordinator machine. Workers don't need the model file — they only receive tensor data over RPC.

models[].default

Mark one model as default: true. This is loaded when running hydra start without -m.

Multiple Workers

You can add multiple worker machines:

workers:
  - host: 192.168.86.36
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052
  - host: 192.168.86.50
    gpus:
      - name: "RTX 3090"
        vram_gb: 24
        rpc_port: 50052

Tensor split is calculated across all GPUs in order: coordinator locals first, then workers top-to-bottom.

Environment Variables

Variable	Description
HYDRA_CONFIG	Path to cluster.yaml