Skip to content

CLI Reference

Jump to bottom
youngharold edited this page Feb 17, 2026 · 18 revisions

CLI Reference

Global Options

hydra [OPTIONS] COMMAND
Option Description
-c, --config PATH Path to cluster.yaml (default: configs/cluster.yaml or $HYDRA_CONFIG)

Speculative Decoding Proxy Commands

hydra proxy start

Start the speculative decoding proxy server. Both draft and target servers must be running.

$ hydra proxy start
Starting speculative decoding proxy...
  Draft:  qwen3:8b @ http://192.168.86.250:11434
  Target: qwen3:32b @ http://192.168.86.36:11434
  Max draft tokens: 8
  Listening on: 0.0.0.0:8088

The proxy runs in the foreground (uvicorn). PID is written to ~/.hydra/proxy.pid.

hydra proxy stop

Stop the speculative decoding proxy.

$ hydra proxy stop
Proxy stopped.

hydra proxy status

Show draft/target server health and speculation stats.

$ hydra proxy status
  ● Draft: qwen3:8b @ http://192.168.86.250:11434
  ● Target: qwen3:32b @ http://192.168.86.36:11434

              Speculation Stats
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Metric           ┃ Value     ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Rounds           │ 39        │
│ Drafted          │ 1247      │
│ Accepted         │ 759       │
│ Acceptance rate  │ 60.9%     │
│ Tokens/round     │ 33.5      │
│ Uptime           │ 120s      │
└──────────────────┴───────────┘

RPC Cluster Commands

hydra status

Show full cluster status: coordinator state, worker health, VRAM totals, and tensor split.

$ hydra status
● Coordinator PID 12345 on :8080
  Health: OK

               RPC Workers
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┓
┃ Address             ┃ Status┃ Latency ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━┩
│ 192.168.86.36:50052 │ alive │ 0.8ms   │
│ 192.168.86.36:50053 │ alive │ 0.9ms   │
└─────────────────────┴───────┴─────────┘

Total VRAM: 76 GB across 4 GPUs
Tensor split: [0.32, 0.32, 0.21, 0.16]
Models: qwen3-72b, deepseek-r1-70b, llama3.3-70b

hydra start

Start the coordinator llama-server. Checks all RPC workers are healthy before launching.

hydra start [-m MODEL]
Option Description
-m, --model NAME Model name from config (default: model with default: true)

Prerequisites: All rpc-server instances must be running on workers.

hydra stop

Stop the coordinator llama-server. RPC workers are not affected.

hydra stop

hydra swap MODEL

Hot-swap to a different model. Stops the coordinator and restarts with the new model. RPC workers persist since they're stateless.

hydra swap deepseek-r1-70b

hydra benchmark

Run a prompt-processing and text-generation benchmark against the running coordinator.

hydra benchmark

Measures:

  • Prompt processing speed (tok/s) with ~512 token prompt
  • Text generation speed (tok/s) with 128 token completion

Requires the coordinator to be running.

Clone this wiki locally