Skip to content

CLI Reference

Jump to bottom
youngharold edited this page Feb 17, 2026 · 18 revisions

CLI Reference

Global Options

tightwad [OPTIONS] COMMAND
Option Description
-c, --config PATH Path to cluster.yaml (default: configs/cluster.yaml or $TIGHTWAD_CONFIG)

Speculative Decoding Proxy Commands

tightwad proxy start

Start the speculative decoding proxy server. Both draft and target servers must be running.

Single-drafter:

$ tightwad proxy start
Starting speculative decoding proxy...
  Draft:  qwen3-8b @ http://192.168.86.250:8081
  Target: qwen3-32b @ http://192.168.86.36:8080
  Max draft tokens: 32
  Listening on: 0.0.0.0:8088

Multi-drafter:

$ tightwad proxy start
Starting speculative decoding proxy...
  Drafters (3):
    - qwen3-8b @ http://192.168.86.250:8081 (llamacpp)
    - qwen3-1.7b @ http://192.168.86.28:8081 (llamacpp)
    - qwen3:1.7b @ http://192.168.86.86:11434 (ollama)
  Target: qwen3-32b @ http://192.168.86.36:8080
  Max draft tokens: 32
  Listening on: 0.0.0.0:8088

The proxy runs in the foreground (uvicorn). PID is written to ~/.tightwad/proxy.pid.

tightwad proxy stop

Stop the speculative decoding proxy.

$ tightwad proxy stop
Proxy stopped.

tightwad proxy status

Show server health, drafter wins, and speculation stats.

Single-drafter:

$ tightwad proxy status
  ● Draft: qwen3-8b @ http://192.168.86.250:8081
  ● Target: qwen3-32b @ http://192.168.86.36:8080

              Speculation Stats
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Metric           ┃ Value     ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Rounds           │ 39        │
│ Drafted          │ 1247      │
│ Accepted         │ 759       │
│ Acceptance rate  │ 60.9%     │
│ Tokens/round     │ 33.5      │
│ Uptime           │ 120s      │
└──────────────────┴───────────┘

Multi-drafter:

$ tightwad proxy status
  Drafters:
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Model       ┃ URL                           ┃ Backend ┃ Health ┃ Wins ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━┩
│ qwen3-8b    │ http://192.168.86.250:8081    │ llamacpp│ alive  │ 6    │
│ qwen3-1.7b  │ http://192.168.86.28:8081     │ llamacpp│ alive  │ 4    │
│ qwen3:1.7b  │ http://192.168.86.86:11434    │ ollama  │ alive  │ 0    │
└─────────────┴───────────────────────────────┴─────────┴────────┴──────┘
  ● Target: qwen3-32b @ http://192.168.86.36:8080

RPC Cluster Commands

tightwad status

Show full cluster status: coordinator state, worker health, VRAM totals, and tensor split.

$ tightwad status
● Coordinator PID 12345 on :8080
  Health: OK

               RPC Workers
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┓
┃ Address             ┃ Status┃ Latency ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━┩
│ 192.168.86.36:50052 │ alive │ 0.8ms   │
│ 192.168.86.36:50053 │ alive │ 0.9ms   │
└─────────────────────┴───────┴─────────┘

Total VRAM: 76 GB across 4 GPUs
Tensor split: [0.32, 0.32, 0.21, 0.16]
Models: qwen3-72b, deepseek-r1-70b, llama3.3-70b

tightwad start

Start the coordinator llama-server. Checks all RPC workers are healthy before launching.

tightwad start [-m MODEL]
Option Description
-m, --model NAME Model name from config (default: model with default: true)

Prerequisites: All rpc-server instances must be running on workers.

tightwad stop

Stop the coordinator llama-server. RPC workers are not affected.

tightwad stop

tightwad swap MODEL

Hot-swap to a different model. Stops the coordinator and restarts with the new model. RPC workers persist since they're stateless.

tightwad swap deepseek-r1-70b

tightwad benchmark

Run a prompt-processing and text-generation benchmark against the running coordinator.

tightwad benchmark

Measures:

  • Prompt processing speed (tok/s) with ~512 token prompt
  • Text generation speed (tok/s) with 128 token completion

Requires the coordinator to be running.

Clone this wiki locally