Skip to content

CLI Reference

Jump to bottom
youngharold edited this page Mar 28, 2026 · 18 revisions

CLI Reference

Global Options

tightwad [OPTIONS] COMMAND
Option Description
-c, --config PATH Path to cluster.yaml (default: configs/cluster.yaml or $TIGHTWAD_CONFIG)

Setup

tightwad init

Auto-discover inference servers on your LAN and generate a cluster.yaml config. Supports both interactive (LAN scan) and non-interactive (direct URL) modes.

tightwad init [--subnet 192.168.1.0/24] [--port 8082] [-o configs/cluster.yaml]
              [--draft-url URL] [--draft-model NAME] [--draft-backend TYPE]
              [--target-url URL] [--target-model NAME] [--target-backend TYPE]
              [--max-draft-tokens N] [-y]
Option Description
--subnet Manual subnet override (auto-detected from your network interface if omitted)
--port Additional ports to scan, repeatable (default: 11434, 8080, 8081)
-o, --output PATH Output config path (default: configs/cluster.yaml)
--draft-url URL Draft server URL — enables non-interactive mode
--draft-model NAME Draft model name (required with --draft-url)
--draft-backend TYPE Draft backend: ollama or llamacpp (auto-detected from port if omitted)
--target-url URL Target server URL — enables non-interactive mode
--target-model NAME Target model name (required with --target-url)
--target-backend TYPE Target backend: ollama or llamacpp (auto-detected from port if omitted)
--max-draft-tokens N Max draft tokens (default: 32)
-y, --yes Skip all confirmation prompts (overwrite existing files)

Non-interactive mode (when --draft-url and --target-url are both provided):

  • Skips LAN scanning entirely
  • Requires --draft-model and --target-model
  • Auto-detects backend from port (11434 → ollama, else → llamacpp), overridable with explicit flags
  • If output file exists: errors unless -y is set
  • Designed for AI agents, CI/CD, and scripted setups
# Agent-friendly one-liner
tightwad init \
  --draft-url http://192.168.86.250:11434 \
  --draft-model qwen3:8b \
  --target-url http://192.168.86.36:11434 \
  --target-model qwen3:32b \
  -y

# Explicit backends + custom draft tokens
tightwad init \
  --draft-url http://192.168.86.250:8081 \
  --draft-model qwen3-8b \
  --draft-backend llamacpp \
  --target-url http://192.168.86.36:8080 \
  --target-model qwen3-32b \
  --target-backend llamacpp \
  --max-draft-tokens 64 \
  -y -o configs/cluster.yaml

Interactive mode (default, when URLs are not provided):

What it does:

  1. Scans your /24 subnet for open ports (11434, 8080, 8081)
  2. Identifies Ollama and llama-server instances
  3. Queries each server for loaded models
  4. Presents a table of discovered servers
  5. Prompts you to pick target (big model) and draft (small fast model)
  6. Generates and writes cluster.yaml 
$ tightwad init
Scanning LAN for inference servers...

          Discovered Servers (192.168.86.0/24)
┏━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ # ┃ Host           ┃ Port  ┃ Backend ┃ Models        ┃ Status  ┃
┡━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ 1 │ 192.168.86.36  │ 11434 │ ollama  │ qwen3:32b     │ healthy │
│ 2 │ 192.168.86.250 │ 11434 │ ollama  │ qwen3:8b      │ healthy │
│ 3 │ 192.168.86.36  │ 8080  │ llamacpp│ glm-4.7-flash │ healthy │
└───┴────────────────┴───────┴─────────┴───────────────┴─────────┘
Scan completed in 4.2s

Select TARGET server:
  1) 192.168.86.36:11434 (ollama)
  2) 192.168.86.250:11434 (ollama)
  3) 192.168.86.36:8080 (llamacpp)
  Enter number (1-3): 1

Select DRAFT server:
  1) 192.168.86.36:11434 (ollama)
  2) 192.168.86.250:11434 (ollama)
  3) 192.168.86.36:8080 (llamacpp)
  Enter number (1-3): 2

Generated config:
proxy:
  host: 0.0.0.0
  port: 8088
  ...

Write to configs/cluster.yaml? [Y/n] y

Config written to configs/cluster.yaml

Next steps:
  tightwad doctor -c configs/cluster.yaml
  tightwad proxy start -c configs/cluster.yaml

Asks for overwrite confirmation if the output file already exists.

Diagnostics

tightwad doctor

Run a full diagnostic check of your cluster setup: configuration, binaries, model files, network connectivity, running services, version matching, and model family compatibility.

tightwad doctor [--fix] [--json]
Option Description
--fix Show suggested fix commands for each failure or warning
--json Output machine-readable JSON report (for scripting)

Exit code: 0 if all checks pass, 1 if any check fails.

$ tightwad doctor --fix
──────────────────────────────── Configuration ─────────────────────────────────
  ✓ Config file  /home/user/tightwad/configs/cluster.yaml
  ✓ YAML parse
  ✓ Models defined  2 model(s): qwen3-32b, llama3.3-70b
  ✓ Workers defined  2 worker(s), 2 remote GPU(s)

─────────────────────────────────── Binaries ───────────────────────────────────
  ✓ Coordinator binary  llama-server → /usr/local/bin/llama-server
  ✗ RPC server binary  rpc-server.exe not found
      → Install llama.cpp and ensure 'rpc-server.exe' is in PATH or use an absolute path in config

──────────────────────────────────── Models ────────────────────────────────────
  ○ Model: qwen3-32b  C:/Users/jarvis/models/Qwen3-32B-Q4_K_M.gguf (remote path, not checkable locally)
  ✓ Tensor split  [0.31, 0.23, 0.15, 0.31] (52 GB total across 4 GPUs)

─────────────────────────────────── Network ────────────────────────────────────
  ✓ RPC 192.168.86.250:50052  RTX 2070 — 1.2ms
  ! RPC 192.168.86.28:50052  Apple M2 Metal — 15.7ms (>10ms, WiFi?)
      → Use wired Ethernet for best RPC performance

─────────────────────────────────── Services ───────────────────────────────────
  ✓ Coordinator process  PID 54321
  ✓ Coordinator /health  http://127.0.0.1:8090/health
  ○ Proxy process  Not started (tightwad proxy start)

─────────────────────────────────── Versions ───────────────────────────────────
  ✓ Local llama-server  version: 1.2.3 (abc1234)
  ✓ Worker 192.168.86.250  version: 1.2.3 (abc1234)
  ○ Worker 192.168.86.28  No ssh_user configured — cannot check remote version

1 check(s) failed.  (10 passed, 1 failed, 1 warnings, 3 skipped)
Run with --fix to see suggested fixes.

Check sections (in order):

Section What it checks
Configuration Config file exists, YAML parses, models and workers defined, structural validation (port ranges, VRAM positive, valid URLs, backend enum, max_draft_tokens range, duplicate RPC addresses)
Binaries coordinator_binary and rpc_server_binary findable via PATH or absolute path
Models Model files exist locally (Windows paths on Mac → SKIP), tensor split sums to ~1.0, MoE VRAM warnings
Network TCP ping each worker RPC port, latency >10ms warns about WiFi
Services PID files for coordinator/proxy, HTTP health endpoints, draft/target reachability
Versions Local llama-server --version, SSH/peer to workers for remote version, mismatch errors (FAIL)
Peer Agents Probe each worker's peer agent (when peer_port configured), check reachability and version
Model Families Detect draft/target architecture via Ollama /api/show or llama-server /props, FAIL on mismatch

JSON output (--json) returns a structured report suitable for CI or monitoring:

tightwad doctor --json | python -m json.tool
{
  "passed": false,
  "sections": [
    {
      "title": "Configuration",
      "results": [
        {"name": "Config file", "status": "pass", "detail": "...", "fix": "", "data": {}}
      ]
    }
  ]
}

Cross-platform notes:

  • Windows model paths (C:/...) detected on Mac/Linux are reported as SKIP, not FAIL
  • SSH version checks use BatchMode=yes to avoid hanging on passphrase prompts
  • Firewall hints are platform-specific (macOS socketfilterfw, Linux ufw)

Config validation checks (reported as WARN, not FAIL):

Check Rule
Port ranges coordinator_port, all rpc_port, proxy.port must be 1-65535
VRAM positive All vram_gb values must be > 0
Coordinator binary Must be a non-empty string
Proxy URLs Draft and target URLs must be valid http:// or https:// URLs
Backend enum Draft and target backend must be "llamacpp" or "ollama"
max_draft_tokens Must be between 1 and 256
Duplicate RPC addresses Warns if two workers share the same host:port

These checks run after YAML parsing succeeds. They catch "you probably didn't mean this" errors before they cause runtime failures.

Log Commands

tightwad logs

View coordinator or proxy logs. Logs are written to ~/.tightwad/logs/.

tightwad logs [coordinator|proxy] [-f] [--clear] [-n LINES]
Option Description
coordinator View coordinator log (default)
proxy View proxy log
-f, --follow Live-tail the log (like tail -f)
--clear Truncate all log files
-n, --lines N Number of lines to show (default: 50) 
$ tightwad logs
llama_model_load: loaded meta data with 36 key-value pairs
llama_model_load: model type = 32B
...

$ tightwad logs -f
# (streams new output in real time, Ctrl+C to stop)

$ tightwad logs proxy
INFO:     Started server process [12345]
INFO:     Uvicorn running on http://0.0.0.0:8088

$ tightwad logs --clear
Logs cleared.

Log files:

  • ~/.tightwad/logs/coordinator.log — llama-server stdout/stderr (append mode)
  • ~/.tightwad/logs/proxy.log — uvicorn access and error logs (append mode)

Logs are not rotated. Use tightwad logs --clear to truncate when they get large.

Speculative Decoding Proxy Commands

tightwad proxy start

Start the speculative decoding proxy server. Both draft and target servers must be running.

Single-drafter:

$ tightwad proxy start
Starting speculative decoding proxy...
  Draft:  qwen3-8b @ http://192.168.86.250:8081
  Target: qwen3-32b @ http://192.168.86.36:8080
  Max draft tokens: 32
  Listening on: 0.0.0.0:8088
  Dashboard: http://127.0.0.1:8088/dashboard

Multi-drafter:

$ tightwad proxy start
Starting speculative decoding proxy...
  Drafters (3):
    - qwen3-8b @ http://192.168.86.250:8081 (llamacpp)
    - qwen3-1.7b @ http://192.168.86.28:8081 (llamacpp)
    - qwen3:1.7b @ http://192.168.86.86:11434 (ollama)
  Target: qwen3-32b @ http://192.168.86.36:8080
  Max draft tokens: 32
  Listening on: 0.0.0.0:8088
  Dashboard: http://127.0.0.1:8088/dashboard

The proxy runs in the foreground (uvicorn). PID is written to ~/.tightwad/proxy.pid.

A live monitoring dashboard is always available at /dashboard when the proxy runs — server health, SVG charts (acceptance rate + tokens/round), and a per-request timing log. See Architecture#Live Dashboard for details.

tightwad proxy stop

Stop the speculative decoding proxy.

$ tightwad proxy stop
Proxy stopped.

tightwad proxy status

Show server health, drafter wins, and speculation stats.

Single-drafter:

$ tightwad proxy status
  ● Draft: qwen3-8b @ http://192.168.86.250:8081
  ● Target: qwen3-32b @ http://192.168.86.36:8080

              Speculation Stats
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Metric           ┃ Value     ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Rounds           │ 39        │
│ Drafted          │ 1247      │
│ Accepted         │ 759       │
│ Acceptance rate  │ 60.9%     │
│ Tokens/round     │ 33.5      │
│ Uptime           │ 120s      │
└──────────────────┴───────────┘

Multi-drafter:

$ tightwad proxy status
  Drafters:
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Model       ┃ URL                           ┃ Backend ┃ Health ┃ Wins ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━┩
│ qwen3-8b    │ http://192.168.86.250:8081    │ llamacpp│ alive  │ 6    │
│ qwen3-1.7b  │ http://192.168.86.28:8081     │ llamacpp│ alive  │ 4    │
│ qwen3:1.7b  │ http://192.168.86.86:11434    │ ollama  │ alive  │ 0    │
└─────────────┴───────────────────────────────┴─────────┴────────┴──────┘
  ● Target: qwen3-32b @ http://192.168.86.36:8080

tightwad chat

Interactive chat with the proxy. Also available as a browser UI at http://localhost:8088.

In speculative mode, each response shows inline speculation stats — rounds, tokens drafted/accepted, acceptance rate, tokens per round, and throughput:

$ tightwad chat
Speculative mode: proxy @ :8088 -> qwen3-32b
Type your message and press Enter. Ctrl+C to quit.

You: What is the capital of France?
AI: The capital of France is Paris.
  ↳ 3 rounds, 96 drafted, 71 accepted (74.0%), 24.3 tok/round, 18.2 tok/s

You: Explain recursion briefly.
AI: Recursion is when a function calls itself to solve smaller instances of the same problem.
  ↳ 5 rounds, 160 drafted, 142 accepted (88.8%), 28.4 tok/round, 22.1 tok/s

Stats are per-response deltas (not cumulative). If the status endpoint is unreachable, stats are silently skipped.

Option Description
--direct Bypass proxy, chat directly with target (no stats shown, for A/B speed comparison)

The --direct flag is useful for comparing speculative decoding speed against direct target inference.

RPC Cluster Commands

tightwad status

Show full cluster status: coordinator state, worker health, VRAM totals, and tensor split.

$ tightwad status
● Coordinator PID 12345 on :8080
  Health: OK

               RPC Workers
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┓
┃ Address             ┃ Status┃ Latency ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━┩
│ 192.168.86.36:50052 │ alive │ 0.8ms   │
│ 192.168.86.36:50053 │ alive │ 0.9ms   │
└─────────────────────┴───────┴─────────┘

Total VRAM: 76 GB across 4 GPUs
Tensor split: [0.32, 0.32, 0.21, 0.16]
Models: qwen3-32b, deepseek-r1-70b, llama3.3-70b

tightwad start

Start the coordinator llama-server. Checks all RPC workers are healthy and verifies llama.cpp version consistency before launching.

tightwad start [-m MODEL] [--skip-version-check] [--ram-reclaim off|on|auto]
Option Description
-m, --model NAME Model name from config (default: model with default: true)
--skip-version-check Skip llama.cpp version matching between coordinator and workers
--ram-reclaim MODE RAM reclaim mode: off, on, or auto (default: from config)

Prerequisites: All rpc-server instances must be running on workers.

Version enforcement: Before launching, the coordinator queries each worker's llama.cpp version via SSH and refuses to start if any version differs from the local binary. This prevents the silent RPC failures caused by protocol mismatches between different llama.cpp builds. Use --skip-version-check to bypass (e.g., when SSH is not available but you've verified versions manually).

tightwad stop

Stop the coordinator llama-server. RPC workers are not affected.

tightwad stop

tightwad swap MODEL

Hot-swap to a different model. Stops the coordinator and restarts with the new model. RPC workers persist since they're stateless.

tightwad swap deepseek-r1-70b

tightwad benchmark

Run a prompt-processing and text-generation benchmark against the running coordinator.

tightwad benchmark

Measures:

  • Prompt processing speed (tok/s) with ~512 token prompt
  • Text generation speed (tok/s) with 128 token completion

Requires the coordinator to be running.

Model Distribution

tightwad distribute

Distribute a model to all worker machines. Supports two methods: rsync (single-source push) and swarm (P2P transfer). By default, auto-selects based on file size.

tightwad distribute MODEL [--method auto|rsync|swarm] [--token TOKEN] [-t host:/path] [--dry-run]
Option Description
--method Transfer method: auto (default), rsync, or swarm
--token TOKEN Bearer token for swarm auth (ignored for rsync)
-t, --target host:/path Override: send to a specific host:path instead of all workers
--dry-run Preview transfers without executing

Auto-selection: Files < 1 GB use rsync. Files >= 1 GB use swarm (P2P scales better for large models).

Swarm flow (when --method swarm):

  1. Creates a manifest if one doesn't exist
  2. Starts a seeder on the local machine (background)
  3. SSHes into each worker and runs tightwad swarm pull
  4. Waits for all workers to finish
  5. Stops the seeder
$ tightwad distribute qwen3-32b --method swarm --token secret
Auto-selected method: swarm
Creating swarm manifest...
Manifest created: 282 pieces
Starting seeder on 192.168.86.36:9080...
Distributing Qwen3-32B-Q4_K_M.gguf to 2 target(s) via swarm...

⠋ 192.168.86.250 (2070): pulling via swarm...  ████████████████ 100%
⠋ 192.168.86.28 (M2): pulling via swarm...     ████████████████ 100%

All 2 transfer(s) complete.

Prerequisites for swarm mode:

  • Workers must have tightwad installed and on PATH
  • Workers must have model_dir set in cluster.yaml
  • SSH access to workers (uses ssh_user from config, BatchMode=yes)

rsync mode (existing behavior):

$ tightwad distribute qwen3-32b --method rsync
Distributing Qwen3-32B-Q4_K_M.gguf to 2 target(s) via rsync...

⠋ 192.168.86.250 (2070): transferring...  ████████████████ 100%
⠋ 192.168.86.28 (M2): transferring...     ████████████████ 100%

All 2 transfer(s) complete.

Manifest Commands

tightwad manifest create

Generate a swarm manifest for a GGUF model file. Splits the file into pieces (default 64 MB) and SHA256-hashes each one. Optionally extracts GGUF metadata (architecture, quantization, parameters).

tightwad manifest create <model.gguf> [--piece-size 64] [--no-inspect] [-o path]
Option Description
--piece-size N Piece size in MB (default: 64)
--no-inspect Skip GGUF metadata extraction
-o, --output PATH Output manifest path (default: <model>.tightwad.manifest) 
$ tightwad manifest create ~/models/Qwen3-32B-Q4_K_M.gguf
⠋ Hashing pieces... ████████████████████ 282/282 pieces

Manifest created: ~/models/Qwen3-32B-Q4_K_M.gguf.tightwad.manifest
  Model:    Qwen3-32B-Q4_K_M
  Size:     18.05 GB
  Pieces:   282 x 64 MB
  Metadata: {"arch": "qwen3", "quantization": "Q4_K_M", "params": "32.8B"}

Swarm P2P Transfer Commands

See Swarm Transfer for full architecture details.

tightwad swarm seed

Start a P2P seeder HTTP server for a model file. If no manifest exists, one is created automatically. Serves pieces to pulling peers.

tightwad swarm seed <model.gguf> [--port 9080] [--host 0.0.0.0] [--token TOKEN] [--allowed-ips CIDR...]
Option Description
--port N Seeder port (default: 9080)
--host ADDR Bind host (default: 0.0.0.0)
--token TOKEN Require Bearer token auth on all endpoints
--allowed-ips CIDR Restrict access to these IP ranges (repeatable, e.g. 192.168.86.0/24)

The seeder exposes:

  • GET /manifest — manifest JSON
  • GET /bitfield — {"have": [0, 1, ...]}
  • GET /pieces/<index> — raw piece bytes
  • GET /health — status JSON

PID is written to ~/.tightwad/swarm-<model>.pid.

$ tightwad swarm seed ~/models/Qwen3-32B-Q4_K_M.gguf --port 9080
Starting swarm seeder...
  Model:  Qwen3-32B-Q4_K_M (Qwen3-32B-Q4_K_M.gguf)
  Pieces: 282/282 (100%)
  Listen: 0.0.0.0:9080

tightwad swarm pull

Pull a model from one or more swarm peers. Supports resume — if interrupted, re-running continues from where it left off. Uses rarest-first piece selection across all peers.

tightwad swarm pull <dest.gguf> --manifest <path-or-url> --peer <url> [--peer <url>...] [--parallel 4] [--token TOKEN]
Option Description
--manifest PATH_OR_URL Path or URL to manifest (required)
--peer URL Peer URL, repeatable (required, at least one)
--parallel N Max concurrent downloads (default: 4)
--token TOKEN Bearer token for authenticated seeders 
$ tightwad swarm pull /tmp/Qwen3-32B.gguf \
    --manifest http://192.168.86.36:9080/manifest \
    --peer http://192.168.86.36:9080 \
    --peer http://192.168.86.250:9080
Pulling Qwen3-32B-Q4_K_M.gguf
  Pieces: 282 total, 282 to download
  Peers:  2
  Parallel: 4
⠋ Downloading pieces... ████████████████████ 282/282

Download complete: /tmp/Qwen3-32B.gguf

tightwad swarm status

Show swarm completion status for a model. Verifies pieces on first run if bitfield is empty.

tightwad swarm status <model.gguf>

$ tightwad swarm status ~/models/Qwen3-32B-Q4_K_M.gguf
Swarm Status: Qwen3-32B-Q4_K_M.gguf
  Model:      Qwen3-32B-Q4_K_M
  Size:       18.05 GB
  Pieces:     282 x 64 MB
  Have:       282/282 (100%)
  Complete!
  Metadata:   {"arch": "qwen3", "quantization": "Q4_K_M"}

Clone this wiki locally