Skip to content

Configuration

Jump to bottom
youngharold edited this page Feb 17, 2026 · 11 revisions

Configuration

Tightwad reads its cluster topology and proxy settings from a YAML config file.

Config Resolution

  1. tightwad -c /path/to/cluster.yaml (CLI flag)
  2. TIGHTWAD_CONFIG environment variable
  3. configs/cluster.yaml in the project directory (default)

Full Reference

# ── Speculative Decoding Proxy ──────────────────────────────────

proxy:
  host: 0.0.0.0          # Bind address for proxy API
  port: 8088              # Proxy API port
  max_draft_tokens: 8     # Tokens to draft per round (default: 8)
  fallback_on_draft_failure: true  # Forward to target if draft is down

  # Single drafter (used when drafters: is absent)
  draft:
    url: http://192.168.86.250:11434   # Draft server URL
    model_name: qwen3:8b               # Model name (display + Ollama routing)
    backend: ollama                     # "ollama" or "llamacpp"

  # Multi-drafter parallelism (overrides draft: when present)
  drafters:
    - url: http://192.168.86.250:8081
      model_name: qwen3-8b
      backend: llamacpp
    - url: http://192.168.86.28:8081
      model_name: qwen3-1.7b
      backend: llamacpp
    - url: http://192.168.86.86:11434
      model_name: qwen3:1.7b
      backend: ollama

  target:
    url: http://192.168.86.36:11434    # Target server URL
    model_name: qwen3:32b
    backend: ollama

# ── RPC Cluster ─────────────────────────────────────────────────

# Coordinator — the machine running llama-server
coordinator:
  host: 0.0.0.0        # Bind address for the API
  port: 8080            # API port
  backend: hip          # "hip" (AMD/ROCm) or "cuda" (NVIDIA)
  gpus:                 # Local GPUs (no rpc_port)
    - name: "7900 XTX #0"
      vram_gb: 24
    - name: "7900 XTX #1"
      vram_gb: 24

# Workers — remote machines running rpc-server
workers:
  - host: 192.168.86.36     # Worker IP address
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052      # Port for this GPU's rpc-server
      - name: "RTX 3060"
        vram_gb: 12
        rpc_port: 50053

# Models available for loading (RPC cluster)
models:
  qwen3-72b:                 # Model identifier (used with tightwad start -m)
    path: /models/Qwen3-72B-Q4_K_M.gguf
    ctx_size: 8192           # Context window (default: 8192)
    predict: 4096            # Max tokens to predict (default: 4096)
    flash_attn: true         # Enable flash attention (default: true)
    default: true            # Load this model by default

  deepseek-r1-70b:
    path: /models/DeepSeek-R1-Distill-Qwen-70B-Q4_K_M.gguf
    ctx_size: 8192
    predict: 4096
    flash_attn: true

# Binary paths (RPC cluster)
binaries:
  coordinator: /usr/local/bin/llama-server
  rpc_server: rpc-server.exe

Proxy Config

The proxy section is optional. If omitted, tightwad proxy commands will show an error and the RPC cluster commands work as before.

proxy.draft / proxy.target

Each server endpoint has:

Field Required Description
url Yes Full URL including port (e.g., http://192.168.86.250:11434)
model_name Yes Model identifier. For Ollama, use the exact tag (qwen3:8b, not qwen3-8b)
backend No "ollama" or "llamacpp" (default: "llamacpp")

proxy.drafters

Optional list of server endpoints for multi-drafter parallelism. When present, draft: is ignored. All drafters race in parallel each round — the proxy picks the best candidate (preferring llamacpp backends with per-token logprobs).

Each entry has the same fields as draft: (url, model_name, backend).

proxy.max_draft_tokens

Number of tokens the draft model generates per speculation round. Higher values mean more potential speedup but lower acceptance rates if the models diverge. Default: 8.

proxy.fallback_on_draft_failure

When true (default), the proxy forwards requests directly to the target if the draft server is unreachable. The proxy stays up but runs at target-only speed.

RPC Cluster Config

coordinator.gpus[].vram_gb

Used to calculate tensor split ratios. Doesn't need to be exact — Tightwad uses these as proportions, not absolute limits. Round to the nearest GB.

workers[].gpus[].rpc_port

Each GPU on a worker needs its own rpc-server instance on a unique port. Standard convention: start at 50052 and increment.

models[].path

Absolute path to the GGUF file on the coordinator machine. Workers don't need the model file — they only receive tensor data over RPC.

models[].default

Mark one model as default: true. This is loaded when running tightwad start without -m.

Multiple Workers

workers:
  - host: 192.168.86.36
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052
  - host: 192.168.86.50
    gpus:
      - name: "RTX 3090"
        vram_gb: 24
        rpc_port: 50052

Tensor split is calculated across all GPUs in order: coordinator locals first, then workers top-to-bottom.

Combined Mode Config

To use speculation over an RPC pool, point the proxy's target at the RPC coordinator URL:

# RPC pool runs the large model across multiple GPUs
coordinator:
  host: 0.0.0.0
  port: 8090
  backend: cuda
  gpus:
    - name: "RTX 4070 Ti Super"
      vram_gb: 16
    - name: "RTX 3060"
      vram_gb: 12

workers:
  - host: 192.168.86.250
    gpus:
      - name: "RTX 2070"
        vram_gb: 8
        rpc_port: 50052
  - host: 192.168.86.28
    gpus:
      - name: "Apple M2 Metal"
        vram_gb: 11
        rpc_port: 50052

# Proxy drafts locally, verifies against the pool
proxy:
  max_draft_tokens: 32
  draft:
    url: http://127.0.0.1:8081       # Any junk hardware running llama-server
    model_name: qwen3-1.7b
    backend: llamacpp                  # MUST be llamacpp for prompt-append verification
  target:
    url: http://192.168.86.36:8090    # Points to RPC pool coordinator
    model_name: qwen3-32b
    backend: llamacpp

Important: The draft backend MUST be llamacpp (not ollama) for combined mode to work. Ollama falls back to text-match verification which provides no batch speedup — you'd get 1.4 tok/s instead of 5.4 tok/s.

Environment Variables

Variable Description
TIGHTWAD_CONFIG Path to cluster.yaml

Clone this wiki locally