Skip to content

Configuration

Jump to bottom
youngharold edited this page Mar 28, 2026 · 11 revisions

Configuration

Tightwad reads its cluster topology and proxy settings from a YAML config file.

Config Resolution

Tightwad checks for configuration in this order:

  1. tightwad -c /path/to/cluster.yaml (CLI flag)
  2. TIGHTWAD_CONFIG environment variable
  3. configs/cluster.yaml in the project directory (default)
  4. Environment variable fallback — if no YAML file exists, Tightwad builds a proxy-only config from TIGHTWAD_* env vars (see below)

This means Docker containers and quick tests can run without any config file at all.

Full Reference

# ── Speculative Decoding Proxy ──────────────────────────────────

proxy:
  host: 0.0.0.0          # Bind address for proxy API
  port: 8088              # Proxy API port
  max_draft_tokens: 8     # Tokens to draft per round (default: 8)
  fallback_on_draft_failure: true  # Forward to target if draft is down

  # Single drafter (used when drafters: is absent)
  draft:
    url: http://192.168.86.250:11434   # Draft server URL
    model_name: qwen3:8b               # Model name (display + Ollama routing)
    backend: ollama                     # "ollama" or "llamacpp"

  # Multi-drafter parallelism (overrides draft: when present)
  drafters:
    - url: http://192.168.86.250:8081
      model_name: qwen3-8b
      backend: llamacpp
    - url: http://192.168.86.28:8081
      model_name: qwen3-1.7b
      backend: llamacpp
    - url: http://192.168.86.86:11434
      model_name: qwen3:1.7b
      backend: ollama

  target:
    url: http://192.168.86.36:11434    # Target server URL
    model_name: qwen3:32b
    backend: ollama

# ── RPC Cluster ─────────────────────────────────────────────────

# Coordinator — the machine running llama-server
coordinator:
  host: 0.0.0.0        # Bind address for the API
  port: 8080            # API port
  backend: hip          # "hip" (AMD/ROCm) or "cuda" (NVIDIA)
  gpus:                 # Local GPUs (no rpc_port)
    - name: "7900 XTX #0"
      vram_gb: 24
    - name: "7900 XTX #1"
      vram_gb: 24

# Workers — remote machines running rpc-server
workers:
  - host: 192.168.86.36     # Worker IP address
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052      # Port for this GPU's rpc-server
      - name: "RTX 3060"
        vram_gb: 12
        rpc_port: 50053

# Models available for loading (RPC cluster)
models:
  qwen3-32b:                 # Model identifier (used with tightwad start -m)
    path: /models/Qwen3-32B-Q4_K_M.gguf
    ctx_size: 8192           # Context window (default: 8192)
    predict: 4096            # Max tokens to predict (default: 4096)
    flash_attn: true         # true/false, or "on"/"off"/"auto" (default: true)
    default: true            # Load this model by default

  deepseek-r1-70b:
    path: /models/DeepSeek-R1-Distill-Qwen-70B-Q4_K_M.gguf
    ctx_size: 8192
    predict: 4096
    flash_attn: true

# Binary paths (RPC cluster)
binaries:
  coordinator: /usr/local/bin/llama-server
  rpc_server: rpc-server.exe

Proxy Config

The proxy section is optional. If omitted, tightwad proxy commands will show an error and the RPC cluster commands work as before.

proxy.draft / proxy.target

Each server endpoint has:

Field Required Description
url Yes Full URL including port (e.g., http://192.168.86.250:11434)
model_name Yes Model identifier. For Ollama, use the exact tag (qwen3:8b, not qwen3-8b)
backend No "ollama" or "llamacpp" (default: "llamacpp")

proxy.drafters

Optional list of server endpoints for multi-drafter parallelism. When present, draft: is ignored. All drafters race in parallel each round — the proxy picks the best candidate (preferring llamacpp backends with per-token logprobs).

Each entry has the same fields as draft: (url, model_name, backend).

proxy.max_draft_tokens

Number of tokens the draft model generates per speculation round. Can be an integer (e.g., 32) or auto for adaptive tuning. Default: 8.

Auto mode (max_draft_tokens: auto or TIGHTWAD_MAX_DRAFT_TOKENS=auto): The proxy adjusts the draft count at runtime based on a rolling window of acceptance rates and draft-vs-verify timing. When acceptance is high and drafts are cheap, it increases aggressively. When acceptance is low or drafts are expensive, it decreases. Starts at 16, range 4-64.

proxy.consensus_mode

Controls multi-drafter token voting. When multiple drafters are configured, their outputs are compared before consulting the target. Tokens where drafters agree can be accepted without a target call. Values: off (default), strict (unanimous), majority (>50%), any_disagree.

proxy.chat_template

Chat template for /v1/chat/completions. Values: auto (default, detects from target model family), chatml, llama3, mistral, gemma, phi, deepseek, command-r. Env var: TIGHTWAD_CHAT_TEMPLATE.

proxy.fallback_on_draft_failure

When true (default), the proxy forwards requests directly to the target if the draft server is unreachable. The proxy stays up but runs at target-only speed.

Backend Presets & Passthrough

Tightwad auto-injects known-good environment variables per backend to prevent common failures:

Backend GPUs Auto-injected env
hip 2+ HSA_ENABLE_SDMA=0, GPU_MAX_HW_QUEUES=1
cuda any (none)

These presets prevent SDMA hangs on ROCm multi-GPU setups without any manual configuration.

Override or extend with extra_args and env in the coordinator section:

coordinator:
  backend: hip
  gpus:
    - name: "7900 XTX #0"
      vram_gb: 24
    - name: "7900 XTX #1"
      vram_gb: 24
  # Additional CLI args appended to llama-server command
  extra_args: ["--no-mmap", "--no-warmup"]
  # Environment variables passed to llama-server process
  # User values override auto-injected presets
  env:
    HSA_ENABLE_SDMA: "1"      # override the default "0" preset
    CUSTOM_VAR: "value"       # add your own

Precedence: Explicit env values in YAML always win over backend presets. Presets fill in any keys not explicitly set.

models[].flash_attn

Controls the --flash-attn flag passed to llama-server. Accepts:

Value Effect
true (default) --flash-attn on
false Flag omitted
"on" / "off" / "auto" --flash-attn <value> (passed verbatim)

The "auto" option lets llama-server decide based on the model and backend.

RPC Cluster Config

coordinator.gpus[].vram_gb

Used to calculate tensor split ratios. Doesn't need to be exact — Tightwad uses these as proportions, not absolute limits. Round to the nearest GB.

workers[].gpus[].rpc_port

Each GPU on a worker needs its own rpc-server instance on a unique port. Standard convention: start at 50052 and increment.

models[].path

Absolute path to the GGUF file on the coordinator machine. Workers don't need the model file — they only receive tensor data over RPC.

models[].default

Mark one model as default: true. This is loaded when running tightwad start without -m.

Multiple Workers

workers:
  - host: 192.168.86.36
    gpus:
      - name: "RTX 4070 Ti Super"
        vram_gb: 16
        rpc_port: 50052
  - host: 192.168.86.50
    gpus:
      - name: "RTX 3090"
        vram_gb: 24
        rpc_port: 50052

Tensor split is calculated across all GPUs in order: coordinator locals first, then workers top-to-bottom.

Combined Mode Config

To use speculation over an RPC pool, point the proxy's target at the RPC coordinator URL:

# RPC pool runs the large model across multiple GPUs
coordinator:
  host: 0.0.0.0
  port: 8090
  backend: cuda
  gpus:
    - name: "RTX 4070 Ti Super"
      vram_gb: 16
    - name: "RTX 3060"
      vram_gb: 12

workers:
  - host: 192.168.86.250
    gpus:
      - name: "RTX 2070"
        vram_gb: 8
        rpc_port: 50052
  - host: 192.168.86.28
    gpus:
      - name: "Apple M2 Metal"
        vram_gb: 11
        rpc_port: 50052

# Proxy drafts locally, verifies against the pool
proxy:
  max_draft_tokens: 32
  draft:
    url: http://127.0.0.1:8081       # Any junk hardware running llama-server
    model_name: qwen3-1.7b
    backend: llamacpp                  # MUST be llamacpp for prompt-append verification
  target:
    url: http://192.168.86.36:8090    # Points to RPC pool coordinator
    model_name: qwen3-32b
    backend: llamacpp

Important: The draft backend MUST be llamacpp (not ollama) for combined mode to work. Ollama falls back to text-match verification which provides no batch speedup — you'd get 1.4 tok/s instead of 5.4 tok/s.

Validation

tightwad doctor validates the parsed config and warns about common mistakes:

  • Port ranges: All ports must be 1-65535
  • VRAM positive: All vram_gb values must be > 0
  • Proxy URLs: Must be valid http:// or https:// URLs
  • Backend enum: Must be "llamacpp" or "ollama"
  • max_draft_tokens: Must be 1-256 (0 or negative breaks the proxy)
  • Duplicate RPC addresses: Two GPUs sharing a host:port will collide

Run tightwad doctor --fix to see fix suggestions for any warnings.

Logging

Coordinator and proxy logs are written to ~/.tightwad/logs/:

File Source
coordinator.log llama-server stdout/stderr (append mode)
proxy.log uvicorn access and error logs (append mode)

View with tightwad logs or tightwad logs proxy. Use tightwad logs --clear to truncate.

Environment Variables

Config file path

Variable Description
TIGHTWAD_CONFIG Path to cluster.yaml

Proxy-only env var config (no YAML needed)

When no YAML config file exists, Tightwad checks for TIGHTWAD_DRAFT_URL and TIGHTWAD_TARGET_URL. If both are set, a minimal proxy-only config is built from env vars. This is the primary config method for Docker deployments.

Variable Maps to Default
TIGHTWAD_DRAFT_URL proxy.draft.url required
TIGHTWAD_DRAFT_MODEL proxy.draft.model_name "draft"
TIGHTWAD_DRAFT_BACKEND proxy.draft.backend "ollama"
TIGHTWAD_TARGET_URL proxy.target.url required
TIGHTWAD_TARGET_MODEL proxy.target.model_name "target"
TIGHTWAD_TARGET_BACKEND proxy.target.backend "ollama"
TIGHTWAD_PORT proxy.port 8088
TIGHTWAD_HOST proxy.host "0.0.0.0"
TIGHTWAD_MAX_DRAFT_TOKENS proxy.max_draft_tokens 8 (or auto)

Example — proxy without any config file:

TIGHTWAD_DRAFT_URL=http://192.168.1.10:11434 \
TIGHTWAD_TARGET_URL=http://192.168.1.20:11434 \
TIGHTWAD_DRAFT_MODEL=qwen3:8b \
TIGHTWAD_TARGET_MODEL=qwen3:32b \
tightwad proxy start

The env var fallback creates a minimal ClusterConfig with only the proxy populated (empty coordinator, no workers, no models). RPC cluster commands like tightwad start and tightwad status won't work in this mode — only proxy commands.

Precedence: If a YAML config file exists (via -c, $TIGHTWAD_CONFIG, or default path), it is always used. Env vars are only checked when no YAML file is found.

Docker Deployment

Tightwad ships a Dockerfile and docker-compose.yml for containerized proxy deployment.

Docker Compose

Edit docker-compose.yml with your server IPs and run:

docker compose up -d

The default compose file includes:

  • network_mode: host for direct LAN access (Linux). On Mac/Docker Desktop, switch to port mapping — see comments in the file.
  • Healthcheck — polls /v1/models every 10s via Python httpx (curl isn't in the slim image). Container is marked unhealthy after 5 consecutive failures.
  • Log volume — mounts ./logs/ to /root/.tightwad/logs so proxy logs persist across container restarts. View with tail -f ./logs/proxy.log or docker compose logs -f.

Docker run

docker build -t tightwad .
docker run --rm --network host \
  -e TIGHTWAD_DRAFT_URL=http://192.168.1.10:11434 \
  -e TIGHTWAD_DRAFT_MODEL=qwen3:8b \
  -e TIGHTWAD_TARGET_URL=http://192.168.1.20:11434 \
  -e TIGHTWAD_TARGET_MODEL=qwen3:32b \
  tightwad

Mac/Docker Desktop notes

--network host doesn't work on Mac. Instead:

  1. Use -p 8088:8088 for port mapping
  2. Replace LAN IPs with host.docker.internal if the servers run on the Docker host
  3. If servers are on other LAN machines, use their real IPs — Docker Desktop routes to the LAN by default
docker run --rm -p 8088:8088 \
  -e TIGHTWAD_DRAFT_URL=http://host.docker.internal:11434 \
  -e TIGHTWAD_TARGET_URL=http://192.168.1.20:11434 \
  -e TIGHTWAD_TARGET_MODEL=qwen3:32b \
  tightwad

Auto-Discovery with tightwad init

Instead of writing YAML by hand, tightwad init scans your LAN for running inference servers, identifies Ollama and llama-server instances, and generates a config interactively. See CLI Reference for full documentation.

Clone this wiki locally