Skip to content

Troubleshooting

Jump to bottom
youngharold edited this page Mar 28, 2026 · 11 revisions

Troubleshooting

Start here: Run tightwad doctor --fix for an automated diagnostic of your setup. It checks config, binaries, network, services, and version compatibility — and suggests fixes for each issue found.

Speculative Proxy Issues

Draft Always Fails / Falling Back to Target

Symptom: tightwad proxy status shows 0 rounds, proxy logs show "Draft failed, falling back to target".

Common causes:

  1. Wrong Ollama model name: Ollama uses colons (qwen3:8b), not dashes (qwen3-8b). A wrong name returns 404 with "model not found" in the response body.
  2. Draft server unreachable: Check the URL in your config. Try curl <draft-url>/api/generate -d '{"model":"qwen3:8b","prompt":"test","raw":true,"stream":false}' manually.
  3. Ollama thinking mode: Qwen3 models on Ollama have thinking enabled by default. The proxy uses raw: true to bypass this. If you're using the /v1/completions endpoint directly, text may be empty because all tokens go to reasoning.

"Proxy already running" Error

Fix: Delete stale PID file:

rm ~/.tightwad/proxy.pid

Chat Completions Return Empty Content

Cause: The chat template is model-family specific. The default Qwen3 template (<|im_start|>) will produce garbage or empty output if the target model is from a different family (e.g., GLM, Llama). Use /v1/completions with raw prompts for cross-family setups, or ensure both draft and target are the same family.

Low Acceptance Rate

Possible fixes:

  1. Use same-family models (Qwen3-8B → Qwen3-32B, not Qwen3 → GLM)
  2. Reduce max_draft_tokens — shorter drafts have higher acceptance rates
  3. For factual/deterministic prompts, acceptance is naturally higher than creative ones

RPC Cluster Issues

RPC Workers Show "down"

Symptom: tightwad status shows workers as down.

Checks:

  1. Is rpc-server actually running on the worker machine?
  2. Is the firewall allowing traffic on the RPC ports (50052, 50053)?
    • Windows: Add inbound rules in Windows Defender Firewall
  3. Can you ping the worker from the coordinator? ping 192.168.86.36
  4. Can you TCP connect? nc -zv 192.168.86.36 50052

Windows firewall fix:

netsh advfirewall firewall add rule name="Tightwad RPC 50052" dir=in action=allow protocol=tcp localport=50052
netsh advfirewall firewall add rule name="Tightwad RPC 50053" dir=in action=allow protocol=tcp localport=50053

Coordinator Starts But No Output

Symptom: tightwad start succeeds but API calls hang or return errors.

Checks:

  1. Is the model file path correct and accessible on the coordinator?
  2. Does the coordinator have enough RAM to mmap the model?
  3. Check coordinator logs: 
    tightwad logs              # last 50 lines
tightwad logs -f           # live tail
  4. Try running llama-server manually to see output: 
    llama-server -m /models/model.gguf -ngl 999 --rpc 192.168.86.36:50052

Coordinator stdout/stderr is logged to ~/.tightwad/logs/coordinator.log in append mode. Proxy logs go to ~/.tightwad/logs/proxy.log.

Slow Generation Speed

Cause: Gigabit Ethernet bottleneck.

Mitigations:

  1. Upgrade to 2.5GbE USB-C adapters (~$25 each) — see Network Optimization
  2. Reduce remote GPU share — put more layers on the coordinator
  3. Reduce context size to lower memory bandwidth requirements
  4. For prompt processing, expect higher network overhead than generation

"Coordinator already running" Error

Symptom: tightwad start says coordinator is already running.

Fix: The PID file at ~/.tightwad/coordinator.pid may be stale. Run tightwad stop first, or if the process is truly dead, delete the PID file manually:

rm ~/.tightwad/coordinator.pid

CUDA Out of Memory on Worker

Symptom: rpc-server crashes or coordinator fails to allocate layers.

Cause: Other processes (e.g., existing llama-server for GLM) are using GPU memory.

Fix: Either stop the competing process or adjust tensor split to give less work to that GPU. You can override in cluster.yaml by reducing vram_gb for that GPU.

Cross-Vendor RPC Not Working

Symptom: Coordinator connects to workers but produces garbage output.

Checks:

  1. Both coordinator and worker must use the same version of llama.cpp
  2. Rebuild both from the same commit
  3. The RPC protocol is not versioned — mismatched builds will fail silently

Tightwad doctor check: Run tightwad doctor before starting — it validates version compatibility. If running manually without tightwad, check versions: llama-server --version on coordinator and rpc-server --version on each worker. All must report the same build number (e.g., version: 8111).

Mac rpc-server Exposes Two Devices

Symptom: Coordinator sees 5 devices when you expect 4. The Mac shows up as two RPC entries (RPC0 and RPC1) at the same address. Tensor split gets applied to the wrong number of devices.

Cause: macOS rpc-server exposes both Metal GPU and CPU as separate backends by default.

Fix: Restrict to Metal only with --device MTL0:

rpc-server --host 0.0.0.0 --port 50052 --device MTL0

To see available device names, start without --device — it will list them (e.g., MTL0, BLAS, CPU).

Coordinator Hangs During Loading — RPC Worker Not Connected

Symptom: llama-server starts but hangs at model loading. RAM usage stays low (~170MB). One or more RPC workers never connect.

Checks:

  1. Is the rpc-server process actually running and listening? Check with netstat -an | findstr 50052 (Windows) or lsof -i :50052 (macOS/Linux)
  2. Can you TCP connect from the coordinator machine? nc -zv <worker-ip> 50052
  3. Version mismatch: If coordinator's llama-server is a different build than the worker's rpc-server, it will hang silently. Ensure all binaries are from the same release.
  4. Windows firewall: You may need both a port rule AND a program rule. The port rule alone may not be sufficient.
  5. macOS firewall: rpc-server must be explicitly allowed. If launched via SSH, the user never sees the "Allow" dialog. Use socketfilterfw --add and --unblockapp (requires sudo).

Tensor Allocation Failure (alloc_tensor_range)

Symptom: alloc_tensor_range: failed to allocate RPC0[...] buffer of size XXXXX

Cause: The coordinator tried to allocate more VRAM than a worker has. Common when:

  • A Mac rpc-server registered as 2 devices (see above), so tensor split values got shifted
  • An RPC worker was down during startup, so its share got redistributed to other devices
  • vram_gb in cluster.yaml overstates the actual available memory

Fix:

  1. Ensure all RPC workers are up and reachable BEFORE starting the coordinator
  2. Use --device MTL0 on Mac workers
  3. Set vram_gb to recommendedMaxWorkingSetSize for Apple Silicon (shown at rpc-server startup)
  4. Recalculate --tensor-split proportionally to actual available VRAM

MoE Models Fail to Distribute via RPC

Symptom: alloc_tensor_range: failed to allocate RPC[...] buffer of size 22GB even though tensor split gives that GPU much less than 22GB of the model.

Cause: MoE (Mixture of Experts) models replicate routing tables and expert selection weights to every device. This adds ~20GB of per-device overhead regardless of tensor split. A GPU assigned only 14% of the model via --tensor-split still needs the full routing tables.

Example: GPT-OSS 120B (~60GB Q4_K_XL) with --tensor-split 0.18,0.14,0.34,0.34 — the 3060 (12GB) gets 14% of weights (~8.4GB) but needs +20GB for routing tables = 28GB total, far exceeding 12GB VRAM.

Detection: tightwad inspect <model.gguf> now shows MoE expert count and per-device shared overhead. tightwad doctor and tightwad start automatically warn when MoE models are assigned to GPUs with insufficient VRAM.

Fix: For MoE models, either:

  1. Use only large GPUs (24GB+) as RPC workers
  2. Skip RPC entirely — use CPU offload on a single machine with enough total VRAM
  3. Use dense model alternatives when multi-GPU distribution is needed

High System RAM Usage on Coordinator During Loading

Symptom: Coordinator machine's system RAM fills up while loading a model across RPC workers.

Cause: llama-server memory-maps (mmap) the entire GGUF file into system RAM before distributing tensor data to RPC workers over the network. A 40GB model will temporarily consume ~40GB of RAM on the coordinator, even though most of it will be sent to remote GPUs.

Mitigation: This is expected behavior during loading. RAM usage should stabilize after loading completes. Ensure the coordinator has enough system RAM for the full model file size, plus some headroom for the OS and KV cache.

Config Validation Warnings

Symptom: tightwad doctor shows WARN entries under Configuration like "Port range", "VRAM positive", or "Duplicate RPC address".

Cause: These are structural validation checks that catch likely errors in cluster.yaml before they cause runtime failures:

Warning Meaning Fix
Port range Port outside 1-65535 Use a valid port number
VRAM positive vram_gb is 0 or negative Set to actual GPU VRAM in GB
Coordinator binary Empty string for binary path Set binaries.coordinator to llama-server or an absolute path
URL valid Draft/target URL isn't valid http(s) Use full URL like http://host:port
Backend valid Backend isn't llamacpp or ollama Fix the backend field
max_draft_tokens Outside 1-256 range Use a value between 1 and 256 (32 recommended)
Duplicate RPC address Two GPUs share the same host:port Give each GPU a unique rpc_port

These are warnings (not failures) — a weird value might be intentional. But they usually indicate a config typo.

Docker Issues

Container Can't Reach LAN Servers

Symptom: docker run starts but proxy immediately shows "Draft failed, falling back to target" or connection errors.

Cause: Docker's default bridge network isolates the container from the host's LAN. Your Ollama/llama-server instances are on LAN IPs that the container can't reach.

Fixes:

  1. Linux: Use --network host (the default in docker-compose.yml). This gives the container direct LAN access.
  2. Mac/Docker Desktop: --network host doesn't work. Use -p 8088:8088 for port mapping. For servers on the Docker host, use host.docker.internal instead of 127.0.0.1 or localhost. For servers on other LAN machines, use their real IPs — Docker Desktop can route to the LAN.
  3. Windows/Docker Desktop: Same as Mac — use -p 8088:8088 and host.docker.internal.

"Config file not found" in Docker

Symptom: Container exits with FileNotFoundError: Config file not found: /app/configs/cluster.yaml

Cause: The Docker image doesn't include configs/cluster.yaml (it's in .dockerignore). The image is designed to use env var config exclusively.

Fix: Set TIGHTWAD_DRAFT_URL and TIGHTWAD_TARGET_URL env vars:

docker run --rm --network host \
  -e TIGHTWAD_DRAFT_URL=http://192.168.1.10:11434 \
  -e TIGHTWAD_TARGET_URL=http://192.168.1.20:11434 \
  tightwad

Or mount a config file:

docker run --rm --network host \
  -v /path/to/cluster.yaml:/app/configs/cluster.yaml \
  tightwad

Docker Compose Port Conflict

Symptom: docker compose up fails with "port is already allocated".

Fix: Either stop whatever is using port 8088, or change TIGHTWAD_PORT in docker-compose.yml:

environment:
  TIGHTWAD_PORT: "9088"

If using port mapping instead of host networking, also update the ports mapping.

tightwad init Issues

No Servers Found

Symptom: tightwad init completes but shows "No inference servers found".

Checks:

  1. Are the servers running? Ollama must be started with OLLAMA_HOST=0.0.0.0 to listen on all interfaces (not just localhost). llama-server needs --host 0.0.0.0.
  2. Wrong subnet? The auto-detection picks the first non-loopback IPv4 address and assumes /24. If you have multiple NICs or a non-standard subnet, use --subnet: 
    tightwad init --subnet 10.0.0.0/24
  3. Firewall blocking scan? The scanner does TCP connect probes on ports 11434, 8080, and 8081. Some firewalls block incoming connections. Temporarily disable the firewall on the server machines to test.
  4. Non-standard port? If your server uses a custom port (e.g., 8082), add it with --port: 
    tightwad init --port 8082

Scan Takes Too Long

Symptom: tightwad init hangs or takes more than 10 seconds.

Cause: The scanner probes 254 hosts × 3 ports with a 1.5s timeout each. If many hosts have filtered (not closed) ports, the scanner waits for each timeout.

Mitigation: Use --subnet to narrow the range if you know which IPs to scan. The scanner uses 128 concurrent connections, so a clean /24 typically completes in 3-5 seconds.

Wrong Subnet Detected

Symptom: tightwad init scans a different subnet than expected (e.g., a Docker bridge or VPN).

Fix: Specify the correct subnet manually:

tightwad init --subnet 192.168.86.0/24

"Only 1 server found"

Symptom: The wizard says it needs at least 2 servers.

Cause: Speculative decoding requires both a draft server and a target server. If you only have one machine, you need to run two model servers on it (e.g., Ollama on 11434 + llama-server on 8081).

Workaround for single-machine testing:

# Terminal 1: large model
OLLAMA_HOST=0.0.0.0 ollama run qwen3:32b

# Terminal 2: small model on a different port
llama-server -m qwen3-1.7b.gguf --port 8081 --host 0.0.0.0

# Terminal 3: init will now find both
tightwad init

Generated Config Has Wrong Model Names

Symptom: The config works for connectivity but the proxy fails with "model not found".

Cause: If a server has no models loaded when scanned, the wizard asks you to type the model name manually. A typo here causes Ollama to return 404.

Fix: Check loaded models with ollama ps (for Ollama) or curl http://host:port/v1/models (for llama-server). Edit the generated configs/cluster.yaml to fix the model name. Ollama model names use colons and tags (e.g., qwen3:8b, not qwen3-8b).

ROCm Multi-GPU Hangs

Symptom: llama-server hangs during model loading or generation on ROCm with 2+ GPUs. No error message — just freezes.

Cause: AMD's SDMA (System DMA) engine and hardware queue scheduling interact poorly on certain motherboards (especially PCIe 2.0 boards without Resizable BAR / Above 4G Decoding).

Fix: As of v0.1.4, Tightwad auto-injects the correct environment variables for hip backends with 2+ GPUs:

HSA_ENABLE_SDMA=0
GPU_MAX_HW_QUEUES=1

If you're using Tightwad's coordinator management (tightwad start), this happens automatically. If you're running llama-server manually, set these env vars:

HSA_ENABLE_SDMA=0 GPU_MAX_HW_QUEUES=1 llama-server -m model.gguf --no-mmap --no-warmup ...

You may also need --no-mmap and --no-warmup on PCIe 2.0 boards — add them via extra_args in your coordinator config:

coordinator:
  backend: hip
  extra_args: ["--no-mmap", "--no-warmup"]

Override presets: If your board handles SDMA fine and you want to disable the preset, set it explicitly:

coordinator:
  backend: hip
  env:
    HSA_ENABLE_SDMA: "1"

ROCm Build Fails

Common issues:

  • Wrong AMDGPU_TARGETS: Use rocminfo to find your GPU's GFX ID. 7900 XTX = gfx1100.
  • Missing ROCm dev packages: Install rocm-dev meta-package
  • CMake can't find HIP: Set CMAKE_PREFIX_PATH=/opt/rocm

Clone this wiki locally