Troubleshooting

Troubleshooting

Speculative Proxy Issues

Draft Always Fails / Falling Back to Target

Symptom: hydra proxy status shows 0 rounds, proxy logs show "Draft failed, falling back to target".

Common causes:

1. Wrong Ollama model name: Ollama uses colons (qwen3:8b), not dashes (qwen3-8b). A wrong name returns 404 with "model not found" in the response body.
2. Draft server unreachable: Check the URL in your config. Try curl <draft-url>/api/generate -d '{"model":"qwen3:8b","prompt":"test","raw":true,"stream":false}' manually.
3. Ollama thinking mode: Qwen3 models on Ollama have thinking enabled by default. The proxy uses raw: true to bypass this. If you're using the /v1/completions endpoint directly, text may be empty because all tokens go to reasoning.

"Proxy already running" Error

Fix: Delete stale PID file:

rm ~/.hydra/proxy.pid

Chat Completions Return Empty Content

Cause: The chat template is model-family specific. The default Qwen3 template (<|im_start|>) will produce garbage or empty output if the target model is from a different family (e.g., GLM, Llama). Use /v1/completions with raw prompts for cross-family setups, or ensure both draft and target are the same family.

Low Acceptance Rate

Possible fixes:

1. Use same-family models (Qwen3-8B → Qwen3-72B, not Qwen3 → GLM)
2. Reduce max_draft_tokens — shorter drafts have higher acceptance rates
3. For factual/deterministic prompts, acceptance is naturally higher than creative ones

---

RPC Cluster Issues

RPC Workers Show "down"

Symptom: hydra status shows workers as down.

Checks:

1. Is rpc-server actually running on the worker machine?
2. Is the firewall allowing traffic on the RPC ports (50052, 50053)?
   • Windows: Add inbound rules in Windows Defender Firewall
3. Can you ping the worker from the coordinator? ping 192.168.86.36
4. Can you TCP connect? nc -zv 192.168.86.36 50052

Windows firewall fix:

netsh advfirewall firewall add rule name="Hydra RPC 50052" dir=in action=allow protocol=tcp localport=50052
netsh advfirewall firewall add rule name="Hydra RPC 50053" dir=in action=allow protocol=tcp localport=50053

Coordinator Starts But No Output

Symptom: hydra start succeeds but API calls hang or return errors.

Checks:

1. Is the model file path correct and accessible on the coordinator?
2. Does the coordinator have enough RAM to mmap the model?
3. Check llama-server logs — Hydra captures stdout/stderr from the process
4. Try running llama-server manually to see output:

   llama-server -m /models/model.gguf -ngl 999 --rpc 192.168.86.36:50052

Slow Generation Speed

Cause: Gigabit Ethernet bottleneck.

Mitigations:

1. Upgrade to 2.5GbE USB-C adapters (~$25 each) — see Network Optimization
2. Reduce remote GPU share — put more layers on the coordinator
3. Reduce context size to lower memory bandwidth requirements
4. For prompt processing, expect higher network overhead than generation

"Coordinator already running" Error

Symptom: hydra start says coordinator is already running.

Fix: The PID file at ~/.hydra/coordinator.pid may be stale. Run hydra stop first, or if the process is truly dead, delete the PID file manually:

rm ~/.hydra/coordinator.pid

CUDA Out of Memory on Worker

Symptom: rpc-server crashes or coordinator fails to allocate layers.

Cause: Other processes (e.g., existing llama-server for GLM) are using GPU memory.

Fix: Either stop the competing process or adjust tensor split to give less work to that GPU. You can override in cluster.yaml by reducing vram_gb for that GPU.

Cross-Vendor RPC Not Working

Symptom: Coordinator connects to workers but produces garbage output.

Checks:

1. Both coordinator and worker must use the same version of llama.cpp
2. Rebuild both from the same commit
3. The RPC protocol is not versioned — mismatched builds will fail silently

ROCm Build Fails

Common issues:

• Wrong AMDGPU_TARGETS: Use rocminfo to find your GPU's GFX ID. 7900 XTX = gfx1100.
• Missing ROCm dev packages: Install rocm-dev meta-package
• CMake can't find HIP: Set CMAKE_PREFIX_PATH=/opt/rocm