Hermes hangs when provider=custom + LiteLLM proxy + Ollama (no chat completion call after probing 404s)

[andrewkangkr]

Hermes hangs (no chat completion call) when provider: custom with LiteLLM proxy as base_url and Ollama models

Related: Possibly extends #7103 (dynamic model discovery) and #10011 (auto-discover models from custom provider endpoints).

Summary

When configuring Hermes Agent with provider: custom pointing to a LiteLLM proxy that fronts Ollama, Hermes performs Ollama/llama.cpp-style endpoint probing (/api/tags, /api/v1/models, /v1/props, /props, /version, /api/show). When all of those return 404 (which is the normal behavior of a generic OpenAI-compatible proxy like LiteLLM), Hermes silently hangs and never issues the actual POST /v1/chat/completions call, eventually hitting request_timeout (60–90s). The chat completion endpoint itself is fully functional — it is never reached.

This is the same silent-fail symptom previously seen when Hermes is invoked from Paperclip with the same provider configuration.

Environment

• Hermes Agent v0.13.0 (2026.5.7) — Windows
• Python 3.12.10, OpenAI SDK 2.24.0
• LiteLLM 1.84.0 (proxy at http://localhost:4000/v1)
• Ollama 0.x (default port 11434, qwen3:14b, gemma4:e4b loaded)

Hermes ~/.hermes/config.yaml (relevant section)

model:
  default: openrouter/openrouter/free
  provider: custom
  base_url: http://localhost:4000/v1
  api_key: sk-litellm-local-dummy

Reproduction

# Ollama running, qwen3:14b loaded
ollama list   # NAME: qwen3:14b ...

# LiteLLM config has a working entry:
# - model_name: qwen3-14b
#   litellm_params:
#     model: openai/qwen3:14b
#     api_base: http://localhost:11434/v1
#     api_key: ollama

# Confirm LiteLLM proxy works directly (8–10s warm response):
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-14b","messages":[{"role":"user","content":"hi"}]}'
# → 200 OK, valid completion

# Now invoke Hermes:
hermes chat -q "hi" -Q -m qwen3-14b
# → hangs ~60-90s, no response, exits with timeout

LiteLLM proxy access log during a Hermes invocation

GET /v1/models/qwen3-14b → 200 OK             (model retrieve, works after specific entry added)
GET /v1/models           → 200 OK             (catalog)
GET /api/v1/models       → 404 Not Found      ← Ollama-style probing
GET /api/tags            → 404 Not Found      ← Ollama list models
GET /v1/props            → 404 Not Found      ← llama.cpp probing
GET /props               → 404 Not Found      ← llama.cpp
GET /version             → 404 Not Found      ← Ollama version
POST /api/show           → 404 Not Found      ← Ollama model info
# (No POST /v1/chat/completions is ever issued — Hermes gives up)

Workaround attempts (all fail)

• --provider openai → Python Traceback (cli init error)
• --provider openrouter → Python Traceback (cli.py:4474 in _init_agent)
• --toolsets file (minimal) → same timeout
• --ignore-user-config --ignore-rules → "Stripped provider prefix" warning + tries Codex (wrong provider)
• Adding specific model_name: qwen3-14b and ollama_chat/qwen3:14b entries to LiteLLM config → model lookup succeeds (200) but Hermes still probes Ollama-native endpoints and hangs

Expected behavior

After receiving 200 OK on GET /v1/models (catalog) and/or GET /v1/models/<id>, Hermes should treat the endpoint as a standard OpenAI-compatible chat-completions backend and call POST /v1/chat/completions, regardless of whether /api/tags / /v1/props / /api/show return 404. The current behavior makes Hermes incompatible with the most common deployment pattern (Hermes → LiteLLM → Ollama).

Suggested fix

In the custom provider profile (plugins/model-providers/custom/__init__.py):

1. If GET /v1/models returns 200 with a valid JSON catalog (OpenAI-compatible shape), stop probing Ollama-native endpoints and proceed straight to POST /v1/chat/completions.
2. Make the probing endpoints opt-in (env var like HERMES_PROBE_OLLAMA_NATIVE=1) rather than the default.
3. Or: when all probing returns 404, log a clear error ("not a known provider, falling back to OpenAI-compatible chat completions") and still call /v1/chat/completions.

Why this matters

This is the canonical multi-provider routing stack documented across the AI agent ecosystem:

Hermes (any LLM) ──→ LiteLLM (router/fallback/budget) ──→ Ollama / OpenRouter / Claude / etc.

Currently the chain breaks at the Hermes → LiteLLM hop for Ollama backends, even when LiteLLM is perfectly functional and Ollama responds within 1 second on direct calls.

[alt-glitch]

Duplicate of #25629 (specifically bug 2: Hermes pre-flight Ollama-native route detection runs regardless of api_mode, blocking proxied setups like LiteLLM). The root cause is identical — when all Ollama-style probes return 404, Hermes never reaches the actual POST /v1/chat/completions.

Related PRs that address this:

• fix(ollama): skip local num_ctx probe for explicit chat completions #25638 — skips local num_ctx probe for explicit chat completions
• fix(agent): skip Ollama/GLM workaround for custom provider + fix stale drain timeout tip #22570 — skips Ollama/GLM workaround for custom providers on private IPs
• feat(provider): add first-class LiteLLM proxy provider #22719 — first-class LiteLLM proxy provider (eliminates 404 flood)
• [Bug]: _query_local_context_length unconditionally probes admin-gated /v1/models/{model} on LiteLLM proxies #25848 — related LiteLLM probe log spam

[KaiFelixBennett]

Running this exact setup (Hermes Agent + LiteLLM proxy + llama.cpp, not Ollama but same issue) and hit the same probing loop. The fix that worked for me: set \provider: litellm\ instead of \provider: custom\ in config.yaml, which skips the Ollama/llama.cpp-style endpoint discovery.

Working config:
\\yaml
model:
default: my-local-qwen
provider: litellm
base_url: http://localhost:4000/v1
api_key: sk-local
\\

With \provider: custom\ Hermes probes for /api/tags, /props, etc. and hangs when they 404. Switching to \provider: litellm\ bypasses that discovery phase entirely.

I documented a full working local setup (llama.cpp + LiteLLM + Hermes Agent) here if it helps: https://github.com/KaiFelixBennett/hermes-claude-code-local — ran 7M+ tokens through it without the hang issue.

[KaiFelixBennett]

Quick field update after another long local run:

• 4h autonomous coding
• 7,256,671 tokens total
• Local stack stayed stable, but Claude Code-style context overhead was still the limiting factor around 64k windows in complex edit loops.

This aligns with the concern in this issue: endpoint health can look perfect (200 OK) while practical coding fails due to context/overhead mismatch.

I published full benchmark notes + architecture details here (would love critical pushback from people running similar stacks):
https://www.linkedin.com/posts/kai-bennett-698732276_ai-opensource-developertools-activity-7465455904142864385-nbAJ

[dobachi]

Same misdetection symptom with vanilla llama-server (not LiteLLM, not Ollama)

Adding another datapoint for the probe-order angle in detect_local_server_type. We hit the same Ollama misidentification with a plain llama.cpp llama-server backend (no proxy, no Ollama anywhere) — just qwen2.5-coder-7b served by /usr/bin/llama-server --ctx-size 65536 on 127.0.0.1:8080.

Recent llama.cpp builds ship an Ollama-compat shim that exposes /api/tags, /api/show, /api/chat. /api/tags answers with a hybrid response — both Ollama's {"models": [...]} AND OpenAI's {"object": "list", "data": [...]} in the same body. The current order in detect_local_server_type probes /api/tags before /v1/props / /props, sees "models" in the response, and returns "ollama". Then query_ollama_num_ctx POSTs /api/show which returns:

{"model_info": {"llama.context_length": 32768}, ...}

That 32768 is the GGUF n_ctx_train for Qwen2.5-Coder-7B, not the runtime --ctx-size 65536 we extended via RoPE. The #30028 guard then fails closed with "Ollama runtime context too small for Hermes tool use" and the agent refuses to start, despite the server having a perfectly good 65K window.

Hermes-side config (provider: llamacpp) doesn't help — detection is server-fingerprint based, not config-declared.

Verified config:

• Hermes Agent v0.16.0 (2026.6.5) on Ubuntu 26.04, Python 3.11
• ~/.hermes/config.yaml model: { provider: llamacpp, base_url: http://127.0.0.1:8080/v1, context_length: 65536 }
• llama-server (b8681) with LLAMA_ARG_CTX_SIZE=65536, LLAMA_ARG_JINJA=1

/props on this server returns the expected default_generation_settings payload (so detect_local_server_type would correctly identify "llamacpp" if the llamacpp probe ran first).

Local workaround we're running: swap the order in detect_local_server_type so the /v1/props+/props probe block runs before the /api/tags block. After the swap, detection returns "llamacpp", query_ollama_num_ctx skips, and the agent boots normally and chats via /v1/chat/completions.

Mentioning here rather than opening a new issue since the underlying mechanism (probe order + Ollama-shape catalogs on non-Ollama backends) is the same one #27331 and #23215 are tackling from different angles — felt useful to flag that the symptom isn't only LiteLLM-fronting-Ollama.

[mlzl1426]

我遇到了同样的问题。根本原因：httpx 默认启用 HTTP/2，而 Ollama 不支持它。解决方法是 http2=False。详情复现脚本：https://github.com/mlzl1426/fix-ollama-httpx-http2
I ran into the same issue. Root cause: httpx enables HTTP/2 by default and Ollama doesn't support it. Fix is http2=False. Details + repro script: https://github.com/mlzl1426/fix-ollama-httpx-http2