[Setup]: Thinking models on `custom`/Ollama (OpenAI-compatible) providers always truncate, and Hermes requests the full GGUF context window causing VRAM blowups

[pippo73]

What's Going Wrong?

I would like to use hermes (desktop o cli) on my windows pc using ollama running on my network
Ecco un testo conciso e mirato per quel campo:

I'm trying to run Hermes Agent entirely against a self-hosted Ollama server
(Docker, NVIDIA GPU, OpenAI-compatible endpoint at http://:11434/v1)
using provider: custom with the thinking-capable model gemma4:12b — no
internet providers.

Hermes connects to the model fine, but every single turn fails with output
truncation:

⚠️  Response truncated (finish_reason='length') - model hit max output tokens   (x3)
agent.conversation_loop: API call failed after 3 retries.
-> UI: "Response remained truncated after 3 continuation attempts"

It fails at response generation. Because model.max_tokens is unset, Hermes
passes max_tokens=None on the chat_completions path, so the provider falls
back to a tiny internal default. With a thinking model, the reasoning tokens
consume the whole budget before any content is emitted, so the response is
cut off immediately and never completes — on a clean, fully-updated install
that already includes the truncation-retry fix from PR #12152 (which only set
a default for NVIDIA NIM, not for generic custom/Ollama providers).

A second failure compounds it: for local Ollama endpoints Hermes auto-detects
the model's full advertised context (262144) and requests num_ctx=262144 on
every call, which Ollama honors by allocating a 262K KV-cache — causing CUDA
errors / VRAM exhaustion and cold-start timeouts unless model.context_length
is set manually.

Vuoi che ti prepari anche il testo per gli altri campi del template (es. Steps to reproduce, Expected behavior, Environment)? Posso adattare le sezioni che avevo già scritto nel report completo a ciascun campo.

Steps Taken

1. Configured Hermes to use my local Ollama server via an OpenAI-compatible
   endpoint. Relevant config.yaml:

   model:
     default: gemma4:12b
     provider: custom
     base_url: http://<server>:11434/v1
   

2. Started Hermes and sent a trivial prompt (e.g. "say hello"). Every turn fails
   with repeated finish_reason='length' truncation and then
   "Response remained truncated after 3 continuation attempts".

3. Updated Hermes to the latest version (now 1 release behind, commit 0d82060)
   to pick up the truncation-retry fix from PR fix: NVIDIA NIM models always truncate due to missing max_tokens default and ephemeral boost not wired to chat_completions #12152. Confirmed the fix is
   present in the installed code:

   grep -rn "_ephemeral_max_output_tokens" agent/*.py
   # -> agent/chat_completion_helpers.py (consumes it)
   # -> agent/conversation_loop.py:1676   (sets the progressive boost)
   

   The problem still persisted — desktop log still showed 3x
   "Response truncated (finish_reason='length')" per turn.

4. Verified the model itself is healthy and that the issue is purely the
   output-token budget Hermes passes, by calling the endpoint directly:

   # tiny budget -> truncates inside "reasoning", empty content
   curl -s http://<server>:11434/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"gemma4:12b","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":10}'
   # -> finish_reason="length", content="", reasoning="..."
   
   # adequate budget -> completes normally
   curl -s http://<server>:11434/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"gemma4:12b","messages":[{"role":"user","content":"Say hello in one short sentence."}],"max_tokens":300}'
   # -> finish_reason="stop", content="Hello!"
   

5. Found that Hermes reads an optional output cap from config
   (agent/agent_init.py:1321, model.max_tokens), and that no default is
   applied for custom/Ollama (only NVIDIA NIM gets the 16384 default).
   Set it explicitly with the app fully closed (Hermes rewrites config.yaml at
   runtime, so edits made while it's running are overwritten):

   model:
     default: gemma4:12b
     provider: custom
     base_url: http://<server>:11434/v1
     max_tokens: 16384
   

6. Separately hit CUDA errors / VRAM exhaustion and slow cold starts. Traced it
   to Hermes auto-detecting the GGUF max context and requesting num_ctx=262144
   (agent/agent_init.py:1603-1650). Confirmed a 64K context fits in VRAM:

   # load the model at 64K and inspect VRAM/offload
   curl -s http://<server>:11434/api/generate \
     -d '{"model":"gemma4:12b","prompt":"hi","options":{"num_ctx":65536},"keep_alive":"1m"}' >/dev/null
   curl -s http://<server>:11434/api/ps
   # -> gemma4:12b, context_length 65536, 9.2 GB, 100% on GPU
   

   Worked around it by capping the context in config:

   model:
     context_length: 65536
     ollama_num_ctx: 65536
   

Installation Method

PowerShell installer (Windows)

Operating System

windows 11

Python Version

Python 3.11.11

Hermes Version

Hermes Agent v0.16.0 (2026.6.5) · upstream 5f6be7f

Debug Report

Debug report uploaded:
  Report       https://paste.rs/BLR8e
  agent.log    https://paste.rs/msSIk
  gateway.log  https://paste.rs/zbsD2
  desktop.log  https://paste.rs/GAeiY

Full Error Output

⚠️ Response truncated (finish_reason='length') - model hit max output tokens (x3)
... agent.conversation_loop: API call failed after 3 retries.
"Response remained truncated after 3 continuation attempts"

What I've Already Tried

• Updated Hermes to the latest version to get the truncation-retry fix from
  PR fix: NVIDIA NIM models always truncate due to missing max_tokens default and ephemeral boost not wired to chat_completions #12152. → Did NOT fix it. The retry-boost code is present, but the base
  max_tokens for custom/Ollama is still None, so boosted retries are still
  too small and truncation continues.

• Lowered reasoning_effort (agent + delegation) from high → medium →
  low. → Helped marginally (shorter reasoning) but did NOT resolve it; turns
  still truncated. (The /v1 endpoint honors reasoning_effort, but think:false
  is ignored on /v1 — only the native /api/chat honors it.)

• Set extra_body: {think: false} on the model. → No effect; the
  OpenAI-compatible /v1 endpoint ignores it.

• Set model.max_tokens: 16384 explicitly in config.yaml. → FIXED the
  truncation for normal turns (responses now finish with finish_reason='stop').
  This is the key workaround, and is essentially the same default PR fix: NVIDIA NIM models always truncate due to missing max_tokens default and ephemeral boost not wired to chat_completions #12152 added
  for NVIDIA NIM — it just isn't applied to custom/Ollama providers.

• Set model.context_length / model.ollama_num_ctx to 65536 to stop Hermes
  from requesting the full 262144 GGUF window. → FIXED the CUDA/VRAM errors and
  slow cold starts. Verified gemma4:12b @ 64K fits 100% on GPU (9.2 GB).

• Tried capping context via Ollama instead of Hermes (OLLAMA_CONTEXT_LENGTH
  env / a Modelfile PARAMETER num_ctx). → Does NOT work: Hermes sends its own
  num_ctx per request, which overrides the server/Modelfile default.

• Server-side keep-alive (OLLAMA_KEEP_ALIVE=24h) to avoid model reload on the
  first request. → Mitigates the cold-start prompt.submit timeout, but is
  unrelated to the truncation bug.

• Note on config editing: Hermes rewrites config.yaml at runtime (e.g. on UI
  model-switch), overwriting hand edits. All config workarounds above only stick
  when applied with the app fully closed.

Summary: the combination of model.max_tokens: 16384 + model.context_length: 65536 makes it work, but both are manual workarounds for defaults that Hermes
should set automatically for custom/Ollama thinking models (as it already does
for NVIDIA NIM).

[alt-glitch]

Duplicate of #43900 — same root cause: custom/Ollama OpenAI-compatible thinking models truncate with finish_reason=length because max_tokens is unset (passed as None on the chat_completions path), so reasoning tokens consume the whole budget; plus the full advertised GGUF context (here 262144) is requested as num_ctx, allocating a huge KV-cache and exhausting VRAM. Tracking the fix on #43900 (see also open fix PR #41500).

[pippo73]

Related: Same root cause as #12152, which added a max_tokens default to fix
truncation — but only for NVIDIA NIM. custom/Ollama endpoints still get no
default, so thinking models keep truncating (finish_reason='length') on an
updated install. #10279 shows the same symptom on another provider, closed with
no fix. This report asks to extend that default (plus a sane num_ctx cap) to
custom/OpenAI-compatible/Ollama providers.

[pippo73]

Duplicate of #43900 — same root cause: custom/Ollama OpenAI-compatible thinking models truncate with finish_reason=length because max_tokens is unset (passed as None on the chat_completions path), so reasoning tokens consume the whole budget; plus the full advertised GGUF context (here 262144) is requested as num_ctx, allocating a huge KV-cache and exhausting VRAM. Tracking the fix on #43900 (see also open fix PR #41500).

Thank you , I will take a look at the posts you pointed me