Bug Description
Hermes appears to be constructing and sending extremely large prompts to local OpenAI-compatible models during agent workflows. This causes multi-minute stalls even when the model is already loaded and no llama-swap reload is occurring.
This happens across model sizes, including large architect models and smaller auditor models. It is not limited to a single GGUF model.
Steps to Reproduce
Environment
macOS
Hermes Agent using custom OpenAI-compatible provider
llama-swap in front of llama.cpp llama-server
Local endpoint: llama-swap /v1
Models launched by llama-swap using ${PORT}
llama.cpp llama-server with --metrics
Models tested include:
Gemma 4 31B IT QAT
Ornith 9B MTP
Other local GGUF models through the same Hermes/llama-swap setup
Expected Behavior
Hermes should keep prompts bounded, especially for smaller utility/auditor models. Smaller models such as 9B auditors should not receive giant accumulated prompts from the full agent session unless explicitly requested.
Hermes should avoid repeatedly sending huge reconstructed prompts when the model is already loaded and the previous interaction was only seconds earlier.
Actual Behavior
Hermes sometimes pauses for several minutes before a response, even when:
the model is already loaded
no other model is resident
the agent was active only 10–20 seconds earlier
llama-swap is not visibly reloading the model
the behavior appears across different model sizes
Why this looks like a Hermes issue
This is not just a slow model issue. The same pattern appears with smaller models such as Ornith 9B MTP. That suggests the bottleneck is Hermes prompt/context construction, model routing, compression behavior, or tool-result/history accumulation.
The issue appears to be that Hermes is sending extremely large prompts to local models during agent workflows, possibly including:
full prior tool output
repeated file contents
skill/tool/MCP context
accumulated agent history
uncompressed or late-compressed session state
auxiliary/background calls routed to the wrong model
llama-swap / llama-server notes
The models are served through llama-swap, but the model is already loaded when the stall occurs. The delay correlates with prompt-prefill volume, not model loading.
Example model command pattern:
/opt/homebrew/bin/llama-server
--host 127.0.0.1
--port ${PORT}
-m ".gguf"
--ctx-size 131072
--parallel 1
--cache-prompt
--metrics
--jinja
--cache-type-k f16
--cache-type-v f16
Using f16 KV improved runtime responsiveness for Gemma, but did not solve the underlying issue of Hermes sending very large prompts.
Request
Please add diagnostics or controls for Hermes context construction, such as:
Show actual prompt token count before each model call.
Show which component contributed tokens:
system prompt
tools
skills
memory
MCP context
file reads
terminal output
prior messages
compression summary
Allow per-model context caps for auxiliary/auditor models.
Allow routing background/auxiliary calls away from main architect models.
Provide a setting to truncate or summarize tool output earlier.
Provide a debug mode that logs why a prompt became very large.
Prevent low-latency utility/auditor models from receiving full-session prompts by default.
Impact
This makes Hermes difficult to use with local models because even small models can stall for minutes if Hermes sends a giant prompt. The issue is especially painful for agent workflows where Hermes asks the user to do something, the user responds seconds later, and Hermes then appears to hang while processing a massive prompt.
Affected Component
Other
Messaging Platform (if gateway-related)
No response
Debug Report
Report https://dpaste.com/E6XA6DQ4W
agent.log https://dpaste.com/FVWB56236
gateway.log https://dpaste.com/FRF9P5CBY
gui.log https://dpaste.com/HDANYL6HZ
desktop.log https://dpaste.com/3UC4YJDGU
Operating System
macOS Tahoe 26.5.2
Python Version
3.11.15
Hermes Version
0.18.2 (2026.7.7.2) upstream 56a8e81
Additional Logs / Traceback (optional)
Metrics evidence from llama.cpp /metrics
From one Gemma 4 31B session:
llamacpp:prompt_tokens_total 570967
llamacpp:prompt_seconds_total 3001.8
llamacpp:tokens_predicted_total 21142
llamacpp:tokens_predicted_seconds_total 1358.87
llamacpp:n_tokens_max 110203
llamacpp:prompt_tokens_seconds 190.208
llamacpp:predicted_tokens_seconds 15.5585
llamacpp:requests_processing 1
llamacpp:requests_deferred 1
llamacpp:n_busy_slots_per_decode 1
The important metric is:
llamacpp:n_tokens_max 110203
At roughly 190 prompt tokens/sec, a 110k-token prompt prefill can take many minutes before generation begins.
Root Cause Analysis (optional)
No response
Proposed Fix (optional)
No response
Are you willing to submit a PR for this?
Bug Description
Hermes appears to be constructing and sending extremely large prompts to local OpenAI-compatible models during agent workflows. This causes multi-minute stalls even when the model is already loaded and no llama-swap reload is occurring.
This happens across model sizes, including large architect models and smaller auditor models. It is not limited to a single GGUF model.
Steps to Reproduce
Environment
macOS
Hermes Agent using custom OpenAI-compatible provider
llama-swap in front of llama.cpp llama-server
Local endpoint: llama-swap /v1
Models launched by llama-swap using ${PORT}
llama.cpp llama-server with --metrics
Models tested include:
Gemma 4 31B IT QAT
Ornith 9B MTP
Other local GGUF models through the same Hermes/llama-swap setup
Expected Behavior
Hermes should keep prompts bounded, especially for smaller utility/auditor models. Smaller models such as 9B auditors should not receive giant accumulated prompts from the full agent session unless explicitly requested.
Hermes should avoid repeatedly sending huge reconstructed prompts when the model is already loaded and the previous interaction was only seconds earlier.
Actual Behavior
Hermes sometimes pauses for several minutes before a response, even when:
the model is already loaded
no other model is resident
the agent was active only 10–20 seconds earlier
llama-swap is not visibly reloading the model
the behavior appears across different model sizes
Why this looks like a Hermes issue
This is not just a slow model issue. The same pattern appears with smaller models such as Ornith 9B MTP. That suggests the bottleneck is Hermes prompt/context construction, model routing, compression behavior, or tool-result/history accumulation.
The issue appears to be that Hermes is sending extremely large prompts to local models during agent workflows, possibly including:
full prior tool output
repeated file contents
skill/tool/MCP context
accumulated agent history
uncompressed or late-compressed session state
auxiliary/background calls routed to the wrong model
llama-swap / llama-server notes
The models are served through llama-swap, but the model is already loaded when the stall occurs. The delay correlates with prompt-prefill volume, not model loading.
Example model command pattern:
/opt/homebrew/bin/llama-server
--host 127.0.0.1
--port ${PORT}
-m ".gguf"
--ctx-size 131072
--parallel 1
--cache-prompt
--metrics
--jinja
--cache-type-k f16
--cache-type-v f16
Using f16 KV improved runtime responsiveness for Gemma, but did not solve the underlying issue of Hermes sending very large prompts.
Request
Please add diagnostics or controls for Hermes context construction, such as:
Show actual prompt token count before each model call.
Show which component contributed tokens:
system prompt
tools
skills
memory
MCP context
file reads
terminal output
prior messages
compression summary
Allow per-model context caps for auxiliary/auditor models.
Allow routing background/auxiliary calls away from main architect models.
Provide a setting to truncate or summarize tool output earlier.
Provide a debug mode that logs why a prompt became very large.
Prevent low-latency utility/auditor models from receiving full-session prompts by default.
Impact
This makes Hermes difficult to use with local models because even small models can stall for minutes if Hermes sends a giant prompt. The issue is especially painful for agent workflows where Hermes asks the user to do something, the user responds seconds later, and Hermes then appears to hang while processing a massive prompt.
Affected Component
Other
Messaging Platform (if gateway-related)
No response
Debug Report
Operating System
macOS Tahoe 26.5.2
Python Version
3.11.15
Hermes Version
0.18.2 (2026.7.7.2) upstream 56a8e81
Additional Logs / Traceback (optional)
Root Cause Analysis (optional)
No response
Proposed Fix (optional)
No response
Are you willing to submit a PR for this?