Bug Description
When using Hermes Agent with LMCache (L1 RAM + L2 disk) for multi-turn vision_analyze tool calls, if two calls share the exact same question text but target different image_url, LMCache's KV cache experiences a cache key collision. This causes the LLM to hallucinate a seamless concatenation of two different images in its analysis output.
The root cause is that LMCache computes its cache key from the text prefix only (the question parameter), without incorporating the image into the key. Hermes' inject_image_file_paths() middleware uses a fixed question template and LLM tend to use the same prompt from previous vision_analyze tool call, making this collision inevitable in multi-image scenarios.
Steps to Reproduce
- Environment: Hermes Agent + LMCache (L1=55GB RAM, L2=200GB disk, blend engine, IsolatedLRU) + vLLM + Open WebUI with custom image injection middleware
- Upload Image A (e.g., Android home screen screenshot) via Open WebUI, send a message that triggers
vision_analyze
- Upload Image B (e.g., a person photo), send a message with the identical prompt that triggers
vision_analyze
- Both calls use the same
question: 這是一張什麼圖片？詳細描述所有細節，包括人物、服裝、姿勢、背景、氛圍等。
- Observe the analysis result for Image B
Expected Behavior
Image B's analysis should describe only Image B's content (the person), with no influence from Image A.
Actual Behavior
Image B's analysis produces a perfect 50/50 splice hallucination of both images: the upper half describes Image A's phone screen icons, the lower half describes Image B's person's legs, as if the two images were seamlessly composited together. The actual image files (verified on disk) and the base64 payload sent to the LLM backend are both correct — the bug occurs exclusively at the KV cache layer.
Evidence
# tool call example
{
"tool_name": "vision_analyze",
"image_url": "/home/thomas2018/image_cache/7a65eaf5-3766-4374-b7b5-493f1618b3b2.jpg",
"question": "這是一張什麼圖片？詳細描述所有細節，包括人物、服裝、姿勢、背景、氛圍等。"
}
| Call |
Image |
Question |
Result |
| 1 |
d34dd879...jpg (Android home screen) |
Prompt A |
✅ Correct: Describes the phone's home screen. |
| 2 |
7a65eaf5...jpg (Cosplay) |
Same Prompt A |
❌ Error: The illusion of a phone icon combined with a woman's legs. |
| 3 |
7a65eaf5...jpg (Cosplay) |
Prompt A with random numbers 7382910283 in font |
✅Correct: Describing a cosplay character |
Call 2 and Call 3 target the same image file, but only Call 3 (with modified question) returns correct results. This confirms the issue is purely a cache key collision.
Affected Component
Messaging Platform (if gateway-related)
Debug Report
version: 0.18.2 [b8880f12] (2026-07-10)
os: Linux 6.12.75+rpt-rpi-v8 aarch64
python: 3.11.15
openai_sdk: 2.24.0
profile: coder
hermes_home: /opt/hermes/profiles/coder
model: qwen-27b-default
provider: custom
terminal: local
features:
toolsets: file, terminal, skills, memory, session_search, clarify, todo, patch, delegate_task
mcp_servers: 0
memory_provider: built-in
gateway: running (systemd (user))
platforms: discord
cron_jobs: 0
skills: 117
config_overrides:
agent.max_turns: 300
agent.gateway_timeout: 3600
compression.threshold: 0.92
display.streaming: True
display.show_reasoning: False
Operating System
Ubuntu 24.04 (Linux 6.12.75+rpt-rpi-v8, ARM64 / Raspberry Pi 5 class)
Python Version
3.11.15
Hermes Version
0.18.2 (2026.7.7.2) · upstream b8880f1
Root Cause Analysis
LMCache's KV cache key is computed solely from the prompt's text prefix. When vision_analyze calls share the same question parameter, LMCache hits a previously cached KV entry from a different image, causing the LLM to decode using incorrect key-value pairs.
The inject_image_file_paths() middleware in Open WebUI (open_webui/utils/middleware.py L2020-L2171) injects vision_analyze calls with a fixed question template, providing no per-image unique identifier. This makes cache collision deterministic whenever multiple images are analyzed with the same prompt.
This is not a Hermes bug per se — it is an LMCache design limitation where image tokens are not included in the cache key computation. However, Hermes can implement a defensive workaround at the tool wrapper layer.
Proposed Fix
Add an automatic image fingerprint to the question parameter in the vision_analyze tool wrapper or in inject_image_file_paths(), ensuring each call has a unique cache key:
# tool call example
{
"tool_name": "vision_analyze",
"image_url": "/home/thomas2018/image_cache/7a65eaf5-3766-4374-b7b5-493f1618b3b2.jpg",
"question": " {KEY_HERE} 這是一張什麼圖片？詳細描述所有細節，包括人物、服裝、姿勢、背景、氛圍等。" <----
}
import hashlib
# In inject_image_file_paths() when constructing the vision_analyze call
file_hash = hashlib.md5(cached_path.encode()).hexdigest()[:8]
question = f"[img:{file_hash}] {original_question}"Or more robustly, at the tool wrapper layer with UUID:
import uuid
def vision_analyze_wrapper(path: str, question: str):
cache_bust = f"[cache:{uuid.uuid4().hex[:8]}]"
return actual_vision_analyze(path=path, question=f"{cache_bust} {question}")This workaround has zero impact on LLM reasoning (the fingerprint is at the prompt prefix and the LLM ignores it), but it completely eliminates LMCache collision with 100% reliability.
Alternative: Upstream fix
The ideal fix is for LMCache to incorporate the image hash or image token prefix into the KV cache key computation for multimodal requests. But for now I would like to uses a workaround in hermes to make it working.
Bug Description
When using Hermes Agent with LMCache (L1 RAM + L2 disk) for multi-turn
vision_analyzetool calls, if two calls share the exact samequestiontext but target differentimage_url, LMCache's KV cache experiences a cache key collision. This causes the LLM to hallucinate a seamless concatenation of two different images in its analysis output.The root cause is that LMCache computes its cache key from the text prefix only (the
questionparameter), without incorporating the image into the key. Hermes'inject_image_file_paths()middleware uses a fixedquestiontemplate and LLM tend to use the same prompt from previous vision_analyze tool call, making this collision inevitable in multi-image scenarios.Steps to Reproduce
vision_analyzevision_analyzequestion:這是一張什麼圖片？詳細描述所有細節，包括人物、服裝、姿勢、背景、氛圍等。Expected Behavior
Image B's analysis should describe only Image B's content (the person), with no influence from Image A.
Actual Behavior
Image B's analysis produces a perfect 50/50 splice hallucination of both images: the upper half describes Image A's phone screen icons, the lower half describes Image B's person's legs, as if the two images were seamlessly composited together. The actual image files (verified on disk) and the base64 payload sent to the LLM backend are both correct — the bug occurs exclusively at the KV cache layer.
Evidence
d34dd879...jpg(Android home screen)7a65eaf5...jpg(Cosplay)7a65eaf5...jpg(Cosplay)7382910283in fontCall 2 and Call 3 target the same image file, but only Call 3 (with modified question) returns correct results. This confirms the issue is purely a cache key collision.
Affected Component
Messaging Platform (if gateway-related)
Debug Report
Operating System
Ubuntu 24.04 (Linux 6.12.75+rpt-rpi-v8, ARM64 / Raspberry Pi 5 class)
Python Version
3.11.15
Hermes Version
0.18.2 (2026.7.7.2) · upstream b8880f1
Root Cause Analysis
LMCache's KV cache key is computed solely from the prompt's text prefix. When
vision_analyzecalls share the samequestionparameter, LMCache hits a previously cached KV entry from a different image, causing the LLM to decode using incorrect key-value pairs.The
inject_image_file_paths()middleware in Open WebUI (open_webui/utils/middleware.pyL2020-L2171) injects vision_analyze calls with a fixed question template, providing no per-image unique identifier. This makes cache collision deterministic whenever multiple images are analyzed with the same prompt.This is not a Hermes bug per se — it is an LMCache design limitation where image tokens are not included in the cache key computation. However, Hermes can implement a defensive workaround at the tool wrapper layer.
Proposed Fix
Add an automatic image fingerprint to the
questionparameter in thevision_analyzetool wrapper or ininject_image_file_paths(), ensuring each call has a unique cache key:Or more robustly, at the tool wrapper layer with UUID:
This workaround has zero impact on LLM reasoning (the fingerprint is at the prompt prefix and the LLM ignores it), but it completely eliminates LMCache collision with 100% reliability.
Alternative: Upstream fix
The ideal fix is for LMCache to incorporate the image hash or image token prefix into the KV cache key computation for multimodal requests. But for now I would like to uses a workaround in hermes to make it working.