[Bug]: Envelope-layout cache breakpoints silently no-op during tool loops (tool messages skipped, empty-assistant markers ignored) — ~2x input cost on OpenRouter + Claude

[jeff-mettel]

Bug Description

On the OpenRouter/envelope cache layout (anthropic_prompt_cache_policy() → (True, False)), the system_and_3 strategy in agent/prompt_caching.py places its "last 3 non-system messages" breakpoints on messages that cannot carry an effective marker during agentic tool loops:

1. role: "tool" messages are skipped entirely. _apply_cache_marker() returns without doing anything for tool messages when native_anthropic=False. The existing test documents why: a top-level cache_control on role: "tool" is invalid on OpenRouter and causes a silent hang. But a marker inside a content part of a tool message is accepted and honored — verified live, details below.
2. Assistant messages with empty content (pure tool_calls turns) get msg["cache_control"] at the message top level, which OpenRouter ignores on the OpenAI-wire path.

In an agentic loop, the trailing messages are almost always tool results and empty assistant tool-call turns — so all three conversation breakpoints silently no-op, and only the system-prompt breakpoint is effective. The entire conversation (including large tool results) is re-billed at full input price on every API call until an assistant text message happens to land in the last-3 window.

Impact: roughly 2× input-token cost on every tool-heavy session via OpenRouter + Claude. In the session where we found this, ~150K tokens (~60% of session cost) were re-billed at full price across 7 consecutive calls.

Related: #20957 reports "caching not applied at all" on OpenRouter + Claude chat_completions. Our data partially contradicts that: markers in content parts (system prompt, user messages, assistant text messages — the existing string-content wrapping path) do produce real cache hits through OpenRouter. The gap is specifically tool messages and empty assistant messages. Possibly also related: #56776.

Steps to Reproduce

1. Configure a Claude model via OpenRouter (provider: openrouter, model: anthropic/claude-sonnet-5, chat_completions).
2. Run any session that triggers a multi-call tool loop (e.g. a task with 8–12 terminal/search_files calls).
3. Watch the per-call cache=X/Y figures in agent.log.

Observed pattern from a real 14-call session (Sonnet-5 via OpenRouter):

Calls	Input tokens	Cached	Note
2–4	26.5–28K	25,998 (98→93%)	system-prompt breakpoint works
5–11	45–50K	25,998 (58→52%)	cache pinned at the system prompt for 7 calls; ~20–23K re-billed full price each call
12–14	50K	49,136–50,174 (98–100%)	cache advances only after assistant text messages reach the last-3 window

Expected Behavior

Conversation breakpoints should land on messages the provider actually honors, so the growing prefix (including tool results) is cache-read on subsequent calls.

Actual Behavior

During tool loops, only the system-prompt breakpoint is effective; the conversation is re-sent uncached on every call.

Verification that the fix direction is safe (live against OpenRouter)

Two experiments against openrouter.ai/api/v1/chat/completions with anthropic/claude-haiku-4.5, marker placed inside a content part of a role: "tool" message ("content": [{"type": "text", "text": ..., "cache_control": {"type": "ephemeral"}}]):

• Hand-built payload: request 1 cached=0, cache_write=40771; identical request 2 cached=40771, cost $0.051 → $0.004. No hang, no 400.
• Same conversation shaped like a real Hermes tool loop, run through the patched apply_anthropic_cache_control(): request 2 cached=28572/28578, cost $0.036 → $0.003.

So the historical hang applies only to top-level cache_control on tool messages; the content-part form is safe and effective.

Proposed Fix

Two changes in agent/prompt_caching.py (patch below, tests included — 17/17 pass):

1. In _apply_cache_marker(): on the envelope layout, wrap a tool message's string content as [{"type": "text", "text": ..., "cache_control": ...}] (same as user messages) instead of skipping. Never set top-level cache_control on tool messages; skip only when the tool message has no content to carry the marker.
2. New _can_carry_marker() used by apply_anthropic_cache_control(): on the envelope layout, exclude messages that can only take a top-level marker (empty-content assistant/tool messages) from last-3 selection, so breakpoints land on messages that count. Native layout behavior is unchanged.

diff --git a/agent/prompt_caching.py b/agent/prompt_caching.py
index 0000000..0000000 100644
--- a/agent/prompt_caching.py
+++ b/agent/prompt_caching.py
@@ -15,16 +15,25 @@ from typing import Any, Dict, List
 def _apply_cache_marker(msg: dict, cache_marker: dict, native_anthropic: bool = False) -> None:
     """Add cache_control to a single message, handling all format variations."""
     role = msg.get("role", "")
     content = msg.get("content")
 
-    if role == "tool":
-        if native_anthropic:
-            msg["cache_control"] = cache_marker
+    if role == "tool" and native_anthropic:
+        # Native Anthropic layout: top-level marker; the adapter moves it
+        # inside the tool_result block.
+        msg["cache_control"] = cache_marker
         return
 
     if content is None or content == "":
+        if role == "tool":
+            # OpenRouter rejects top-level cache_control on role:tool (silent
+            # hang) and an empty message has no content part to carry the
+            # marker — skip. Non-empty tool content falls through below and
+            # gets the marker on a content part, which OpenRouter honors
+            # (verified live: full cache read on the second request).
+            return
         msg["cache_control"] = cache_marker
         return
 
     if isinstance(content, str):
         msg["content"] = [
             {"type": "text", "text": content, "cache_control": cache_marker}
         ]
         return
 
     if isinstance(content, list) and content:
         last = content[-1]
         if isinstance(last, dict):
             last["cache_control"] = cache_marker
 
 
+def _can_carry_marker(msg: dict, native_anthropic: bool) -> bool:
+    """True if a marker on this message is actually honored by the provider.
+
+    On the native Anthropic layout every message works (top-level markers are
+    relocated by the adapter). On the envelope layout (OpenRouter et al.) only
+    markers inside content parts are honored: empty-content messages (e.g.
+    assistant turns that are pure tool_calls) and empty tool messages would
+    receive a top-level marker the provider ignores — wasting one of the four
+    breakpoints. Skip those so the breakpoints land on messages that count.
+    """
+    if native_anthropic:
+        return True
+    content = msg.get("content")
+    if content is None or content == "":
+        return False
+    if isinstance(content, list):
+        return any(isinstance(part, dict) for part in content)
+    return isinstance(content, str)
+
+
 def _build_marker(ttl: str) -> Dict[str, str]:
@@ -71,9 +80,14 @@ def apply_anthropic_cache_control(
     if messages[0].get("role") == "system":
         _apply_cache_marker(messages[0], marker, native_anthropic=native_anthropic)
         breakpoints_used += 1
 
     remaining = 4 - breakpoints_used
-    non_sys = [i for i in range(len(messages)) if messages[i].get("role") != "system"]
+    non_sys = [
+        i
+        for i in range(len(messages))
+        if messages[i].get("role") != "system"
+        and _can_carry_marker(messages[i], native_anthropic)
+    ]
     for idx in non_sys[-remaining:]:
         _apply_cache_marker(messages[idx], marker, native_anthropic=native_anthropic)
 
     return messages

Test changes: test_tool_message_skips_marker_on_openrouter → asserts content-part wrapping (and that tool_call_id and top-level are untouched); new tests for empty-tool skip, envelope breakpoint selection skipping empty assistant turns, and native-layout behavior unchanged.

One consideration for maintainers: the qwen/OpenCode envelope paths share this function. The policy docstring in agent_runtime_helpers.py already describes the envelope layout as "markers on inner content parts", so this change aligns the tool-message behavior with that contract, but we only live-tested the OpenRouter→Anthropic route.

Happy to open a PR with the patch + tests if useful.

Affected Component

Agent core (agent/prompt_caching.py)

Version

Hermes Agent v0.18.0 (2026.7.1), upstream 30e947e0, macOS (darwin arm64), provider OpenRouter, model anthropic/claude-sonnet-5.

[Ahmett101]

I'm working on this. Branch will be at: https://github.com/Ahmett101/hermes-agent/tree/fix/57845-prompt-cache-tool-message-markers

The reporter's own diff (in the issue body) is the contract I'm implementing — I rebuilt it in-tree and verified end-to-end:

• Validated against agent/prompt_caching.py — present lines 15–79; the diff applies cleanly
• Bug-catch verified: stashed agent/prompt_caching.py → the new test suite collection-errors on the missing _can_carry_marker import; unstashed → 24/24 in tests/agent/test_prompt_caching.py + 24/24 in tests/run_agent/test_anthropic_prompt_cache_policy.py (total 48/48 in the prompt-caching family)

Plan / what landed:

• agent/prompt_caching.py:
  • _apply_cache_marker(role="tool", native_anthropic=False): instead of early-returning, fall through and wrap the (string) content as [{"type": "text", "text": ..., "cache_control": ...}]. Empty / None tool content skips (no carrier). Native Anthropic path is unchanged.
  • _can_carry_marker(msg, native_anthropic): True on native (adapter relocates top-level); on the envelope layout only carriers with non-empty content (string or non-empty list) qualify, so empty user / empty empty-assistant-tool_call / empty tool messages are filtered out of last-3 selection.
  • apply_anthropic_cache_control(...): filter non_sys through _can_carry_marker before selecting non_sys[-remaining:]. Top-level behaviour (system-first, native layout, 4-breakpoint cap) is unchanged.
• tests/agent/test_prompt_caching.py:
  • Updated test_tool_message_skips_marker_on_openrouter semantics to assert content-part wrapping (was: assert no marker at all). Documented in the docstring that we deliberately don't set top-level cache_control on tool messages.
  • 4 new _apply_cache_marker tests: empty-content tool skip; tool_call_id/name preserved across wrapping; non-tool empty-content still gets top-level marker (regression guard so the fix doesn't accidentally exclude user/assistant paths).
  • 4 new _can_carry_marker tests across native + envelope combinations.
  • 4 new envelope-breakpoint apply tests: empty-assistant tool-call turn gets skipped from the last-3 window on the envelope layout; empty tool-message carries nothing and breaks no selection; native Anthropic behavior is preserved (empty assistant + empty tool message still get the top-level marker, exactly as before).

Compose-with-other-PRs notes (worth flagging — three are in flight that touch the same envelope-cache signal):

• fix(agent): stop claiming OpenRouter Claude prompt caching #20969 (LeonSGP43) — explicitly DISABLES OpenRouter-Claude caching. If fix(agent): stop claiming OpenRouter Claude prompt caching #20969 lands first this PR is benign because the upscale path no-ops for provider=openrouter, model=anthropic/*; if this PR lands first, fix(agent): stop claiming OpenRouter Claude prompt caching #20969 will need its policy-table mental model extended to account for the corrected envelope markers. Orthogonal — the change targets different files (this PR agent/prompt_caching.py, fix(agent): stop claiming OpenRouter Claude prompt caching #20969 anthropic_prompt_cache_policy() in agent/agent_runtime_helpers.py).
• fix: enable prompt caching for Claude models on OpenAI-wire gateways #56779 (Enough1122) — ENABLES Claude caching on the OpenAI-wire. This PR assumes the envelope-layout Claude path is enabled (its premise is the same: cache markers flow to OpenRouter). Composes cleanly with fix: enable prompt caching for Claude models on OpenAI-wire gateways #56779; in fact this PR's row of fixes is the natural complement — enable the policy, then make the markers land on messages OpenRouter honors.
• feat: port OpenClacky token-saving mechanisms — idle compression, double cache markers, chunked summarization #33172 (xuezhiyi007) — adds system_and_3_double_tail strategy using two cache_control markers on the same trailing message. Independent code path (different breakpoint-selection strategy), no overlap with this PR's _apply_cache_marker content-part-wrapping change. May collide on agent/prompt_caching.py if rebased; happy to rebase onto whichever lands second.

Cross-platform impact: None (pure-Python dict-shape logic, no env vars, no paths).
Tests: 48/48 in the prompt-caching family; no touched code outside agent/prompt_caching.py + its test.
Verifier/runner unknown: the reporter is running hermes-agent 0.18.0 (an older snapshot). The diff is a literal line-for-line copy of the reporter's patch — should apply identically.

[Ahmett101]

Cross-referencing from a duplicate-effort check.

PR #57876 (LavyaTandel, branch fix/prompt-caching-markers-option-b) covers the same root cause and ships the same approach — content-part wrapping for non-empty tool content, native_anthropic-gated skip for empty/None tool and assistant turns, plus a _can_carry_marker filter used by apply_anthropic_cache_control for envelope breakpoint selection. Their description maps 1:1 to this issue's reproduction data and the verification report.

I had a parallel branch (fix/57845-prompt-cache-tool-message-markers) with the same mechanism — closing locally rather than opening a competing PR; #57876 is the single source of truth for the fix.