Bedrock Converse path never injects cachePoint — Claude prompt caching silently disabled

[aws-yz]

Summary

Bedrock Converse API path never sends cachePoint markers, so prompt caching is silently disabled for Claude on Bedrock. Users pay full input-token cost on every turn even though Bedrock has supported Claude prompt caching via cachePoint since early 2026 (5-minute default, and 1-hour extended TTL for Opus 4.5+ / Sonnet 4.5+ / Haiku 4.5+ / Opus 4.6 / Opus 4.7).

On a typical Hermes session with ~50k static prefix tokens, this is roughly a 10x input-cost overspend on every follow-up turn.

Root cause

In agent/bedrock_adapter.py, build_converse_kwargs() constructs the Converse request but never injects any cachePoint content blocks into system, toolConfig.tools, or the last user message.

Meanwhile in run_agent.py the gate for prompt caching is:

# run_agent.py:876
self._use_prompt_caching = (is_openrouter and is_claude) or is_native_anthropic

Bedrock is excluded entirely, and even if it weren't, apply_anthropic_cache_control() emits Anthropic-native cache_control markers, which the Converse API does not accept — the Converse format is a distinct cachePoint content block per AWS docs.

Repro

Run any Hermes session against global.anthropic.claude-opus-4-7 (or any Claude on Bedrock). Inspect the Converse response usage — you will never see cacheReadInputTokens / cacheWriteInputTokens fields.

Expected

For Claude models on Bedrock Converse, the agent should inject cachePoint checkpoints at stable breakpoints (system prompt, tools, last user message prefix), producing cacheReadInputTokens > 0 on follow-up turns.

Proposed fix

Inject up to 3 cachePoint blocks inside build_converse_kwargs() in agent/bedrock_adapter.py when the model is a Claude family model. Expose TTL via env var (default 5m, 1h for long sessions).

Minimal patch (verified working on Opus 4.7 in us-east-1):

# agent/bedrock_adapter.py, inside build_converse_kwargs()
_cache_ttl_env = (os.environ.get("HERMES_BEDROCK_CACHE_TTL") or "5m").strip().lower()
if _cache_ttl_env not in ("5m", "1h"):
    _cache_ttl_env = "5m"
_cache_enabled = (
    os.environ.get("HERMES_BEDROCK_PROMPT_CACHE", "1") not in ("0", "false", "False")
    and "claude" in (model or "").lower()
)
_cache_point = {"cachePoint": {"type": "default", "ttl": _cache_ttl_env}}

# after building system_prompt
if _cache_enabled and system_prompt:
    system_prompt = list(system_prompt) + [_cache_point]

# after building tool_list (inside the tool branch)
if _cache_enabled:
    tool_list = list(converse_tools) + [_cache_point]
kwargs["toolConfig"] = {"tools": tool_list}

# after converse_messages is finalized
if _cache_enabled and converse_messages:
    last = converse_messages[-1]
    if last.get("role") == "user" and isinstance(last.get("content"), list):
        last["content"] = list(last["content"]) + [_cache_point]

Verification output

Running against global.anthropic.claude-opus-4-7 in us-east-1 with a ~10k-token static prefix:

=== TTL = 5m ===
Call #1:  cacheWriteInputTokens=10870  cacheReadInputTokens=0   cacheDetails.ttl="5m"
Call #2:  cacheWriteInputTokens=17     cacheReadInputTokens=10853
→ ✅ CACHE WORKING

=== TTL = 1h ===
Call #1:  cacheWriteInputTokens=10870  cacheReadInputTokens=0   cacheDetails.ttl="1h"
Call #2:  cacheWriteInputTokens=17     cacheReadInputTokens=10853
→ ✅ CACHE WORKING

Supported models (per AWS docs)

Model	Min tokens/checkpoint	TTLs
Claude Opus 4.5 / 4.6 / 4.7	4,096	5m, 1h
Claude Sonnet 4.5	1,024	5m, 1h
Claude Haiku 4.5	4,096	5m, 1h
Claude Opus 4 / 4.1, Sonnet 4, 3.7 Sonnet, 3.5 Haiku, 3.5 Sonnet v2	1,024–2,048	5m

Max 4 cache checkpoints per request; the proposed patch uses 3.

Environment

• Hermes 4.9
• Provider: bedrock (api_mode=bedrock_converse)
• Model: global.anthropic.claude-opus-4-7
• Region: us-east-1

Happy to submit a PR if useful.

[harshidwasekar]

Bumping on this, would save a lot of cost if enabled natively

[pgregg88]

Adding a concrete data point from a recent verification of PR #12191's cachePoint patch on a real Bedrock + Opus 4.7 workload.

Setup

• Hermes Agent CLI, 6-minute interactive session
• Provider: AWS Bedrock, model us.anthropic.claude-opus-4-7 (cross-region inference profile)
• PR fix: add Bedrock Converse cache points for Claude #12191 cherry-picked locally onto main
• HERMES_BEDROCK_PROMPT_CACHE env var enabled
• 25 API calls, 53 messages

Result (from /usage)

Bucket	Tokens
Fresh input	55
Cache read	1,369,379
Cache write	42,135
Output	7,113

Cost

• Without caching: ~$7.24
• With caching: ~$1.13
• Savings: ~$6.11 per session, ~84%

Cache hit rate held at 99% across the entire session per agent.log (cache=X/Y field on every API call #N line).

This corroborates the original P1 severity tagging — for any production deployment of Hermes against Bedrock-hosted Claude, the unpatched cost is roughly 6× the patched cost on multi-turn workloads. The savings scale linearly with conversation length, so the impact is more pronounced on long-running agent sessions than on one-shot prompts.

Two adjacent gaps worth noting that surfaced while verifying this fix end-to-end:

1. Pricing-table side has no cache_read / cache_write entries for any Bedrock model. Even after PR fix: add Bedrock Converse cache points for Claude #12191 lands, /usage reports Total cost: unknown on every Bedrock session because _OFFICIAL_DOCS_PRICING only carries input_cost_per_million and output_cost_per_million for the bedrock provider rows. A user can't see the savings the patch produces without manually computing them from agent.log. (Closed PR fix(pricing): strip Bedrock regional prefixes for cost lookup #19797 fixed part of this; the cache pricing rows are still missing.)

2. Cross-region inference profile prefix isn't normalized in the pricing path. us.anthropic.claude-opus-4-7 (the canonical Bedrock form for cross-region profiles) misses the bare anthropic.claude-* keys in the table. Same root cause as fix(pricing): strip Bedrock regional prefixes for cost lookup #19797.

Both are downstream of #11970 — once cachePoint emission is flowing, the pricing layer needs to keep up so users can see what the patch is doing for them. Happy to share the _normalize_bedrock_model_id helper + cache pricing entries we ended up writing locally if it would help land the broader fix.

[JiaDe-Wu]

This is a big one. We maintain the Bedrock Converse adapter (PR #10549) and have been looking at this. The Converse API supports cachePoint markers in the system and messages arrays — the challenge is figuring out where to inject them without breaking the prompt caching invariant that Hermes relies on (no mid-session system prompt mutations). Likely approach: inject cachePoint at the end of the system prompt block and after the last user message, matching what the Anthropic SDK does automatically on the anthropic_messages path. On our roadmap but non-trivial.

[JiaDe-Wu]

Update: #12191 already implements this correctly (cachePoint on system prompt, tool list, and last user message). I reviewed it as the Bedrock adapter author and verified the approach is sound — it just needs a maintainer to assign a reviewer. Rather than open a competing PR, I've left a detailed endorsement on #12191. That's the one to watch for this fix.

[teknium1]

Thanks for the detailed report, repro, and verification output — this was a real P1 when you filed it.

The premise no longer holds on current main, for a reason that postdates this issue. On 2026-05-12 we landed dual-path Bedrock routing (hermes_cli/runtime_provider.py, the is_anthropic_bedrock_model() branch): Claude models on Bedrock now route through the AnthropicBedrock SDK (api_mode='anthropic_messages'), not the Converse API. The Converse path (build_converse_kwargs) is now reached only by non-Claude models (Nova, DeepSeek, Llama), which don't support Anthropic-style prompt caching.

On the anthropic_messages path that Claude now takes, prompt caching is fully wired:

• anthropic_prompt_cache_policy() returns (should_cache=True, native_layout=True) for Claude-on-Bedrock, so cache_control breakpoints are injected.
• normalize_usage() has an anthropic_messages branch that reads cache_read_input_tokens / cache_creation_input_tokens, so cache savings are accounted in /usage.

Verified live against the exact models in your repro:

global.anthropic.claude-opus-4-7  -> anthropic path, should_cache=True, native_layout=True
us.anthropic.claude-opus-4-7      -> anthropic path, should_cache=True, native_layout=True
anthropic.claude-sonnet-4-5       -> anthropic path, should_cache=True, native_layout=True
amazon.nova-pro-v1:0              -> Converse path (no Anthropic caching, by design)

Feeding the exact /usage numbers from the verification comment (1,369,379 cache read / 42,135 cache write) through normalize_usage(api_mode="anthropic_messages") reproduces them cleanly — no Converse cachePoint patch needed to get there.

Closing as resolved-on-main. The one genuinely-still-open piece is the Bedrock cache pricing rows that @JiaDe-Wu flagged (no cache_read/cache_write cost entries, so /usage shows "cost unknown" even though caching works) — tracking that separately in issue #50295.