AWS Bedrock models - error about prompt caching

[robertkern]

Hey we're testing out Onyx internally and really impressed with the system so far.

I have been able to get responses with some Anthropic models via AWS Bedrock but I haven't had success with any other models so wanting to know if others are experiencing the same or is it a me problem.

We're on Onyx version 2.7.1

I'm only looking at serverless models, not marketplace ones.

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: BedrockException - b'{"message":"You invoked an unsupported model or your request did not allow prompt caching. See the documentation for more information."}'

Above is the error shown in the stacktrace.

Models that are working:

• us.anthropic.claude-opus-4-5-20251101-v1:0
• us.anthropic.claude-sonnet-4-5-20250929-v1:0

Examples of models that are not working:

• google.gemma-3-12b-it
• us.anthropic.claude-3-sonnet-20240229-v1:0

They all seem to give me that unsupported model or your request did not allow prompt caching error.

Thanks!

[kimnamu]

Thanks for the detailed write-up, @robertkern — listing the exact model IDs that work and fail made this easy to trace, and the Bedrock side is quick to confirm.

The error is Bedrock rejecting a cache checkpoint on a model that doesn't support prompt caching. In 2.7.1, Onyx's prompt cache picked the Anthropic adapter for every Bedrock model (backend/onyx/llm/prompt_cache/providers/factory.py), so cache_control was sent with Gemma requests too. #7435 (shipped in v2.10.0) limits that to Bedrock model IDs containing anthropic., which fixes the Gemma case.

Checked directly against the Bedrock Converse API in us-east-1, same prompt, only the cachePoint block differs:

Model	With cachePoint	Without
google.gemma-3-12b-it	HTTP 403 "You invoked an unsupported model or your request did not allow prompt caching"	HTTP 200
us.anthropic.claude-sonnet-4-5-20250929-v1:0	HTTP 200, cacheReadInputTokens/cacheWriteInputTokens in usage	HTTP 200
us.anthropic.claude-3-haiku-20240307-v1:0	HTTP 403, same message	HTTP 404 (marked legacy)

Two notes for after upgrading:

• Anthropic models on Bedrock that aren't in the explicit prompt caching table (the Claude 3.x generation) still get cache_control, because the check only looks for anthropic. in the model ID. us.anthropic.claude-3-sonnet-20240229-v1:0 itself now returns HTTP 404 "This model version has reached the end of its life", with or without caching.
• If you need one of those, ENABLE_PROMPT_CACHING=false in the backend containers' environment turns the mechanism off entirely (backend/onyx/configs/model_configs.py, default true, present in 2.7.1 as well).

Reproduction and code

$ curl -s -w 'HTTP %{http_code}\n' -X POST https://bedrock-runtime.us-east-1.amazonaws.com/model/google.gemma-3-12b-it/converse \
  -H "Authorization: Bearer $AWS_BEARER_TOKEN_BEDROCK" -H 'Content-Type: application/json' \
  -d '{"system":[{"text":"You are terse."},{"cachePoint":{"type":"default"}}],"messages":[{"role":"user","content":[{"text":"Say hi"}]}],"inferenceConfig":{"maxTokens":20}}'
{"message":"You invoked an unsupported model or your request did not allow prompt caching. See the documentation for more information."}
HTTP 403

$ # same request without the cachePoint block
{"stopReason":"end_turn","usage":{"inputTokens":16,"outputTokens":3,"totalTokens":19}} Hi.
HTTP 200

Adapter selection, v2.7.1 vs after #7435 (79302f19b, "fix: bedrock non-anthropic prompt caching"):

# v2.7.1
elif provider in [LlmProviderNames.ANTHROPIC, LlmProviderNames.BEDROCK]:
    return AnthropicPromptCacheProvider()

# main
elif llm_config.model_provider == LlmProviderNames.ANTHROPIC or (
    llm_config.model_provider == LlmProviderNames.BEDROCK
    and ANTHROPIC_BEDROCK_TAG in llm_config.model_name  # "anthropic."
):
    return AnthropicPromptCacheProvider()

$ git tag --contains 79302f19b | grep -E '^v2\.[0-9]+\.[0-9]+$' | sort -V | head -1
v2.10.0