Prompt caching for Anthropic is not working even with the new options

[rajatady]

We have tried adding the following in three different versions:

providerOptions: {
                    openrouter: {
                        // cache_control also works
                        // cache_control: { type: 'ephemeral' }
                        cacheControl: { type: 'ephemeral' },
                    },
                },
                experimental_providerMetadata: {
                    openrouter: {
                        // cache_control also works
                        // cache_control: { type: 'ephemeral' }
                        cacheControl: { type: 'ephemeral' },
                    },
                },
                // @ts-ignore
                providerMetadata: {
                    openrouter: {
                        // cache_control also works
                        // cache_control: { type: 'ephemeral' }
                        cacheControl: { type: 'ephemeral' },
                    },
                },

and added directly at the message level as well as individual parts. The caching with open router is not working at all.

We added a hack to override the fetch implementation and overriding the body but even then only the system prompt caching works.

[louisgv]

cc @matannahmani

Note that anthropic caching only trigger if the message is >1024 or so tokens. Docs ref: https://openrouter.ai/docs/features/prompt-caching#anthropic-claude

[rajatady]

The message size is close to 10k tokens.

[LouisCuvelier]

I can confirm I'm getting the same issue. My system prompt is ~4500 tokens.

On Openrouter's Activity page, this is always : "usage_cache": null,.

  streamText({
    model: openrouter("anthropic/claude-3.7-sonnet"),
    messages: [
      {
        role: "system",
        content: systemPrompt,
        providerMetadata: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
        experimental_providerMetadata: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
        providerOptions: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
      },
      ...messages,
    ],
  });

In case it can help, I'm already passing extraBody to the openrouter() instance.

export function openrouter(
  model: OpenRouterModel,
  options?: OpenRouterOptions,
): LanguageModelV1 {
  const defaultOptions: OpenRouterOptions = {
    isOnline: false,
    sort: "price",
    maxResults: 3,
  };

  const provider = createOpenRouter({
    apiKey: env.OPENROUTER_API_KEY,
    extraBody: {
      provider: {
        sort: options?.sort || defaultOptions?.sort,
      },
      ...((options?.isOnline || defaultOptions?.isOnline) && {
        plugins: [
          {
            id: "web",
            max_results: options?.maxResults || defaultOptions?.maxResults,
          },
        ],
      }),
    },
  });

  return provider(model);
}

Versions :

    "@openrouter/ai-sdk-provider": "^0.4.5",
    "@ai-sdk/react": "^1.2.9",
    "ai": "^4.3.6",

[rajatady]

I can confirm I'm getting the same issue. My system prompt is ~4500 tokens.

On Openrouter's Activity page, this is always : "usage_cache": null,.

  streamText({
    model: openrouter("anthropic/claude-3.7-sonnet"),
    messages: [
      {
        role: "system",
        content: systemPrompt,
        providerMetadata: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
        experimental_providerMetadata: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
        providerOptions: {
          openrouter: {
            cacheControl: { type: "ephemeral" },
          },
        },
      },
      ...messages,
    ],
  });

In case it can help, I'm already passing extraBody to the openrouter() instance.

export function openrouter(
  model: OpenRouterModel,
  options?: OpenRouterOptions,
): LanguageModelV1 {
  const defaultOptions: OpenRouterOptions = {
    isOnline: false,
    sort: "price",
    maxResults: 3,
  };

  const provider = createOpenRouter({
    apiKey: env.OPENROUTER_API_KEY,
    extraBody: {
      provider: {
        sort: options?.sort || defaultOptions?.sort,
      },
      ...((options?.isOnline || defaultOptions?.isOnline) && {
        plugins: [
          {
            id: "web",
            max_results: options?.maxResults || defaultOptions?.maxResults,
          },
        ],
      }),
    },
  });

  return provider(model);
}

Versions :

    "@openrouter/ai-sdk-provider": "^0.4.5",
    "@ai-sdk/react": "^1.2.9",
    "ai": "^4.3.6",

I got the system prompt caching to work by extending fetch, intercepting the request and adding the param right before the request is sent out. But this did not solve the caching issue for the User messages.

[rajatady]

@louisgv Any idea why the user message caching is not working? This is a critical feature for us. Is this working for anyone?

[louisgv]

Will try with a large system prompt on my end to see if we can reproduce.

It seems that the issue happens for he following:

[ Large system (w. cache), user]

[ Large user (w. cache), ]

Are there any other cases?

[mjp0]

Not working for me either on Gemini models. Requests are processed normally but cache is not activated. System message is 189794 tokens, so it definitely should work. I've multiple cached messages at the beginning of the convo, but I've tried caching only one, etc. but nothing works.

Messages look like this:

{
      role: "system",
      content: "...long system message",
      providerMetadata: {
        openrouter: {
          cacheControl: { type: 'ephemeral' },
        },
      },
    },

[louisgv]

So the proper shape for prompt caching looks like this:

messages: [
          {
            role: 'system',
            content:
              'You are a podcast summary assistant. You are detail oriented and critical about the content.',
          },
          {
            role: 'user',
            content: [
              {
                type: 'text',
                text: 'Given the text body below:',
              },
              {
                type: 'text',
                text: `<LARGE BODY OF TEXT>`,
                providerOptions: {
                  openrouter: {
                    cacheControl: { type: 'ephemeral' },
                  },
                },
              },
              {
                type: 'text',
                text: 'Who're the speakers?',
              },
            ],
          },
        ],

Since the content part maps 1-1 with the message content for Anthropic.

[NathanColosimo]

Is this working for anyone? My format is like so:

return [
                { 
                    role: "system", 
                    content: systemPrompt, 
                    providerOptions: { 
                        anthropic: { cacheControl: { type: "ephemeral" } 
                        } 
                    } 
                },
                {
                    role: "user",
                    content: [
                        {
                            type: "text",
                            text: userPrompt,
                            providerOptions: {
                                openrouter: { cacheControl: { type: "ephemeral" } }
                            }
                        }
                    ]
                }
            ];

I also tried without the providerOptions in the system prompt and just cache control on the user. Still didn't work.

Are there any patterns at all that are working for anyone? Do I need to have text before and after the cached user prompt?

[rajatady]

No. Its not working for me too. Tried a lot of different things.

…

On Thu, 24 Apr 2025 at 10:19 PM, Nathan Colosimo ***@***.***> wrote: *NathanColosimo* left a comment (OpenRouterTeam/ai-sdk-provider#35) <#35 (comment)> Is this working for anyone? My format is like so: return [ { role: "system", content: systemPrompt, providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } }, { role: "user", content: [ { type: "text", text: userPrompt, providerOptions: { openrouter: { cacheControl: { type: "ephemeral" } } } } ] } ]; I also tried without the providerOptions in the system prompt and just cache control on the user. Still didn't work. Are there any patterns at all that are working for anyone? Do I need to have text before and after the cached user prompt? — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWMVWPLT7XWUEB7UZPP4NL23EIZXAVCNFSM6AAAAABYFHMWOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMRYGI2TSNZWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[rajatady]

@louisgv Is anything at all being done about this? I even created a sample project to test the caching for user requests, and yet its not working. We are on track to do over 1 Billion tokens each month. We have already paid over $1600 to Openrouter just in the last 10 days and its growing 10-20% nearly week on week.

So if this is not a priority for you guys to have a look at and fix, I think its best we take our business somewhere else. Its been nearly 2 months since this issue was opened.

[peterje]

Also appear to be affected by this. Previously had Anthropic cache hit rates in the 60% range. After porting to OpenRouter and preserving renamed providerOptions, cache rate is 0%.

@rajatady @mjp0 Did you ever find a resolution?

[mjp0]

Also appear to be affected by this. Previously had Anthropic cache hit rates in the 60% range. After porting to OpenRouter and preserving renamed providerOptions, cache rate is 0%.

@rajatady @mjp0 Did you ever find a resolution?

Nope, it's broken and it's not looking like the team here is particularly interested fixing this. I switched to Gemini API.