Leverage Anthropic's cache with generateObject

[theboolean]

Hey! I noticed that using generateObject function, it seems impossible to leverage Anthropic's cache when the schema changes.

e.g. in the following example:

// request 1
await generateObject({
  model: anthropic('claude-3-5-sonnet-20241022', {
    cacheControl: true,
  }),
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: context,
          experimental_providerMetadata: {
            anthropic: { cacheControl: { type: 'ephemeral' } },
          },
        },
      ],
    },
  ],
  schema: z.object({
    field1: z.string(),
  }),
});

// request 2
await generateObject({
  model: anthropic('claude-3-5-sonnet-20241022', {
    cacheControl: true,
  }),
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: context,
          experimental_providerMetadata: {
            anthropic: { cacheControl: { type: 'ephemeral' } },
          },
        },
      ],
    },
  ],
  schema: z.object({
    field2: z.string(), // <-- only this line changed
  }),
});

will write another cache for the second request, instead of using the written cache from the first, and the only thing that changed is the JSON schema (it works correctly if I ask again for field1).

My use case is: context is huge, and I need different LLM calls to address different needs (some calls for example need previous results to work), and I'd love to be able to cache it and add some following messages with the different instructions.

I understand Anthropic uses tools, system and messages (in this order) to prefix the caches, but wondering if there is something I can do to properly cache only the prompt, and reuse it for different JSON outputs (or even between generateObject/generateText requests).

Thanks for any kind of help! 🙏

[timholwerda]

The issue stems from how caching prompts work in relation to the generateObject function's structured output implementation.

According to Anthropic's documentation: "Prompt caching references the entire prompt - tools, system, and messages (in that order) up to and including the block designated with cache_control."

Looking into the generateObject implementation, it leverages the defined schema in different ways based on the mode parameter:

• 'auto': The provider selects the optimal mode for the model.
• 'tool': A tool with the JSON schema as parameters is provided, and the provider is instructed to use it.
• 'json': The JSON schema and instructions are injected into the system prompt. If the provider supports JSON mode, it's enabled. If the provider supports JSON grammars, the grammar is used.

Because the schema is injected into the system prompt, any schema change will always invalidate the cache according to the caching order mentioned above.

I created a workaround using generateText that uses cache. This approach attempts to achieve the same structured output as generateObject. However, it has limitations:

• It doesn't validate the output schema or throw errors if object generation fails.
• It doesn't provide the schema in system messages, as the CoreSystemMessage type doesn't accept Parts like CoreUserMessage does (according to the AI SDK documentation: https://sdk.vercel.ai/docs/reference/ai-sdk-core/core-message#coresystemmessage).

const { text, usage, providerMetadata } = await generateText({
      model: anthropic(anthropicModel),
      temperature: 0,
      maxTokens: 64000,
      messages: [
        {
          role: 'system',
          content: systemPrompt // Prompts to only respond using a user provided JSON schema.
        },
        {
          role: 'user',
          content: [
            {
              type: 'file',
              data: pdfBase64,
              mimeType: 'application/pdf'
            }
          ],
          providerOptions: {
            anthropic: {
              cacheControl: { type: 'ephemeral' },
            }
          }
        },
        {
          role: 'user',
          content: [
            {
              type: 'text',
              text: userPrompt // Prompt includes the Zod schema converted to JSON.
            }
          ]
        }
      ]
    });

     // The anthropicParser function is just a JSON.parse wrapped by try/catch, but could be extended to include validation.
    const object = anthropicParser(text);

    return {
      prompt: object,
      usage: usage,
    };