@effect/ai-openai Responses system messages encoded as string prevent prompt cache hits

[advait]

Summary

@effect/ai-openai serializes Prompt.SystemMessage for the OpenAI Responses API as an input item whose content is a plain string:

messages.push({
  role: getSystemMessageMode(config.model as string),
  content: message.content
})

For OpenAI Responses API prompt caching, that wire shape appears not to be cache-equivalent to typed input text content. In direct OpenAI API probes, large byte-stable system prompts encoded as content: string consistently reported usage.input_tokens_details.cached_tokens: 0, while the same kind of prompt encoded as content: [{ type: "input_text", text }] reported cached input tokens on repeated calls.

Versions checked

• Affected local version: @effect/ai-openai@4.0.0-beta.78
• Latest beta checked: @effect/ai-openai@4.0.0-beta.83
• Latest npm latest dist-tag checked: @effect/ai-openai@0.40.0
• Current effect-smol main also appears to use the same content: message.content system-message encoding.

Source on current effect-smol main:
https://github.com/Effect-TS/effect-smol/blob/d7eb77366731f07acb89f94b60e4ec0739220bdd/packages/ai/openai/src/OpenAiLanguageModel.ts#L816-L820

Minimal reproduction

This uses the OpenAI Responses API directly so it does not depend on any downstream application. It compares two semantically equivalent system-message encodings with a large stable prefix:

1. content: string
2. content: [{ type: "input_text", text }]

Run with Node 18+:

OPENAI_API_KEY=... node repro-openai-responses-system-cache.mjs

repro-openai-responses-system-cache.mjs:

const apiKey = process.env.OPENAI_API_KEY;
const model = process.env.OPENAI_MODEL ?? "gpt-4.1-mini";

if (!apiKey) {
  throw new Error("Set OPENAI_API_KEY before running this repro");
}

const nonce = `${Date.now()}-${Math.random().toString(16).slice(2)}`;

const stablePrompt = (label) =>
  [
    `PROMPT-CACHE-REPRO ${label} nonce=${nonce}`,
    ...Array.from(
      { length: 180 },
      (_, index) =>
        `CACHE-STABLE-${label}-${String(index + 1).padStart(4, "0")}: This stable system instruction line is intentionally identical across repeated requests for this layout.`,
    ),
  ].join("\n");

const readUsage = (responseJson) => {
  const usage = responseJson.usage ?? {};
  const details = usage.input_tokens_details ?? {};
  return {
    inputTokens: usage.input_tokens ?? 0,
    cachedTokens: details.cached_tokens ?? 0,
    outputTokens: usage.output_tokens ?? 0,
  };
};

const postResponse = async (body) => {
  const response = await fetch("https://api.openai.com/v1/responses", {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
    },
    body: JSON.stringify(body),
  });
  const json = await response.json();
  if (!response.ok) {
    throw new Error(`OpenAI request failed: ${JSON.stringify(json, null, 2)}`);
  }
  return json;
};

const runLayout = async (layoutName, systemContent) => {
  const results = [];
  for (let call = 1; call <= 3; call++) {
    const started = Date.now();
    const json = await postResponse({
      model,
      max_output_tokens: 16,
      input: [
        { role: "system", content: systemContent },
        {
          role: "user",
          content: [
            {
              type: "input_text",
              text: `Variable suffix ${call}: reply with only cache probe ${call}.`,
            },
          ],
        },
      ],
    });
    results.push({
      layout: layoutName,
      call,
      ms: Date.now() - started,
      id: json.id,
      ...readUsage(json),
    });
  }
  return results;
};

const stringSystem = await runLayout("system-content-string", stablePrompt("STRING"));
const inputTextSystem = await runLayout("system-content-input-text-array", [
  { type: "input_text", text: stablePrompt("INPUT_TEXT") },
]);
const results = [...stringSystem, ...inputTextSystem];

for (const result of results) {
  console.log(JSON.stringify(result));
}

const stringMaxCache = Math.max(...stringSystem.map((result) => result.cachedTokens));
const inputTextMaxCache = Math.max(...inputTextSystem.map((result) => result.cachedTokens));

if (stringMaxCache !== 0) {
  throw new Error(`Expected string system content to have 0 cached tokens, got ${stringMaxCache}`);
}
if (inputTextMaxCache <= 0) {
  throw new Error(
    `Expected input_text system content to report cached tokens after repeated calls, got ${inputTextMaxCache}`,
  );
}

console.error(
  `Repro passed: string max cached=${stringMaxCache}, input_text max cached=${inputTextMaxCache}`,
);

Locally observed output

I ran the script above with gpt-4.1-mini. The string-content layout had no cache hits; the typed input_text layout reported a real cache hit:

{"layout":"system-content-string","call":1,"inputTokens":4368,"cachedTokens":0,"outputTokens":16}
{"layout":"system-content-string","call":2,"inputTokens":4368,"cachedTokens":0,"outputTokens":16}
{"layout":"system-content-string","call":3,"inputTokens":4368,"cachedTokens":0,"outputTokens":16}
{"layout":"system-content-input-text-array","call":1,"inputTokens":4549,"cachedTokens":0,"outputTokens":16}
{"layout":"system-content-input-text-array","call":2,"inputTokens":4549,"cachedTokens":4352,"outputTokens":16}
{"layout":"system-content-input-text-array","call":3,"inputTokens":4549,"cachedTokens":0,"outputTokens":16}

The repro passed with:

Repro passed: string max cached=0, input_text max cached=4352

Note: OpenAI prompt caching can be opportunistic, so the script checks that at least one repeated typed input_text call reports cached tokens. The important difference is that the plain string system content never reported cached tokens in this run, while the typed input_text system content did.

Expected behavior

Prompt.SystemMessage should be encoded in the Responses API input shape that preserves OpenAI prompt-cache eligibility for large stable system prompts.

Proposed fix

Change system-message conversion from plain string content to typed input text content:

case "system": {
  messages.push({
    role: getSystemMessageMode(config.model as string),
-   content: message.content
+   content: [{ type: "input_text", text: message.content }]
  })
  break
}

This keeps the same system/developer role logic while using the wire shape that OpenAI reports as prompt-cacheable.

Why I think this belongs in @effect/ai-openai

This is not asking for manual cache management. OpenAI prompt caching remains automatic. The issue is that two semantically equivalent Responses API message encodings do not appear cache-equivalent in provider usage reports. Since @effect/ai-openai chooses the wire representation for Prompt.SystemMessage, callers cannot fix this while staying inside the provider-neutral LanguageModel API.

@effect/ai-openai already maps usage.input_tokens_details.cached_tokens into Response.Usage.inputTokens.cacheRead, so after changing the request shape downstream callers can observe provider cache hits without any additional API surface.