AI Gateway: Anthropic structured output (response_format → output_config) not translated

[03-CiprianoG]

Problem

When using the Helicone AI Gateway with Anthropic models (e.g., claude-4.6-sonnet/anthropic), structured output via response_format does not work. The gateway passes response_format through to Anthropic's API, but Anthropic ignores it — response_format is an OpenAI-specific parameter.

The model returns prose/markdown instead of JSON, breaking Output.object(), generateObject(), and any structured output workflow.

Root Cause

The gateway's toAnthropic() function in packages/llm-mapper/transform/providers/openai/request/toAnthropic.ts translates OpenAI request bodies to Anthropic format. It correctly handles messages, tools, tool_choice, reasoning, cache_control, context_editing, and web_search — but completely ignores response_format.

Anthropic's equivalent is output_config.format (GA since Claude Sonnet 4.5):

// OpenAI format (what the gateway receives):
{
  "response_format": {
    "type": "json_schema",
    "json_schema": { "schema": { ... }, "strict": true, "name": "response" }
  }
}

// Anthropic format (what the gateway should send):
{
  "output_config": {
    "format": {
      "type": "json_schema",
      "json_schema": { "schema": { ... }, "name": "response" }
    }
  }
}

Reproduction

Using @helicone/ai-sdk-provider with Vercel AI SDK:

import { generateText, Output } from 'ai';
import { createHelicone } from '@helicone/ai-sdk-provider';
import { z } from 'zod';

const helicone = createHelicone({ apiKey: process.env.HELICONE_API_KEY });
const model = helicone('claude-4.6-sonnet/anthropic');

const { output } = await generateText({
  model,
  output: Output.object({
    schema: z.object({
      name: z.string(),
      age: z.number(),
      hobbies: z.array(z.string()),
    }),
  }),
  prompt: 'Generate a random person profile.',
});
// ❌ Throws: NoObjectGeneratedError
// Model returns markdown like "# Person Profile\n**Name:** John..."

Helicone request log evidence

Request body sent to gateway:

{
  "model": "claude-4.6-sonnet/anthropic",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": { "type": "object", "properties": { "name": { "type": "string" }, ... } },
      "strict": true,
      "name": "response"
    }
  },
  "messages": [...]
}

Response: Prose/markdown (not JSON) — Anthropic ignores response_format.

What should happen

The toAnthropic() function should translate response_format to output_config.format:

// In toAnthropic():
if (openAIBody.response_format?.type === 'json_schema') {
  antBody.output_config = {
    format: {
      type: 'json_schema',
      json_schema: {
        schema: openAIBody.response_format.json_schema.schema,
        name: openAIBody.response_format.json_schema.name,
        ...(openAIBody.response_format.json_schema.description && {
          description: openAIBody.response_format.json_schema.description,
        }),
      },
    },
  };
}

Note: Anthropic's output_config does not support strict: true (OpenAI-specific), so it should be omitted.

Workaround

Currently using a jsonTool workaround in @helicone/ai-sdk-provider: for Anthropic models, the provider converts responseFormat into a fake json tool with tool_choice: required, forcing Claude to respond via tool call with valid JSON. This works but is a hack — native output_config support in the gateway would be the proper fix.

Related: Helicone/ai-sdk-provider#25

Environment

• AI Gateway: ai-gateway.helicone.ai
• @helicone/ai-sdk-provider: 1.0.12
• ai (Vercel AI SDK): 5.x
• Affected models: All anthropic/* and */anthropic models
• Not affected: openai/*, google/* models (they use response_format natively)

References

• Anthropic Structured Output docs — output_config.format parameter
• OpenAI Structured Output docs — response_format parameter
• Anthropic structured output (Output.object / generateObject) broken — response_format ignored ai-sdk-provider#25 — related issue on the SDK provider side