Claude structured output hard-fails on large schemas instead of falling back ("compiled grammar is too large")

[tombeckenham]

Summary

Since the native single-pass tool+schema combining for Claude 4.5+ (v0.22–0.23), chat({ outputSchema }) routes Claude structured output through Anthropic's native output_config. For a large/complex schema, Anthropic rejects the request and chat() emits a hard RUN_ERROR with no recovery:

output_config.format.schema: Invalid schema: The compiled grammar is too large,
which would cause performance issues. Simplify your tool schemas or reduce the
number of strict tools.

The same schema + model worked before v0.23 (it went through the lenient json_schema → tool-use path).

Why this looks fixable in the library

• @tanstack/ai already ships a fallback finalization path. For the native Anthropic adapter that path is structuredOutput(), which uses a forced structured_output tool (non-strict input_schema, so Anthropic does not compile a grammar) — exactly the lenient path that worked pre-v0.23.
• But the native path doesn't fall back to it when the provider rejects the schema — it just errors.
• strict: false does not help on the output_config path: Anthropic compiles the grammar either way.

Expected

When a provider rejects the native structured-output schema (grammar too large / too complex), gracefully fall back to the lenient forced-tool path, and/or expose a per-call strategy option so large schemas can opt into the lenient path.

Repro

• @tanstack/ai-openrouter, model anthropic/claude-sonnet-4.6 (also Opus 4.6 / 4.8).
• chat({ adapter, outputSchema, stream: true }) with a sufficiently large/nested Zod schema (~50+ properties across nested objects with many string fields; ~8KB converted JSON schema is enough).
• Smaller schemas on the same model work; every non-Anthropic provider (OpenAI GPT-5.4, Grok, Gemini Flash, DeepSeek, Mistral, …) accepts the same schema. Only Anthropic models hit it.

Versions

@tanstack/ai@0.26.0, @tanstack/ai-openrouter@0.12.1

---

Agreed design (2026-06-02)

The lenient fallback is the forced-tool-use ("structured tool") path, not prompt-based JSON. For the native Anthropic adapter, structuredOutput() already implements it; OpenRouter needs a tool-based mode added.

1. New per-call option on chat() (@tanstack/ai):

structuredOutput?: 'auto' | 'native' | 'tool' // default: 'auto'

• native — provider's native JSON-schema/grammar path (Anthropic 4.5+ output_config; OpenRouter json_schema + strict). No fallback.
• tool — force the lenient forced-tool path (Anthropic structured_output tool; new OpenRouter forced-tool mode). Skips native-combined.
• auto (default) — try native; if the provider rejects the schema, transparently re-run via the tool path.

2. Adapter contract (@tanstack/ai): new optional predicate

isStructuredOutputSchemaError?(error: unknown): boolean

Each provider recognizes its own "schema too large / invalid" rejection (Anthropic: compiled grammar is too large / output_config.format.schema). Keeps brittle provider strings out of the core.

3. Engine fallback (@tanstack/ai): in auto mode the engine records a fallback-eligible schema rejection (instead of surfacing it as a terminal RUN_ERROR/onError); the activity layer re-runs in tool mode. Streaming buffers only the pre-commit lifecycle so a recovered fallback yields one clean run.

4. Scope: fix both @tanstack/ai-anthropic (native adapter) and @tanstack/ai-openrouter (the repro). OpenRouter gains a forced-tool structured-output mode for tool/fallback.

5. Coverage: unit tests (engine fallback + per-adapter predicates), E2E (large-schema fallback scenario), docs (docs/structured-outputs/*), and the structured-outputs / adapter-configuration agent skills, plus a changeset.

[tombeckenham]

One note on this. The reason I ran into this problem is that I had code in another project that was created before native schema processing existed. The schemas in that code had a tonne of optional fields, and .catch() to string fields as I was asking the llm to produce a schema by passing it into a prompt. This used to fail often and I had to add runtime checks and make fields optional so that zod.parse wouldn't fail all the time. Those fallbacks cause the problem. It translates into a lot of union fields which quickly exceed Anthropic's limits.

I imagine a lot of others would have the same problem which is why I think a fallback is worthwhile. The proper fix is to go back through schemas and remove anything that uses .catch() or that you mark optional. Llms handle direct orders better. You make strings required and it'll fill them in