fix: stop sending strict tools to Anthropic by default - #49
Merged
Conversation
Anthropic compiles every strict tool schema on a request into one decoding grammar and rejects the whole request with "The compiled grammar is too large" past an undocumented ceiling. alphaXiv's 13-tool assistant clears it, so every turn 400d on claude-sonnet-4-6 while the same payload compiled fine on claude-sonnet-5. Trimming enums, descriptions or whole schema keywords never got under it; only deleting tools did. Strict is now off unless a model opts in with `strictTools`, which is only realistic for a small fixed toolset. Losing it costs little in practice: it only shapes arguments once the model has already decided to call a tool, and a 39-query researcher corpus on claude-sonnet-4-6 with strict off produced 151 tool calls with zero rejected arguments and zero invented researcher links, matching the recorded gpt-5.6 baseline. String formats and their regex patterns also move into the prompt instructions for Anthropic, next to the length and numeric bounds already handled that way. Zod pairs z.iso.date() with a leap-year-aware regex that a grammar has to compile literally, and "must be a valid date" says the same thing. OpenAI keeps them native. Assisted-by: claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthropic compiles every strict tool schema on a request into one decoding grammar and rejects the whole request with "The compiled grammar is too large" past an undocumented ceiling. alphaXiv's 13-tool assistant clears it, so every turn 400d on claude-sonnet-4-6 while the same payload compiled fine on claude-sonnet-5. Trimming enums, descriptions or whole schema keywords never got under it; only deleting tools did.
Strict is now off unless a model opts in with
strictTools, which is only realistic for a small fixed toolset. Losing it costs little in practice: it only shapes arguments once the model has already decided to call a tool, and a 39-query researcher corpus on claude-sonnet-4-6 with strict off produced 151 tool calls with zero rejected arguments and zero invented researcher links, matching the recorded gpt-5.6 baseline.String formats and their regex patterns also move into the prompt instructions for Anthropic, next to the length and numeric bounds already handled that way. Zod pairs z.iso.date() with a leap-year-aware regex that a grammar has to compile literally, and "must be a valid date" says the same thing. OpenAI keeps them native.
Assisted-by: claude-opus-5