RFC: custom-grammar tools for the default catalog (gpt-5.5+)

[justrach]

Context

2083d6ea landed the gating infrastructure: any ToolDefinition with grammar: Some(ToolGrammar { ... }) becomes an OpenAI Tool::Custom on gpt-5.5+ and silently falls back to a Tool::Function (JSON-schema) on every other model. The plumbing is in place; no default tool currently declares a grammar.

The open question: should we retrofit grammars onto existing default tools (e.g. read, fs_search) so they take advantage of the feature on gpt-5.5+?

Why it's tempting

• Compactness. read foo.rs:10-20 is fewer tokens than {\"file_path\":\"foo.rs\",\"range\":{\"start_line\":10,\"end_line\":20}}. Multiplied across millions of tool calls, this is real latency + cost savings.
• Token-level constraint. A grammar prevents the model from emitting invalid syntax at all (it's enforced at decode time), where JSON-schema is advisory — the model can still produce malformed JSON we have to reject and retry.
• Model affinity. Anecdotal evidence suggests gpt-5.5+ handles single-expression tool calls more naturally than JSON object construction.

Why it's not obvious

1. Multi-provider asymmetry. Anthropic, Google, Codex, Bedrock, OpenAI chat-completions don't support custom grammars. Every grammar-enabled tool needs two input shapes — a flat grammar form (gpt-5.5+) and a JSON form (everything else). That's twice the surface area to test, document, and keep in sync.
2. Parser cost. JSON args round-trip via serde for free. A grammar tool's text output has to be parsed back into the executor's structured input — every grammar tool is one more hand-written or generated parser, with its own bugs and edge cases.
3. Brittle to extension. Adding an optional field to a JSON-schema tool is one line. Adding it to a grammar tool means rewriting the grammar, the parser, and refreshing the test corpus.
4. No win for already-structured inputs. Default catalog tools (read, write, patch, fs_search, shell, fetch, todo_*) are all multi-field structured calls. JSON-schema is doing real work for them — type-checking integers, enforcing required fields, gating enums. Flattening into one grammar string strictly removes information.
5. Debuggability. JSON-schema validation errors are clear. A grammar parse failure mid-token is harder to diagnose, and partial parses can succeed silently with the wrong fields.

Where grammar would actually help

Grammar tools shine when the whole input is one free-form expression, not when there's structured metadata. Plausible new tools, not retrofits:

• math_eval — `2 * (3 + 4)`. Grammar enforces well-formed arithmetic.
• sql_query with a known schema — grammar can enforce table/column names exist.
• code_search_dsl — `def:foo type:fn file:*.rs` style queries with a strict vocabulary.
• regex_tester — input is one regex, grammar enforces meta-regex syntax.
• shell_constrained — a vocabulary-restricted shell DSL (only `ls`, `grep`, `cat`, etc.).

The default catalog has none of these shapes today.

Recommendation

Don't retrofit read, fs_search, etc. The cost (dual input shapes, parser maintenance, brittle to extension) outweighs the marginal compactness/correctness win on a single provider tier.

Do prototype on a new, grammar-native tool if/when one shows up. That gives us:

• A real measurement of the compactness/correctness benefit on gpt-5.5+.
• A reference implementation of the parser pattern.
• Something we can A/B test against an equivalent JSON-schema tool.

If after one experiment grammars demonstrably outperform JSON for our workload, then revisit retrofitting — but with data, not vibes.

Open questions

• What's the actual token-savings and tool-call success-rate delta on gpt-5.5+? (Run an eval on a grammar-vs-JSON pair before deciding.)
• How does grammar-tool latency compare to JSON-tool latency on the OpenAI Responses API?
• Is there a candidate first-mover tool worth designing grammar-first? Math eval? Structured-query search? Something else?

Related

• Grammar feature: a7751534
• Gating: 2083d6ea

[justrach]

Research findings (post-RFC)

Pulled together evidence from OpenAI docs, community reports, and structured-output benchmarks. The picture is more nuanced than the original post suggested.

Where OpenAI itself recommends custom grammar tools

From the GPT-5 cookbook, the documented examples are exactly the kinds of free-form expression inputs the RFC predicted — not retrofits of existing structured tools:

• PostgreSQL queries — Lark grammar constraining output to SELECT … LIMIT … WHERE/ORDER BY shape.
• Timestamps — regex grammar enforcing date/time formats.
• Code snippets / DSLs — pattern-constrained text.

OpenAI's own framing: "useful for generating structured text like SQL queries, code snippets, or any output that must match a specific pattern."

Empirical evidence on grammar-constrained generation

JSONSchemaBench (Geng et al., 2025, arXiv:2501.10868) — 10K real-world schemas across Guidance, Outlines, llama.cpp, XGrammar, OpenAI, Gemini. Reports a real reliability/accuracy tradeoff.

Key findings (ACL 2025):

• Reliability win: constrained decoding reduces JSON formatting errors from 38.2% → 0% in zero-shot scenarios.
• Accuracy cost: models drop 17.1% accuracy on structured tasks when constrained.

For Forge specifically: gpt-5+ class models already have very low JSON-formatting error rates with retries, so the 38.2 pp reduction is mostly captured by the retry loop already. The 17.1 pp accuracy drop, however, applies on top — the model becomes more conservative in token choice when constrained, which can degrade the content of the call even when the shape is correct.

CFG conformance is not strictly guaranteed on GPT-5

Surprising finding: GPT-5 custom Lark outputs sometimes violate the supplied grammar. OpenAI engineer wzhang acknowledged: "In all instances the model is constrained to the grammar. Sometimes the model can go out of distribution on unbounded regexes."

Practical implications:

• Unbounded regex patterns are unreliable. Bounded enum forms (CITY: "prague" | "paris") work; /[^\s\+\:]+/ does not.
• Output validation is still required. You can't skip the parser/validator step on the grounds that "the grammar enforces it" — empirically it doesn't always, especially with permissive patterns.
• This applies on Responses API too, though slightly better than chat completions per the same thread.

Engine performance (not a constraint here)

Modern constrained-decoding engines (XGrammar, llguidance) run at 40-50 μs/token with near-zero overhead. Performance is not a reason to avoid grammar tools today.

Updated recommendation

The findings reinforce — and partly sharpen — the original recommendation:

1. Still no retrofit on read/fs_search/shell/patch. The 17.1 pp accuracy regression on already-structured tasks is a real risk, and OpenAI's own use cases do not include this shape.
2. First grammar candidate should be a bounded-enum DSL, not a free-form regex. code_search_dsl with a closed vocabulary (def:, type:, file:) is more likely to actually conform to its grammar than a free-form text expression.
3. Build in output validation when we do ship a grammar tool. Treat the CFG as an advisory shaping signal, not a hard guarantee.
4. The strongest empirical case for a first prototype is sql_query against a known schema — directly matches OpenAI's own example, has clear value, has a natural parser (the SQL engine itself).

References

• OpenAI Cookbook: GPT-5 New Params and Tools — official docs with SQL/timestamp examples
• OpenAI Function Calling Guide
• JSONSchemaBench paper — empirical comparison
• Community report on Lark CFG conformance
• pydantic-ai issue #2513: free-form function calling
• llguidance / XGrammar — reference engines

[justrach]

Local eval — gpt-5.5 timestamp parsing (May 2026)

Ran a tiny direct-API eval to get a real signal. Methodology and data below.

Setup

• Model: gpt-5.5 (released 2026-04-23) via the Responses API
• Task: timestamp parsing — natural-language → YYYY-MM-DD [HH:MM]. Mirrors OpenAI's own cookbook example for regex grammars.
• Variants:
  • JSON-schema function tool: record_timestamp({year, month, day, hour, minute}) with strict bounds (month: 1..=12, etc.)
  • Custom grammar tool: same name, regex grammar [0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])( ([01][0-9]|2[0-3]):[0-5][0-9])?
• Prompts (5 total): "August 12th 2025 at 3PM"; "New Year's Eve 1999, one minute before midnight"; Apollo 11 day; "March 15, 2024 at 10:30 AM"; V-E Day
• Forced tool calls via tool_choice: {type: allowed_tools, mode: required, ...}

Results

Metric	JSON-schema	Grammar (regex)
Correct	5/5	5/5
Conforms to expected shape	5/5	5/5
Mean output tokens	43.4	44.6
Per-prompt out tokens	35,35,35,35,77	63,63,17,21,59
Errors	0	0

Key takeaways

1. At this scale, the two formats are indistinguishable on accuracy and tokens. 5/5 vs 5/5; ~1 token mean difference (noise).
2. The "model goes out of distribution on grammars" failure mode from the GPT-5 community thread did not reproduce on gpt-5.5 — 5/5 conformance on a bounded regex. (Caveat: our regex was tightly bounded with explicit character classes and alternatives, exactly the workaround the thread recommended. The unbounded-regex failure mode hasn't been re-tested.)
3. The 17.1 pp constrained-decoding accuracy penalty from JSONSchemaBench is not refuted — that was measured on harder structured tasks; ours was trivially structured. Both hit ceiling.
4. Per-prompt token variance is high (17 → 77 within the same variant), suggesting the model uses internal reasoning even when the visible output is short. Sample size 5 is way too small to draw cost conclusions.

Bug found in our wire format hypothesis

First eval attempt failed with HTTP 400 on every grammar request. The OpenAI Responses API expects format.{type, syntax, definition} with syntax and definition as siblings of type: "grammar", not nested under a grammar sub-object. async-openai's serialization (CustomToolParamFormat::Grammar(CustomGrammarFormatParam { definition, syntax })) flattens correctly via serde's tagged-enum default, so our Rust path is fine — only my manual Python eval got it wrong. Worth a regression test if we ever construct the request shape by hand.

Updated recommendation (v2)

The eval is too small + too easy to change the recommendation. But it does provide one new datapoint:

• Conformance is real on gpt-5.5 with bounded grammars. The GPT-5 era failure mode does not auto-extrapolate.
• Token cost parity confirmed at ceiling — no compactness win to capture either.

Net: still don't retrofit read/fs_search/etc. If we want to actually find the regime where grammar tools beat JSON-schema, we need:

• A harder task where 5/5 isn't ceiling (e.g. ambiguous date strings, long input, irregular formats)
• Larger N (50+ per variant)
• A task where JSON-schema constraint is genuinely hard to encode but grammar is natural (the bounded-DSL case the RFC predicted)

Eval script: /tmp/grammar_eval.py and /tmp/grammar_only.py (kept off-repo since they had API key handling).

Calls budget

15 of 18 dev-key calls used. Key has been disclosed in chat — recommend rotation in the OpenAI dashboard regardless.

[justrach]

Use-case taxonomy: where custom-grammar tools actually fit

Distilled from the OpenAI cookbook, the constrained-decoding literature, and the eval above. The pattern that wins: the entire tool input is one expression in a language with strict syntax, where invalid syntax silently means a wrong call. Multi-field structured calls do not fit this pattern.

Strong fits — model emits the target language directly

Tool shape	Why grammar wins	Notes
SQL against a known schema	Bounded table/column names; SQL parser is the only post-validator needed	OpenAI's own headline cookbook example
Math / formula expressions	2 * (3 + 4) - sqrt(16) — grammar prevents missing parens, undefined functions	Trivial Lark grammar
Regex generation	Meta-regex syntax is finite; model can't emit (unclosed group	Useful for "build me a pattern that matches X" tools
Date/timestamp DSL	Fixed format (ISO-8601, RFC3339, custom log fmt)	The eval task above; 5/5 conformance on gpt-5.5
Cron expressions	Strict 5-field grammar, easy to enforce	Common silent-fail target

Likely fits — write a small DSL anyway

Tool shape	Example	Bound
Structured search DSL	author:foo after:2024-01-01 tag:bug	Bounded keys, semi-bounded values
Symbol references	Module::SubModule::function, pkg/path:Symbol@version	Linker-defined
Git refspecs	branch:main..feature@{2.days.ago}	Git itself parses
Strict globs	**/*.{ts,tsx}	Finite brace expansion
Citation / bibref IDs	[Author2024], arxiv:2501.10868, doi:10.x/y	Each ID format is tiny
Strict identifiers	ISBN, IATA, postal codes, license plates	One per tool

Niche but real

• Constrained code generation: "single Python expression no longer than 80 chars" — grammar prevents multi-statement drift.
• Shell DSL with whitelist: Lark grammar restricted to ls|cat|grep|head|tail + safe flags. Useful for sandboxed sub-tools.
• CSS/color values: #RRGGBB, rgb(0..255, 0..255, 0..255), fixed named-color set.
• Terminal escape sequences with strict alphabet.
• Meta-tools: a tool whose input is itself a grammar definition (PEG/Lark playground).

Where grammar specifically does NOT help

Tool shape	Why JSON wins
Multi-independent-field tools (read, write, edit, patch, fs_search, fetch, shell)	JSON schema validates per-field; flattening loses structure
Free-form natural-language input (followup, plan, chat-style)	No grammar to enforce → grammar mode degenerates to .*
Variable-length lists (todo_write with N items)	JSON arrays > any hand-written grammar
Tools that will gain fields over time	JSON schema extension = one line; grammar rewrite + parser update

What this means for codegraff's catalog

Today: zero default tools fit the strong/likely-fit profile. The catalog is built around multi-field structured operations — every entry is in the "JSON wins" column. The grammar-gating infrastructure (2083d6ea) is dormant by design.

Realistic future candidates, if/when added:

1. Structured search DSL (author:foo file:*.rs def:fn) — natural fit for grammar, plausibly useful for an agent.
2. SQL query tool against project metadata — if codegraff ever indexes the codebase as a queryable schema, this is the OpenAI-cookbook canonical example.
3. Regex-tester / pattern-builder tool — narrow but high-value when it shows up.

Anything outside this list is probably JSON-schema territory.

[justrach]

Closing as decided.

The RFC's own recommendation, from the issue body:

Don't retrofit read, fs_search, etc. The cost (dual input shapes, parser maintenance, brittle to extension) outweighs the marginal compactness/correctness win on a single provider tier.

Do prototype on a new, grammar-native tool if/when one shows up. That gives us:

• A real measurement of the compactness/correctness benefit on gpt-5.5+.
• A reference implementation of the parser pattern.
• Something we can A/B test against an equivalent JSON-schema tool.

If after one experiment grammars demonstrably outperform JSON for our workload, then revisit retrofitting — but with data, not vibes.

The gating infrastructure (a7751534 + 2083d6ea) is already in place; the answer to the open question is "no retrofit". When/if a grammar-native tool shows up (the issue suggests math_eval, sql_query, code_search_dsl, regex_tester, or shell_constrained as plausible candidates), open a fresh issue scoped to that specific tool with an A/B eval plan.

Closing to clear the queue \u2014 the work guidance lives in the body for anyone designing a future grammar tool.