patternanchor

llama.cpp's JSON-Schema pattern to GBNF converter rejects every unanchored pattern, which JSON Schema defines as the normal case; the patterns it does accept (explicitly anchored) compile to exactly the right language.

JSON Schema draft 2020-12 defines pattern with unanchored ECMA-262 semantics: a string satisfies the constraint if the regex matches some substring (search), and anchors are not implied. So {"type":"string","pattern": "[0-9]{3}"} is a valid schema, satisfied by "a123b". GBNF grammars accept only whole-token derivations, so llama.cpp's _visit_pattern in common/json-schema-to-grammar.cpp has a real semantic gap to bridge. This repo checks, against the real emitter and the real acceptor, what it actually does.

Result

Two findings, linked.

1. Unanchored patterns are rejected outright. For a pattern that does not begin with ^ and end with $, json_schema_to_grammar throws std::invalid_argument("JSON schema conversion failed: Pattern must start with '^' and end with '$'") and emits no grammar. Every one of the eight unanchored patterns tested ([0-9]{3}, [0-9]+, ab, [ab]{2}, a|b, ., 0*1, (a|b){2}) throws; every one of the eight explicitly anchored counterparts compiles. So a spec-valid JSON Schema with an unanchored pattern cannot be turned into a grammar at all: a consumer that passes {"pattern":"[0-9]{3}"} gets an exception, not a constraint. This diverges from the JSON Schema spec, which defines pattern as unanchored and does not require ^/$. It also refutes the pre-registered prediction, which expected the converter to silently impose whole-string fullmatch (over- constraining) rather than to reject the schema; the actual behavior is the more consequential one, an outright conversion failure.

2. Anchored patterns are sound and complete. For an explicitly anchored pattern ^p$, the emitted grammar accepts exactly the strings that fully match p. Over 65,532 membership checks (twelve anchored patterns, each tested against every string up to length 6 over the alphabet {0,1,a,b}), the real GBNF acceptor agrees with std::regex_match on every string: zero mismatches. Since ^...$ anchors to the whole string, this is exactly the JSON Schema semantics for the anchored pattern, so on the inputs the converter accepts it is correct.

Consequence and honest framing

Requiring ^...$ may be an intentional design choice: for constrained generation you usually want the whole output to be the pattern, not merely to contain it, so anchoring to the whole token is natural. But JSON Schema does not require anchors, and a large fraction of real-world pattern values in the wild are written unanchored. Under llama.cpp's converter those valid schemas do not degrade to a looser or tighter grammar, they fail with an exception, so any tool that forwards user- or model-supplied JSON schemas (llama-cpp-python, Ollama, LM Studio, llama-server) will error on a spec-valid input rather than constrain it. Whether that is a bug or a documented limitation is a judgment for upstream; the mechanical facts, that unanchored patterns throw and anchored patterns are exactly correct, are what this repo certifies.

Why the audit is trustworthy

• Two real production components. The device under test is the real json_schema_to_grammar from libllama-common, called with an nlohmann::ordered_json schema at the exact ABI (v3.12.0) of the exported symbol; membership is decided by the real GBNF acceptor from libllama, the same code constrained decoding runs. Only the reference regex and the enumeration are mine. Build 9760, commit 6ee0f65793.
• Two genuinely different deciders. Membership is a grammar stack automaton; the reference is std::regex, a backtracking NFA. They cannot share a bug, and the verdict is a categorical accept/reject. Full-string membership requires an accepting stack at end of input, so a valid prefix cannot cause a spurious accept.
• Bidirectional positive control. An anchored ^[ab]{2}$ grammar is confirmed to accept "ab" and reject "abc" and the empty string; and flipping one terminal in an emitted grammar ([ab] to [ac]) is confirmed to make the differential report a mismatch, proving the comparator fires on a known-bad grammar and is not self-confirming.
• No confound leaks. The alphabet {0,1,a,b} needs no JSON escaping, so the JSON-encoded string is verbatim; the reference patterns are drawn from the feature set common to ECMA-262 and std::regex; a pattern that fails to compile in the reference is dropped, never scored.

Run

./run.sh

Needs the Homebrew llama.cpp at /opt/homebrew (both libllama and libllama-common). nlohmann json v3.12.0 is vendored as json.hpp. Fully CPU, deterministic; no model, no decode, no GPU; runs in well under a second.

Files

• patternanchor.cpp — links the real emitter and acceptor; the anchor-requirement characterization, the anchored-pattern membership certificate, and the bidirectional control.
• llama-grammar.h — grammar acceptor declarations from the pinned commit (to call the real symbols), unmodified.
• json.hpp — vendored nlohmann json v3.12.0 (MIT), to construct the schema at the emitter's ABI.
• PREREG.md — the pre-registration, including the prediction that was falsified.
• results/main.json — machine-checked output.

Scope

Fully CPU, deterministic, exact; no model, no decode, no GPU. The certificate covers string pattern schemas over the enumerated pattern set and string ball; format, minLength/maxLength, and non-ASCII patterns are out of scope. The audited property is exact set membership of the emitted grammar against the regex, decided by the real acceptor, plus the exact compile-or-throw behavior of the emitter.

License

MIT (the audit code and results). json.hpp is nlohmann json, MIT.