Add constrained decoding for JSON schema response formats

[pminervini]

Tentative fix for #210.

This adds a small decoder for OpenAI-compatible response_format / Responses text.format JSON modes: during generation, tokens that would cause the current output to stop being a valid prefix of the requested JSON object/schema are masked out.

This is roughly the same idea as in llama.cpp’s JSON-schema grammar path, where the schema guides the decoding process. Right now it supports /v1/chat/completions and /v1/responses, handles json_object and json_schema, disables thinking for structured JSON outputs (not sure about this, will check how e.g. vllm handles this for LRMs), and keeps the schema out of the prompt ("follow this JSON schema" also seems to work for simple schemas -- example in the issue -- but I would not trust that too much).

Will share a script to stress-test this with schemas of varying complexity.

[pminervini]

Q: why not e.g. https://github.com/mlc-ai/xgrammar or https://github.com/guidance-ai/llguidance ? @fry69

See convo in #210 -- happy to ditch this and e.g. add -DENABLE_LLGUIDANCE (llama.cpp also does that)

[nhwaani]

I tested this PR on an M5 Max 128GB and it works well for the main structured-output cases.

Environment:

• MacBook Pro M5 Max, 128GB (Mac17,7)
• macOS 26.5
• DeepSeek V4 Flash q2 GGUF
• Server: ./ds4-server --port 8010 --ctx 4096 --tokens 512 --power 70

What passed:

• ./ds4_test --server
• make test
• chat response_format: {"type":"json_object"}
• chat response_format: {"type":"json_schema"}
• Responses API text.format: {"type":"json_schema"}
• enum / const / integer / boolean schema
• nested array-of-objects schema
• simple anyOf

One small edge case:

• For a schema with minItems: 2 and maxItems: 2, the model sometimes generated only whitespace until finish_reason=length.
• Cause: leading whitespace is still a valid JSON prefix, so whitespace tokens stayed allowed forever.

I opened a small follow-up PR against this branch:

pminervini#1

The fix only masks pure whitespace tokens while the output buffer is still only whitespace. Once JSON has started, whitespace is allowed normally.

[pminervini]

hey! hope you guys won't mind, I think this may be a better way of enabling support for structured outputs in ds4: pminervini#1 (comment) -- will try to look into this over the weekend

[pminervini]

Let me close this, will try to propose a better one.

[fry69]

FYI: I found some time to toy around with structured outputs and checked this PR out (apparently gh has no problems with closed PRs), merged it with current main (fec25da) and it passes my small, naive test suite with flying colors 👍

Looking forward for another stab at this.