feat(core): caller-supplied output-level ITokenConstraint (whole-turn grammar, not just tool args)

[pekkah]

Summary

Extend the token-constraint system (introduced in #374) so a caller can supply an output-level ITokenConstraint that is active across the whole assistant turn — not only inside tool-call arguments. Today the constraints (JsonToolArgumentConstraint, GemmaToolArgumentConstraint, QwenCoderToolArgumentConstraint) are schema-driven and scoped to a tool call's arguments. Consumers want to enforce a grammar over the free-form output stream itself (e.g. balanced XML-ish tags), reusing the exact same masking machinery.

Motivation / concrete use case

Downstream project Ayu is adding multi-modal output tagging: the model marks regions of its reply as spoken-only vs display-only with <say>…</say> / <show>…</show>, so a voice+text assistant can speak a summary while showing detail. A streaming splitter routes the tags to channels best-effort, but on a local ~12B the tags occasionally leak or go unbalanced, and a malformed tag can send raw markup to TTS. A grammar constraint that guarantees the top-level turn matches:

output := ( text | "<say>" text "</say>" | "<show>" text "</show>" )*

would make malformed/unbalanced tags impossible at the token level — the reliability tier above prompt+parsing. This is a loose constraint: free text everywhere, only forcing that once a < begins a marker it completes to a valid tag and that opens/closes balance.

(Design doc in Ayu: docs/multimodal-output-tagging.md, §2.1 and §8.)

What already exists (thanks to #374)

• ITokenConstraint — Accept(token) advances grammar state; Filter(logits) masks forbidden tokens to -inf while IsConstraining; byte-identical / zero-overhead when off.
• ToolSchemaCompiler → CompiledObject/CompiledNode; GrammarVocabulary; Sampler applies the filter; ContinuousBatchingEngine per-slot constraint handling; gated by SharpInferenceServerOptions.ToolGrammar / SHARPI_TOOL_GRAMMAR=1.

So the masking mechanism, vocab view, and batching integration are all in place. The gap is purely how a constraint is selected and scoped: it is currently tied to tool-argument decoding, activated inside a tool call.

Proposed change

Allow a caller-supplied ITokenConstraint (or a factory Func<..., ITokenConstraint>) to be attached to a generation request independent of tool schemas — e.g. on SamplingParams / the Generate*Async entry points (and a corresponding server/CLI opt-in), active for the whole turn.

Design points to decide:

1. Composition. When an output-level constraint is active and a tool call opens mid-turn (tool-argument constraint also active), the effective mask should be the logical AND of both constraints' allowed sets. Define how constraints stack (a CompositeTokenConstraint, or a small stack the sampler ANDs).
2. Scoping / activation. The output-level constraint must be able to keep IsConstraining == false during ordinary free text (so the common path stays byte-identical) and only clamp when it needs to (mid-marker, at a required close). This is already expressible via IsConstraining; just needs the plumbing to install a non-tool constraint.
3. Interaction with existing tool-arg path. Ensure enabling an output constraint does not force SHARPI_TOOL_GRAMMAR, and vice-versa; they should be independently switchable.
4. Streaming / continuous batching. Per-slot, same as the tool-arg constraints (the mask applies on the true next-token distribution).

The output-tag grammar itself (a small FSM) would live in the consumer; SharpInference only needs to accept and run a caller-provided ITokenConstraint. (If a generic GBNF/FSM constraint builder is in scope, even better, but not required for this ask.)

Acceptance criteria

• A consumer can pass an ITokenConstraint on a generate request that is honored for the whole turn, not just tool arguments.
• With no such constraint supplied, decoding is byte-identical to today (zero overhead preserved).
• An output-level constraint composes correctly with the tool-argument constraint when a tool call opens mid-turn (masks AND'd).
• Works under continuous batching (per-slot) and streaming.
• A small sample constraint (e.g. balanced <say>/<show> tags) demonstrates the whole path, mirroring ContinuousBatchingConstraintTests.

Related

• feat(engine): schema/grammar-constrained decoding for tool-call arguments #374 (constraint machinery), Tool-arg grammar: Qwen3-Coder XML <function=…> argument constraint (#374/#376 follow-up) #383/feat(core): constrain array/object value SHAPE inside Qwen3-Coder XML parameters (#383 follow-up) #386 (tool-argument shape constraints) — sibling work on the argument side; this is the output-stream side.