Repository navigation
Replies: 1 comment
|
@aldehir Daniele, the combinatorial explosion you mentioned for min/max integers in CFGs is a hard constraint issue. Enforcing arbitrary key orders via 2^k rules forces the model into a local optimum, prioritizing syntactic validity over semantic coherence. When the model's natural bias conflicts with the grammar, the sampling distribution gets distorted, leading to lower quality outputs or dead ends. Removing the grammar sampler for non-strict tools allows backend sampling to leverage the model's inherent probabilistic reasoning, which handles out-of-order keys naturally and significantly faster. The proposed --tool-grammar options offer the necessary flexibility to balance reliability with performance without sacrificing generative capabilities. Shifting toward self-correction over rigid constraints is the right architectural move for scalable tool calling. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
llama.cpp currently always uses grammar-constrained decoding when tools are provided. This poses a few challenges:
common/json-schema-to-grammar.cpp.2^krules forkkeys.apply()is very slow when it has to mask the full vocab. Today that only happens when a sampled token is rejected, but backend sampling needs the mask before sampling, at every decode step.Given these challenges, I propose removing reliance on the grammar sampler unless requested by the client. This will require the following changes:
Improve parsers to handle known deviations by models and rely more on self-corrective behavior. Examples:
Add an option
--tool-grammar {off,loose,strict,auto}where:off: Never constrain, even when a tool definesstrict: true.loose: Constrain syntax and tool names, but only constrain tool arguments for tools withstrict: true.strict: Constrain syntax, tool names, and tool arguments regardless of thestrictsetting.auto: Only constrain if at least one tool containsstrict: true. All tools withstrict: falseor nostrictfield are treated asloose.Simplify grammars for
stricttools and structured outputs to closely resemble OpenAI's constraints.The goals of this proposal are:
I am posting this to share the direction I think llama.cpp's tool calling should take, and to get feedback while I work on it.
All reactions