Skip to content

Conversation

@marek-hradil
Copy link
Contributor

Grammar constructors have now pretty large number of exceptions on which they panic. Some of them are even not documented in the doc string of the functions, maybe because even llama.cpp does not mention them.

For example, when the grammar_str does not include grammar_root, upon calling grammar_lazy it just throws

Caused by:
  process didn't exit successfully: `...` (signal: 6, SIGABRT: process abort signal)

which is terrible for figuring out what is wrong (ive been there).

This PR fixes this, adding GrammarError and sanitizing the inputs to grammar and grammar_lazy functions.
The change is however breaking as we need a way to propagate the errors - the output type thus moved from Option<LlamaSampler> to Result<LlamaSampler, GrammarError>.

@MarcusDunn
Copy link
Contributor

the grammar code was created just as llama.cpp started it and has been untouched since. Thanks for taking the time to improve it!

@MarcusDunn MarcusDunn merged commit 288811e into utilityai:main Nov 28, 2025
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants