How does set Grammar works in AutoGGUFModel?

[Modaffinity]

Hello,

In AutoGGUFModel, setGrammar asks for a bool; however, setGrammar(True) gives me a value type error, and it seems that setGrammar needs a string instead. Does this mean I can pass my BNF-like grammar directly to setGrammar? It is the same for setSystemPrompt().

[DevinTDHa]

Hi @Modaffinity
You are right: You can specify a BNF-like grammar with setGrammar as a string, which is then passed to the model backend.

Seems like there are type annotation errors with the setters (this does not affect functionality). This will be fixed in the future release. Thanks for asking!