Grammar Guided response from model.

[NeevJewalkar]

Llama.cpp already has an implementation of grammars. Adding this to the ollama API, would bring more flexibility. This would also help in making reliable agentic llms.

[grantCelley]

There is a few PRs over the past year.

PR #565
Last updated: December 19th, 2024
The only thing I can tell is it opens up grammar from llama.cpp to the API. This was simple implementation but stopped it seems it was updated in pr #1606. This has merge conflicts.

PR #830
Last updated: April 29th, 2024
It allowed a json schema as the grammars. The last comment just explained why this is a good idea. Before that the last activity on the Pull request. This has merge conflicts.

PR #1606
Last updated: April 30th, 2024
This is an updated version of pr #565. This got a lot of comments. This has merge conflicts.

PR #3618
Last updated: April 23rd, 2024
This was an modification of pr #1606. It works for a JSON schema. It seems it worked as of April 23rd.

PR #2404
Last update: April 30th, 2024
This used format argument to allow grammar. Instead JSON you put in the Grammar. This is has conflicts now.

PR #2754
Last update: April 9th, 2024
This would allow a cli output onto ollama. This has conflicts now.

PR #3303
Last update: April 11th, 2024
This would add function calling. It says that it is a work in progress but does not have any recent activity nor any comments on how well this is going.

Why am I adding this to here?
This seems to be wanted by a bunch of people. None of the pull requests were merged and with 1.33 I think it caused a bunch of these pull requests not to work. I am trying to use LLMs for decision making. I need them to output in a standard format that I set.

Why aren't these merged yet?
I really don't know. This is something I am trying to figure out. I am wondering if this is not how the maintainers want the API to be. Some guidance would be nice.

[richardanaya]

i am able to easily break Ollama's JSON support like this even with embedding JSON schema in prompts.

[NeevJewalkar]

it wld be helpful if ollama had support for custom grammar, since the json grammar file provided by llama.cpp isnt really that helpful

[richardanaya]

I created an extensive PR for supporting grammar and go-side server validation here

#4525