Microsoft/guidance-ai integration with Ollama

[JanMP]

https://github.com/guidance-ai/guidance seems to be a easy and efficient way to generate tightly controlled output (like e.g json). Is there a way to use it with models provided by ollama?

[antonpolishko]

I'm interested in guidance support as well.

Skimming through sources looks like guidance supports only OpenAI API endpoint out of the box.

Technically it's possible to wrap either Ollama's API to adhere to OpenAI APIs or create a guidance.llms.ollama. But somebody need to implement that.

Alternatively you can use langchain that already supports ollama server endpoint and has lanchain extraction example

[breadchris]

llama.cpp has support for a grammar during inference ggerganov/llama.cpp#1773. Guidance is essentially this, but perhaps implementing this would be in another issue.

[technovangelist]

A few weeks ago we added format: json via the api and the cli. This allows for specifying that the output must be well formed json and allows you to specify the schema to be used. It seems to cover all the aspects of the issue, so I will go ahead and close it now. If you think there is anything we left out, reopen and we can address. Thanks for being part of this great community.