Ollama provider

[rnbwdsh]

Pulls available models from /tags endpoint.

Calls /generate endpoint for generating response.

API doc: https://editor.swagger.io/?url=https://raw.githubusercontent.com/marscod/ollama/main/api/ollama_api_specification.json

TODO: tests
TODO: maybe you can use the openapi lib client? Nicer request structure?

[rnbwdsh]

I just realized both ollama and openrouter support the /chat syntax with List[Dict[Literal["system"|"user"|"assistant"]: str]], so that you can condition the LLM to always start with

package plain

....

So I'd rework the connector to use the /chat instead of /generate (probably try to use the same client as openrouter) and I'd split the prompt into prompts:
{"system": "you are a test engineer, and you should write tests for the code the user provides.",
"user": challenge_message,
"assistant": `

package plain

import {
  TODO
}


func`}

This would also require small modification for the runner.

[chadbrewbaker]

Can we make this a CLI provider too?

Change this so it takes a constructor that has the command line string to shell out to. Should support ollama cli, llm cli ... that saves all the network headache.

eval-dev-quality/provider/ollama/ollama.go

Line 39 in e106327

func (p *ollamaQueryProvider) Query(ctx context.Context, modelIdentifier string, promptText string) (response string, err error) {

For testing I would just mostly mock, but you could roll the Shakespeare mini llm for integration tests. https://github.com/ggerganov/llama.cpp/blob/master/examples/train-text-from-scratch/README.md

[zimmski]

@rnbwdsh thanks for the PR! We have now merged all the bits and pieces, and quite some changes were necessary to get this working, tested and integrated. We did not do the automatic installation for Windows/MacOS. So if you want to add that, that would be great! Can guide you through. We are adding documentation soonish, and then a tutorial + video in the coming days.

[zimmski]

I just realized both ollama and openrouter support the /chat syntax with List[Dict[Literal["system"|"user"|"assistant"]: str]], so that you can condition the LLM to always start with

package plain

....

So I'd rework the connector to use the /chat instead of /generate (probably try to use the same client as openrouter) and I'd split the prompt into prompts: {"system": "you are a test engineer, and you should write tests for the code the user provides.", "user": challenge_message, "assistant": `

package plain

import {
  TODO
}


func`}

This would also require small modification for the runner.

Adding to task list

[zimmski]

Can we make this a CLI provider too?

Change this so it takes a constructor that has the command line string to shell out to. Should support ollama cli, llm cli ... that saves all the network headache.

eval-dev-quality/provider/ollama/ollama.go

Line 39 in e106327

func (p *ollamaQueryProvider) Query(ctx context.Context, modelIdentifier string, promptText string) (response string, err error) {

For testing I would just mostly mock, but you could roll the Shakespeare mini llm for integration tests. https://github.com/ggerganov/llama.cpp/blob/master/examples/train-text-from-scratch/README.md

Adding this to our task list as well. Unfortunately did not find a good way of doing this yet. Might be that the more functionality we add, the harder this gets 😿