💡 idea: Ollama support

[Twilek-de]

Provide direct Ollama support for a quick and easy self hosted setup.

[crspeller]

@Twilek-de Just noted Ollama adding OpenAI compatibility: https://ollama.ai/blog/openai-compatibility
Does that work for you?

[Twilek-de]

Oh good catch. Just fired up Ollama with their latest docker container. OpenAI compatible endpoint works fine when evoking it with curl. With Mattermost AI plugin nothing happens. There is no error message but the ollama container does not ramp up CPU time, as it does when it is running a query. Don´t know what is happing.

I have checked that Mattermost is really seeing the ollama container in the docker network, it does. My settings look like this.

Running curl http://ollama:11434/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama2","messages": [{"role": "system","content": "You are a helpful assistant."},{"role": "user","content": "Hello!"}]}' inside the mattermost container works just fine, so it is not connectivity.

Mattermost config.json looks like this:

"mattermost-ai": { "config": { "allowPrivateChannels": true, "allowedTeamIds": "chat", "enableLLMTrace": false, "enableUserRestrictions": false, "imageGeneratorBackend": "Ollama", "llmBackend": "Ollama", "onlyUsersOnTeam": "", "services": [ { "apiKey": "", "defaultModel": "llama2", "id": "j5dr1bnste", "name": "Ollama", "password": "", "serviceName": "openaicompatible", "url": "http://ollama:11434", "username": "" } ], "transcriptBackend": "Ollama" } },

[Twilek-de]

Hmm Ollama supports a subset of the OpenAI API, namely completion. There is no support for streaming yet and I seem to remember that the MM AI Plugin uses the streaming endpoint....

Update:
Scratch that, the Ollama docu says streaming is supported as well...

[crspeller]

@Twilek-de It works for me if I set the API URL to http://localhost:11434/v1 Maybe just add the /v1?
If that doesn't work for you please post the mattermost server logs and any logs from ollama.

[Twilek-de]

Yes! That was it. Works now for me too. Thanks!

[Twilek-de]

Hmm somehow the new version does time out. Is there any way to edit the timeout,,,
{
"caller": "app/plugin_api.go:1003",
"error": "timeout streaming",
"level": "error",
"msg": "Streaming result to post failed",
"plugin_id": "mattermost-ai",
"timestamp": "2024-03-12 22:22:33.939 Z"
}

[Twilek-de]

Found the "streamingTimeoutSeconds" setting. Is working again now.