[FEATURE] provide the option to use a self hosted AI (e.g. ollama)

[speedyconzales]

I am happy to see support for AI integration. But I do would prefer an option for self hosted AI

[pbek]

Yes, that would be the next logical step. 😉 As long they are OpenAI API compatible...

[pbek]

Which self-hosted AI did you have in mind?

[speedyconzales]

e.g. ollama https://github.com/ollama/ollama/blob/main/docs/api.md
https://ollama.com/blog/openai-compatibility

[pbek]

Ok, thank you. I'm currently struggling to get this into the script engine. I may need to rewrite some parts of the AI service handling...

[pbek]

I had to rewrite and rearrange a lot of stuff to make this happen. 😅

24.6.2

• A new scripting hook openAiBackendsHook was added, that is called when the OpenAI
  service config is reloaded (for #3037)
  • For example, this is also done when the script engine is reloaded
  • You can use it to provide config for custom OpenAI backends, like your own OpenAI API compatible LLMs
  • Please take a look at the
    openAiBackendsHook documentation
    for more information

[pbek]

There now is a new release, could you please test it and report if it works for you?

[pbek]

I've already found some small issues (like you need to set an apiKey, but there is none in ollama and you still need to reload the script engine manually after adding the script so the OpenAI UI reloads). But the main issue with ollama seems to be that the API result doesn't seem to be compatible with the API result from OpenAI. I'm currently looking into what's different. I didn't have this issue with Groq...

[pbek]

Hm, I only got an empty result back from ollama over the API. 🤔 What is your experience?

[pbek]

I added a script ollama-integration to the script repository for you to test.

[pbek]

The requests are the same with the OpenAI API and Ollama, but what Ollama returns with https://github.com/ollama/ollama?tab=readme-ov-file#chat-with-a-model is not at all what is returned at https://platform.openai.com/docs/api-reference/chat/create!

[eljamm]

Hey, I'd like to thank you for the amazing program and the continuous improvements.

I was curious to try this new feature with llama-cpp since I don't use ollama and it seems to be working:

What do you call a pineapple that lives under the sea? <ctrl+space>
What do you call a pineapple that lives under the sea? An under-pine in an a-peeling disguise! 

With an API base URL: http://localhost:3030/v1/chat/completions

Is there anything specific you'd like me to test?

[eljamm]

It seems that in Ollama, you also need to append the /v1/chat/completions endpoint to the hostname, according to the OpenAI compatibility · Ollama Blog.

[pbek]

Nice! Thank you very much for the hint. I'll try that then!

[pbek]

Hey, I'd like to thank you for the amazing program and the continuous improvements.

Thank you! 😉 This feature took longer than anticipated (and will take some more tweaking)!

[pbek]

It seems that in Ollama, you also need to append the /v1/chat/completions endpoint to the hostname

Works perfectly, thank you very much! http://127.0.0.1:11434/v1/chat/completions works perfectly!

[pbek]

@eljamm, if you want to contribute a script for llama-cpp to https://github.com/qownnotes/scripts it would be great. 😉
Would be almost the same as https://github.com/qownnotes/scripts/tree/master/ollama-integration anyway 😊

[eljamm]

@pbek Glad I could help and I don't mind writing the script for llama-cpp, but in my opinion it would it be better if the script was written for any local AI server in general since they more than likely support the OpenAI endpoint and it would be redundant to re-write the script for each one every time.

What do you think?

[eljamm]

Wrote a PR in qownnotes/scripts#237

[pbek]

Please see qownnotes/scripts#237 (comment). One script only supporting one endpoint will not do it if you have multiple backends...

[pbek]

Plus, you need to find out the correct endpoints yourself...

[eljamm]

Well, as far as I know, all OpenAI-compatible backends support the /v1/chat/completions endpoint, so the only thing that will change will be the server address.

In this regard, I thought a general script would be better as it's like a template which users can use to make their own scripts.

This means that we only have to maintain one script and the users won't think only the ollama or llama-cpp backends are supported.

[eljamm]

Is something like qownnotes/scripts#238 what you had in mind?

[speedyconzales]

it is already working with ollama. I could test it now. Thank you for your effort including this.
From the user perspective I have the following expectations:

1. since you only need the endpoint to the ollama instance. I would expect I have to set my specific local URL on the same section (settings page -> AI) where the user sets the API keys for the other AI backends. Not a new script I have to include. That is also the case for other backends like llama-cpp. I would just expect a new input field in the AI section. If I set it then it will be available to choose from the "enable AI support" menu.
2. I don't want to remember the models I have available on my ollama instance and then have to set them as a variable in qownnotes. I would expect that after setting my endpoint it automatically gets/reads all available models. So that I can choose from them, as I would with the Groq or the OpenAI backend.

But like I said, it is working for me and again thank you for adding this. Those expectations are just improvements for the usability that I wanted to share. It did confuse me for a second why I had to add another script to add my own AI endpoint, although there is a dedicated AI settings page.

[pbek]

since you only need the endpoint to the ollama instance. I would expect I have to set my specific local URL on the same section (settings page -> AI) where the user sets the API keys for the other AI backends. Not a new script I have to include. That is also the case for other backends like llama-cpp. I would just expect a new input field in the AI section. If I set it then it will be available to choose from the "enable AI support" menu.

I didn't want to go through the hassle of implementing a UI for adding multiple backends. Doing that in scripts turned out to be a hassle too, but at least it's more flexible now. There can be scripts preconfigured for certain backends, so you don't need to research the endpoint URL.

I don't want to remember the models I have available on my ollama instance and then have to set them as a variable in qownnotes. I would expect that after setting my endpoint it automatically gets/reads all available models. So that I can choose from them, as I would with the Groq or the OpenAI backend.

In the 3rd party OpenAI UIs I worked with, you were supposed to configure them too, because maybe you don't want to pay for an expensive one.

Is there even an OpenAI API to get the models?

But since the custom ones have script now those network requests to fetch the models (even to non-OpenAI APIs) could be done in those scripts! 🥳🎉

[pbek]

24.6.3

• Improvements to the new scripting hook openAiBackendsHook were made
  (for #3037)
  • The OpenAI UI elements will now be reloaded properly, when the script-engine
    is reloaded after a script was installed
  • The apiKey attribute in the scripting hook is now optional, because local
    APIs don't need it
  • There now is a script ollama-integration in the script repository that
    allows integration with Ollama
  • There now is a script llama-cpp-integration in the script repository that
    allows integration with llama.cpp (for #238, thank you, @eljamm)
  • In the AI settings there now also is the script-repository mentioned

There now is a new release, could you please test it and report if it works for you?

[eljamm]

Tested again and everything is working as expected.

[pbek]

Thanks for testing!

[pbek]

@speedyconzales, you might like this: qownnotes/scripts@93735c6 (ollama-integration: fetch installed models automatically)