Using ClaraVerse with a local Ollama LLM

[loficat]

I encountered issues when attempting to configure ClaraVerse to use a local Ollama LLM model on a CachyOS Linux system. As I found no documentation on this, I thought I'd share what I did to get it to work. Please note that these settings may change as ClaraVerse evolves (and I may not be communicating them well).

Within ClaraVerse, use the Settings page to add a provider.

1. The "Requires API" field must be enabled and "ollama" needs be entered into the field.

This is despite the instructions within the Settings page in the Configure Provider section specifically stating "(disable for local providers like Ollama)". As an aside, having this section and the Admin > Add Provider section (which differs) is confusing. Please note that using the Admin > Add Provider section appears to fail, as it may process the API option differently (this is speculation, as I didn't troubleshoot/verify this).

2. The "OpenAI" style API Base URL of "http://172.17.0.1:11434/v1" (use your configuration should it differ) needs to be used (as opposed to the default Ollama API URL). Although using "localhost" worked when adding the provider, it failed when attempted to chat with the local model. However, I'm not sure if this a verified issue or secondary to an app issue (as the Chat window did not appear to refresh data).

3. Only LLMs that possess tool-calling capability may be used, otherwise an error occurs. ClaraVerse sends OpenAI-style tool-calling payloads (that include a "tools: []" field in the request body). I saw no option to change this configuration.

Hopefully, someone else finds this information useful. Perhaps this information can used for the documentation or the app can be changed to make the configuration easier. All the best.