Add ollama model server example

[MichaelClifford]

This PR is intended as an example of how we could integrate ollama into our project. Open to any questions or discussions that arise form this PR 😄

• Added a new directory model_services where we can store information about the different model services available.
• Added model_services/ollama which includes an extremely simple Containerfile that calls the existing ollama/ollama:latest as well as a README.md describing how this model service can be used.
• Updated the chatbot-langchain/chabout_ui.py so that it works well with either the llamacpp or the ollama model service.

[MichaelClifford]

cc @slemeur

[slemeur]

Thanks for the ping @MichaelClifford !
Looks very promising!

[MichaelClifford]

One caveat worth mentioning with this approach is that is has the user download the models to their host machine using ollama's CLI tooling. Meaning they already have ollama running on their machine, so why use the containerized version instead?

I think this experience could be improved by baking the model pulling steps into the ai studio somewhere.

[rhatdan]

Needs a rebase.

[rhatdan]

LGTM

[ericcurtin]

One caveat worth mentioning with this approach is that is has the user download the models to their host machine using ollama's CLI tooling. Meaning they already have ollama running on their machine, so why use the containerized version instead?

If you reorder the steps, run "podman run" and then do a "ollama pull" inside the container, you can just use the ollama binary inside the container for everything. It's probably better to avoid having to update two separate binaries.

You don't need to install ollama directly on your host machine to use it (and in fact that can be kinda nice you can do things rootless, etc.). The only thing you need to install is Nvidia drivers/container toolkit/etc. if you are using Nvidia.

I do have some ideas in this space partially related to podman-ollama project on my GitHub account I'd like to share with you guys sometime.

Another thing to checkout is Universal Blue's integration, they start the ollama container as a quadlet.

I think there's some advantages to trying to do this in a daemonless way though.

I think this experience could be improved by baking the model pulling steps into the ai studio somewhere.

[ericcurtin]

I have some MRs open in Ollama upstream also that help install Ollama in a containerised way:

https://github.com/ollama/ollama/pulls/ericcurtin

there's a PR backlog in Ollama though.

[MichaelClifford]

If you reorder the steps, run "podman run" and then do a "ollama pull" inside the container, you can just use the ollama binary inside the container for everything. It's probably better to avoid having to update two separate binaries.

Fully agree. My comment was mainly an artifact of how we were (are) managing models. via volume mounts onto the host machine without write permissions. Meaning, there'd need to be a unique model management path to deal with ollama's registry. Mainly I think we can probably come up with a better way manage model files in general.

I do have some ideas in this space partially related to podman-ollama project on my GitHub account I'd like to share with you guys sometime.

Sounds good. Want to open an new issue to start that discussion?

[ericcurtin]

Linked discussion:

#349