Multi ollama server

[Espinaydelfin]

Question:

Is there a way to connect multiple instances of Ollama with load balancing to OpenWeb UI, considering the increasing number of concurrent users and the need to distribute the workload while maintaining a single web server?

Background:

Due to the growing number of concurrent users, our system is experiencing performance issues, and we require a solution that can balance the load across multiple servers. Our ideal scenario would be to implement load balancing on the Ollama server instances, which are currently overwhelmed, rather than introducing additional web servers.

Current Architecture:

Our current architecture consists of a single OpenWeb UI instance connected to a single Ollama server instance. As the number of concurrent users increases, the Ollama server becomes a bottleneck, affecting overall system performance.

Requirements:

We are looking for a solution that can:

Connect multiple Ollama instances: Allow us to connect multiple Ollama server instances to OpenWeb UI.
Load balancing: Implement load balancing across these Ollama instances to distribute the workload and improve performance.
Single web server: Maintain a single OpenWeb UI instance as the entry point for all users, if possible.
Scalability: Ensure that the solution is scalable and can handle increasing numbers of concurrent users.
Potential Solutions:

We are open to exploring different approaches, including:

Ollama clustering: Implementing Ollama clustering to distribute the workload across multiple instances.
Load balancing software: Using load balancing software (e.g., HAProxy, NGINX) to distribute traffic across multiple Ollama instances.

Has anyone implemented a similar solution in the past? What recommendations can you provide based on your experience? We're looking to set up an on-premises infrastructure, as cloud environments are not feasible for us due to high costs. Fortunately, we already have multiple local servers available for deployment. Any insights or best practices you can share would be greatly appreciated.

[bet0x]

@Espinaydelfin Not using Ollama would be a starter, Ollama is for small stuff, not production ready. It doesn't scale and it is a wrapper to llama.cpp. You should migrate from ollama.

[Espinaydelfin]

I actually already found a solution to this issue in a previous discussion, but thank you all for your input and comments anyway!

[rgaricano]

what solution?

I have marked this, by right now i don't use multi...

https://github.com/softmaxer/ollamanet
https://github.com/distantmagic/paddler
https://github.com/mostlygeek/llama-swap
https://github.com/presbrey/ollamafarm

(on ollama's github there are more info about)

[Espinaydelfin]

After researching in various threads, I was able to find a solution that meets my needs. I am implementing a system that generates multiple instances of Ollama, configuring one as a master and replicating all changes on that instance (updates and model configuration) to the remaining servers.
The replication is done taking into account the hardware capabilities of each machine. If a model cannot be loaded on the target machine due to VRAM limitations, it is not replicated, even if it is available on the master instance. To achieve this, I am using a JSON configuration file that indicates the VRAM requirements for each model and a script that returns the amount of VRAM on the server.
When Open-webui connects, I redirect it to a proxy located on the master instance. This proxy returns the master's configuration in a transparent way for Open-webui, making it appear as if it were a single instance of Ollama. However, internally, the system balances the load between nodes based on their availability and capabilities.
For now, I am testing this solution with 1 node and 1 master instance, and it seems to be working correctly. However, I still require significant testing and development to deploy a production-ready version.

[kchr]

@Espinaydelfin Sounds like a good solution, have you published this somewhere?