feat: smarter ollama load balancing

[tjbck]

right now it just uses random.choices, refer to my comment here:

open-webui/backend/apps/ollama/main.py

Line 39 in 8ed5759

# TODO: Implement a more intelligent load balancing mechanism for distributing requests among multiple backend instances.

[asedmammad]

@tjbck I can work on this one. I am considering implementing a configuration parameter that allows users to select from various strategies.

[tjbck]

@asedmammad Feel free to create a draft PR! I'll also get actively involved in this one!

[lewismacnow]

@tjbck I can work on this one. I am considering implementing a configuration parameter that allows users to select from various strategies.

I'd be very thankful for an ability to restrict models and/or Users to a specific connection (i.e. prevent/stop load balancing in open-webui in some situations).
I drew some examples of these situations in #1527

My reason for this is:

The loading/unloading process adds seconds to the response so if multiple users are activiely using multiple models which are shared bewteen connections, we will potentially add time to responses (whilst the model is removed/added to memory) instead of benefiting from load balancing over multiple connections.

[longregen]

I'd like to bump this since now Ollama has support for multiple loaded models at the same time: ollama/ollama#3418

In my testing with the "rc5" of 0.33, it worked just fine without changes on open-webui