Concurrency and multiple calls

[enriquesouza]

Hi, I would like to know if running Ollama and making multiple calls is possible. I would love to add a server and use it for my users.

Therefore, when testing it, I saw it is waiting until a process finishes when I use the liteLLM proxy.

Is it possible?

[austin-starks]

Plus one. I love Ollama but I'm failing to see how I could deploy it as a cloud server.

[easp]

Right now you'd need to start multiple ollama servers on different ports and put them behind a reverse proxy.

I don't have any inside knowledge, but I'd expect this to change since Llama.cpp, which Ollama uses, has added support for batched requests, which is much more efficient than load balancing among separate instances.

[enriquesouza]

Thanks. I used it with docker and docker swarm, and then it solved with 10 instances running, therefore it is very limited yet. I hope a new version can fix that limitation.

[pdevine]

Going to close this as a dupe of #358