multiple users on the same llama.cpp server: is it possible?

[ChrisDelapierre]

I am using llama.cpp server and it is connected to open webui. Open webui allows multiple users. I wonder how llama.cpp handles the situation where multiple users make queries to the server at the same time via open webui? How does llama.cpp handles context of each user?

[mikhail-shevtsov-wiregate]

When configuring model in llama-server you can specify number of parallel requests:

-np,   --parallel N                     number of server slots (default: -1, -1 = auto)
                                        (env: LLAMA_ARG_N_PARALLEL)

You need to account that context size will be divided equally:

-c,    --ctx-size N                     size of the prompt context (default: 0, 0 = loaded from model)
                                        (env: LLAMA_ARG_CTX_SIZE)

So if you need 2 parallel requests to model with 256k context size you need to specify context 512k size

If all slots occupied - request will wait in queue and processed once new slot is available.