[Usage]: I don't know how to set the maximum number of simultaneous API requests to be processed when calling an API

[Tu1231]

Your current environment

When I execute the following script
docker run --runtime nvidia --gpus all
-v /home/model-tran/models/DeepSeek-R1-Distill-Qwen-32B/:/deepseek-r1-32b
-v ~/.cache/huggingface:/root/.cache/huggingface
--env "HF_HUB_OFFLINE=1"
-p 8000:8000
--ipc=host
-d
vllm/vllm-openai:latest
--gpu-memory-utilization 0.9
--tensor-parallel-size 4
--max-concurrency 20
--model /deepseek-r1-32b
appear
api_server.py: error: unrecognized arguments: --max-concurrency 20

How would you like to use vllm

I want to set the maximum concurrent number of requests for external API calls to VLLM models when there are too many requests

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[FanYaning]

use --max_num_seqs=20

[hmellor]

--max_num_seqs=20 determines the number of requests that the engine will process simultaneously. But it does not limit the number of concurrent requests that the API will accept.

There is no option for this. See #13395

[FanYaning]

--max_num_seqs=20 determines the number of requests that the engine will process simultaneously. But it does not limit the number of concurrent requests that the API will accept.

There is no option for this. See #13395

I found in parameter --limit-concurrency https://www.uvicorn.org/settings/

I suggest that vllm's uvicorn server add these parameters

[hmellor]

Good find, I don't think --limit-max-requests makes sense for online serving, but the other two options could be useful.

Although they don't appear to discriminate which requests. So, I think, if the concurrency limit is reached with chat requests, then other endpoints like /v1/models will also refuse to respond.

Would that be a problem for you @Tu1231 or @FanYaning?

[FanYaning]

I agree what you said.

it will be too complicate to discriminate which requests to refused. So, it is fine for me that refusing to respond all coming requests when the concurrency limit is reached.

[jk1333]

This is good point.
To limit the GPU cycle and not requests being queued, send something 503 or 429 is necessary.
But VLLM is pretty weak about brute force attack and suppose client holds the tcp connection then vllm become zombie that still response from ping but gpu cycles are 0 but all schedulers are running state.
Currently put some circuit breaker like load balancer can control concurrency because we can not control uvicorn through vllm interface