🚀 The feature, motivation and pitch
vLLM already supports --max-model-len auto, which is very useful because vLLM can determine the maximum context length that fits the current memory budget after loading the model and profiling memory usage.
I would like to request combining this auto behavior with a user-defined upper bound.
Example:
vllm serve <model> \
--max-model-len auto \
--max-model-len-cap 131072
Semantics:
fitted_len = maximum context length vLLM can fit in the configured memory budget
effective_max_model_len = min(fitted_len, max_model_len_cap)
Motivation
auto is great when the goal is “use as much context as possible”. However, on shared or restricted hardware, this can be undesirable. If a model supports a very large context length, e.g. 1M tokens, auto may consume the available vLLM memory budget for KV cache, leaving less room for other local applications or parallel serving processes.
Simply reducing --gpu-memory-utilization or setting a fixed memory budget does not fully solve this. The useful context length depends on model weight size, architecture, KV cache dtype, parallelism, and other runtime details. A small model can consume the same memory budget as a larger model simply by fitting more context.
Pitch
Use the best context length that fits, but never exceed the deployment/product limit.
This would be useful for local serving, small-business internal deployments, workstations, and shared GPUs.
Since vLLM already has the auto-fit logic and updates the effective max model length internally, this may be a relatively small code change: apply an optional cap after the auto-fit value is computed, before finalizing the engine configuration.
Alternatives
Example:
Desired context is 128k, but it might not always fit completely and the model has a much larger max context limit.
Fixed --max-model-len
vllm serve --max-model-len 131072
This works only if the chosen value fits. If it does not fit, startup fails instead of gracefully falling back to a smaller fitting value.
Plain --max-model-len auto
This is robust, but it may select a context length larger than the deployment wants to expose, causing unnecessary KV-cache memory usage.
--gpu-memory-utilization or --kv-cache-memory-bytes
These help limit memory, but they do not express the actual intent: automatically fit the context length, while capping the served maximum context.
External gateway limit
A proxy can reject requests above 128k tokens, but vLLM may still reserve KV cache for a larger auto-selected context length. This gives the right API behavior, but not the desired memory behavior.
Additional context
No response
Before submitting a new issue...
🚀 The feature, motivation and pitch
vLLM already supports
--max-model-len auto, which is very useful because vLLM can determine the maximum context length that fits the current memory budget after loading the model and profiling memory usage.I would like to request combining this
autobehavior with a user-defined upper bound.Example:
Semantics:
fitted_len = maximum context length vLLM can fit in the configured memory budget
effective_max_model_len = min(fitted_len, max_model_len_cap)
Motivation
auto is great when the goal is “use as much context as possible”. However, on shared or restricted hardware, this can be undesirable. If a model supports a very large context length, e.g. 1M tokens, auto may consume the available vLLM memory budget for KV cache, leaving less room for other local applications or parallel serving processes.
Simply reducing --gpu-memory-utilization or setting a fixed memory budget does not fully solve this. The useful context length depends on model weight size, architecture, KV cache dtype, parallelism, and other runtime details. A small model can consume the same memory budget as a larger model simply by fitting more context.
Pitch
Use the best context length that fits, but never exceed the deployment/product limit.
This would be useful for local serving, small-business internal deployments, workstations, and shared GPUs.
Since vLLM already has the auto-fit logic and updates the effective max model length internally, this may be a relatively small code change: apply an optional cap after the auto-fit value is computed, before finalizing the engine configuration.
Alternatives
Example:
Desired context is 128k, but it might not always fit completely and the model has a much larger max context limit.
Fixed --max-model-len
vllm serve --max-model-len 131072
This works only if the chosen value fits. If it does not fit, startup fails instead of gracefully falling back to a smaller fitting value.
Plain --max-model-len auto
This is robust, but it may select a context length larger than the deployment wants to expose, causing unnecessary KV-cache memory usage.
--gpu-memory-utilization or --kv-cache-memory-bytes
These help limit memory, but they do not express the actual intent: automatically fit the context length, while capping the served maximum context.
External gateway limit
A proxy can reject requests above 128k tokens, but vLLM may still reserve KV cache for a larger auto-selected context length. This gives the right API behavior, but not the desired memory behavior.
Additional context
No response
Before submitting a new issue...