[Roadmap] vLLM Roadmap Q4 2024

Themes.

As before, we categorized our roadmap into 6 broad themes: broad model support, wide hardware coverage, state of the art performance optimization, production level engine, strong OSS community, and extensible architectures. As we are seeing more

Broad Model Support

Enhance LLM Support
- Hybrid/Interleaved Attention ([Feature]: Alternating local-global attention layers #9464)
Enhance Multi-Modality in vLLM ([RFC]: Multi-modality Support on vLLM #4194)
Enhance Support for State Space Models (Mamba)
Reward Model API ([RFC]: Reward Modelling in OpenAI Compatible Server #8967)
Arbitrary HF model (a collaboration with Hugging Face!)
- [Model]: Add transformers backend support #11330
Whisper

Help wanted:

Expand coverage for encoder-decoder models (Bert, XLMRoberta, BGE, T5)
- Support BERTModel (first encoder-only embedding model) #9056
- Support Cross encoder models #10400
API for streaming input (in particular for audio)

Hardware Support

A feature matrix for all the hardware that vLLM supports, and their maturity level
Expanding features support on various hardwares
- Fast PagedAttention and Chunked Prefill on Inferentia
- Upstream of Intel Gaudi
- Enhancements in TPU Support
- Upstream enhancements in AMD MI300x
- Performance enhancement and measurement for NVIDIA H200
- New accelerator support: IBM Spyre

Help wanted:

Design for pluggable, out-of-tree hardware backend similar to PyTorch’s PrivateUse API
Prototype JAX support

Performance Optimizations

Turn on chunked prefill, prefix caching, speculative decoding by default
Optimizations for structured outputs
Fused GEMM/all-reduce leveraging Flux and AsyncTP
Enhancement and overhead-removal in offline LLM use cases.
Better kernels (FA3, FlashInfer, FlexAttention, Triton)
Native integration with torch.compile

Help wanted:

A fast ngrams speculator
Sparse KV cache framework ([RFC]: Support sparse KV cache framework #5751)
Long context optimizations: context parallelism, etc.

Production Features

KV cache offload to CPU and disk
Disaggregated Prefill
More control in prefix caching, and scheduler policies
Automated speculative decoding policy, see Dynamic Speculative Decoding

Help wanted

Support multiple models in the same server

OSS Community

Enhancements in performance benchmark: more realistic workload, more hardware backends (H200s)
Better developer documentations for getting started with contribution and research

Help wanted

Documentation enhancements in general (styling, UI, explainers, tutorials, examples, etc)

Extensible Architecture

Full support for torch.compile
vLLM Engine V2: Asynchronous Scheduling and Prefix Caching Centric Design (vLLM's V1 Engine Architecture #8779)
A generic memory manager supporting multi-modality, sparsity, and others

If any of the items you wanted is not on the roadmap, your suggestion and contribution is still welcomed! Please feel free to comment in this thread, open feature request, or create an RFC.

Historical Roadmap: #5805, #3861, #2681, #244

Support for KV cache compression

upstream https://github.com/IsaacRe/vllm-kvcompress/tree/main - related issues (3532, 5751)

Do we have plans to support #5540? We are having a production level use case and would really appreciate if someone can look into it for Q4 onwards.

Hi, do we have any follow-up issue or Slack channel for the "KV cache offload to CPU and disk" task? Our team has previously explored some "KV cache offload" work based on vLLM, and we’d be happy to join any relevant discussion or contribute to the development if there's such chance~

Personally, also looking forward to know more about "More control in prefix caching, and scheduler policies" part😊.

@simon-mo

@simon-mo hi，regarding the topic “KV cache offload to CPU and disk”, I previously implemented a version that stores kv cache in a local file(#8018). Of course, I also did relevant abstractions and can add other media. Is there a slack channel for this? We can discuss the specific scheme. I am also quite interested in this function.

@zeroorhero

@sylviayangyy @zeroorhero thank you for your interests! Yes. @KuntaiDu has created a #feat-kvcache-offloading to discuss that.

Do we have plans to support #5540? We are having a production level use case and would really appreciate if someone can look into it for Q4 onwards.

It looks like LoRA is now supported. Are you encountering any issues?

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.

In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

Whether there is an opportunity to participate in changes related to speculative decoding? I'm working on some of the practices that are going to help you

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.

In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

I second this. We are using vLLM to host our production inference servers and all of our downstream applications rely on guided json decoding to ensure that output is parsable. There is a significant performance difference between guided and non-guided decoding and any performance improvements would be helpful to increase throughput.

@JC1DA

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.
In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

I second this. We are using vLLM to host our production inference servers and all of our downstream applications rely on guided json decoding to ensure that output is parsable. There is a significant performance difference between guided and non-guided decoding and any performance improvements would be helpful to increase throughput.

Hey, I maintain the guidance project and we worked on the first proposal in #6273 . Looks like vLLM has changed significantly since then, but if there is appetite for upgraded/more performant guided decoding work from the maintainers, we're happy to take another look and investigate a new PR. In particular, guidance (and our high performance rust implementation in llguidance already does async computations on CPU, calculates fast forward tokens, etc. and is typically accelerative for JSON schema.

@JC1DA @mmoskal

Do we have plans to support #5540? We are having a production level use case and would really appreciate if someone can look into it for Q4 onwards.

It looks like LoRA is now supported. Are you encountering any issues?

Yes, if we look at the class in mixtral_quant.py, it does not have SupportsLora which means lora is not supported for quantized Mixtral. but for mixtral.py, we have SupportsLora included in MixtralForCausalLM. I have a LORA adapter trained which I want to use on top of mixtral-awq model without merging, directly as a hot swap. Let me know if you know a better way to tackle this situation

Do we have plans to support #5540? We are having a production level use case and would really appreciate if someone can look into it for Q4 onwards.

It looks like LoRA is now supported. Are you encountering any issues?

Yes, if we look at the class in mixtral_quant.py, it does not have SupportsLora which means lora is not supported for quantized Mixtral. but for mixtral.py, we have SupportsLora included in MixtralForCausalLM. I have a LORA adapter trained which I want to use on top of mixtral-awq model without merging, directly as a hot swap. Let me know if you know a better way to tackle this situation

I'm guessing you explicitly set the quantization, right? If so, you can try removing that argument and test it out, like the following script:

llm = LLM(
    model="Mixtral-8x7B-Instruct-v0.1-GPTQ",
    trust_remote_code=True,
    gpu_memory_utilization=0.6,
    enable_lora=True,
)

@JC1DA

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.
In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

I second this. We are using vLLM to host our production inference servers and all of our downstream applications rely on guided json decoding to ensure that output is parsable. There is a significant performance difference between guided and non-guided decoding and any performance improvements would be helpful to increase throughput.

Hey, I maintain the guidance project and we worked on the first proposal in #6273 . Looks like vLLM has changed significantly since then, but if there is appetite for upgraded/more performant guided decoding work from the maintainers, we're happy to take another look and investigate a new PR. In particular, guidance (and our high performance rust implementation in llguidance already does async computations on CPU, calculates fast forward tokens, etc. and is typically accelerative for JSON schema.

@JC1DA @mmoskal

Improvements in guided generation performance would be very welcome. There is a helpful comment by @stas00 from last month with a nice summary of where things currently stand.

@JC1DA

Quick update -- we've made an initial PR to support guidance as a backend, which does improve performance over current implementations (#10217). Of course, better support for concurrency in general would also help guidance get significantly faster. Happy to support there and help if we can too!

@JC1DA

I am interested in optimizations related to speculative decoding. Is there an opportunity to get involved?

@wanghongyu2001

I have a somewhat similar question to @wanghongyu2001: if someone is interested in contributing to a specific aspect of vLLM, what’s the recommended path to get involved? Specifically, are there any suggested learning resources to systematically understand the vLLM codebase and, in particular, the v1 architecture?

In addition to navigating through the codebase, are there other structured ways to ramp up, such as design docs, or suggested youtube videos (in case I miss anything), any important PRs/files worth reading through? Would be thrilled to dive in and contribute to the project. Any guidance would be much appreciated!

@JC1DA

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.
In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

I second this. We are using vLLM to host our production inference servers and all of our downstream applications rely on guided json decoding to ensure that output is parsable. There is a significant performance difference between guided and non-guided decoding and any performance improvements would be helpful to increase throughput.

Hey, I maintain the guidance project and we worked on the first proposal in #6273 . Looks like vLLM has changed significantly since then, but if there is appetite for upgraded/more performant guided decoding work from the maintainers, we're happy to take another look and investigate a new PR. In particular, guidance (and our high performance rust implementation in llguidance already does async computations on CPU, calculates fast forward tokens, etc. and is typically accelerative for JSON schema.
@JC1DA @mmoskal

Improvements in guided generation performance would be very welcome. There is a helpful comment by @stas00 from last month with a nice summary of where things currently stand.

Do we have plans to improve concurrency performance for guided decoding? Enabling guided_json for concurrent requests results in significant throughput and latency degradation. (#3567)

Enhancements in concurrency performance for guided decoding would greatly benefit high-volume, real-time applications.

We could definitely use a thread pool to process logits list in parallel. As VLLM can run different number of logits processors for each logits in a batch, batched logits processor seems complex to implement.
However, using thread-pool also requires some mandatory changes from the guided decoding libraries:

it must be thread-safe. From what I experimented so far, lm-format-enforcer seems to be not thread-safe and failed in some tests if running with a thread pool
Pytorch in-place operations removal, again these ops failed if using in thread pool
efficient implementation to release GIL immediately after called

Also, I think VLLM is capable of providing multiple output tokens per sequence per step, we can leverage it for fast-forwarded tokens in JSON guided generation (super beneficial to improve performance)

Interested in thoughts/plan on EXL2 support: #3203

@JC1DA

Any plans on improving guided decoding? There's a long standing RFC for it (#5423) and previous attempts have been made (e.g. #6273). Unfortunately seems to have been forgotten since.
In particular I'd love to see it become async (logit mask or biases can be calculated while GPU is working on calculating logits) and fast forwarding tokens when the next few tokens are deterministic.

I second this. We are using vLLM to host our production inference servers and all of our downstream applications rely on guided json decoding to ensure that output is parsable. There is a significant performance difference between guided and non-guided decoding and any performance improvements would be helpful to increase throughput.

Hey, I maintain the guidance project and we worked on the first proposal in #6273 . Looks like vLLM has changed significantly since then, but if there is appetite for upgraded/more performant guided decoding work from the maintainers, we're happy to take another look and investigate a new PR. In particular, guidance (and our high performance rust implementation in llguidance already does async computations on CPU, calculates fast forward tokens, etc. and is typically accelerative for JSON schema.
@JC1DA @mmoskal

Improvements in guided generation performance would be very welcome. There is a helpful comment by @stas00 from last month with a nice summary of where things currently stand.

Do we have plans to improve concurrency performance for guided decoding? Enabling guided_json for concurrent requests results in significant throughput and latency degradation. (#3567)

Enhancements in concurrency performance for guided decoding would greatly benefit high-volume, real-time applications.

Integrating xgrammar could be a good choice: https://github.com/mlc-ai/xgrammar .

[ ] Better kernels (FA3, FlashInfer, FlexAttention, Triton)

What kernel is VLLM using as of right now? Asking in consideration of #10780

Hello,

I noticed that you have already merged the PR regarding this bug【function_name: Union[str, None] = current_tool_call.get("name")】. Could you please inform me which version of the latest supported image resolves this issue? Additionally, could you share the timeline for releasing the new version image? Currently, I am using version vllm/vllm-openai:v0.6.3.post1.

Thank you.

Fused GEMM/all-reduce leveraging Flux and AsyncTP
Looking forward to this optimization and hoping to use it as soon as possible. Has it been implemented yet？

Good news

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!

Navigation Menu

Repository navigation

[Roadmap] vLLM Roadmap Q4 2024 #9006

Themes.

Broad Model Support

Hardware Support

Performance Optimizations

Production Features

OSS Community

Extensible Architecture

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Roadmap] vLLM Roadmap Q4 2024 #9006

Description

Themes.

Broad Model Support

Hardware Support

Performance Optimizations

Production Features

OSS Community

Extensible Architecture

Activity

IsaacRe commented on Oct 2, 2024

ksjadeja commented on Oct 4, 2024

yangsijia-celina commented on Oct 12, 2024

zeroorhero commented on Oct 12, 2024

simon-mo commented on Oct 14, 2024

jeejeelee commented on Oct 16, 2024

iiLaurens commented on Oct 19, 2024

Yunhai-Hu commented on Oct 19, 2024

devdev999 commented on Oct 22, 2024

Harsha-Nori commented on Oct 22, 2024

ksjadeja commented on Oct 29, 2024

jeejeelee commented on Oct 29, 2024

dbuades commented on Oct 29, 2024

5 remaining items

Harsha-Nori commented on Nov 15, 2024

wanghongyu2001 commented on Nov 24, 2024

Toubat commented on Nov 24, 2024

JC1DA commented on Nov 26, 2024

gpgn commented on Nov 27, 2024

dongxiaolong commented on Nov 29, 2024

jannikstdl commented on Nov 29, 2024

yumc2573 commented on Dec 18, 2024

double-vin commented on Mar 27, 2025

double-vin commented on Mar 31, 2025

github-actions commented on Jun 30, 2025

github-actions commented on Jul 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions