[RFC]: Performance Roadmap

[simon-mo]

Anything you want to discuss about vllm.

This is a meta RFC tracking some of the performance enhancement works we are prioritizing.

• [RFC]: Isolate OpenAI Server Into Separate Process #6797
• [RFC]: Single Program Multiple Data (SPMD) Worker Control Plane #6556
• [RFC]: A Graph Optimization System in vLLM using torch.compile #6378
• [RFC]: Multi-Step Scheduling #6854
• [RFC]: Asynchronous Output Processor #6913

[richardliaw]

Let's pin this issue!

[zhisbug]

@SolitaryThinker please pin the multi-step inference issue

[w013nad]

If we're talking performance, can someone explain how sglang is 50-100% faster than vLLM while using vLLM as a backend? Their code base looks like a pure Python vLLM wrapper. How are they that much faster?

[simon-mo]

To be fully transparent, we are still figuring it out. @KuntaiDu is reproducing benchmark here: #6794 (comment)

[youkaichao]

@w013nad there are some niche details, for example, HF duplicates KV heads from 8 to 16, but haven't update the config. We are waiting for HF to update the config and the weight from 16 to 8.

However, sglang directly patches it:

https://github.com/sgl-project/sglang/blob/8628ab9c8bdf9b01c4671e3c6caabf49afd73395/python/sglang/srt/managers/controller/model_runner.py#L126

which means they use less memory for kv cache, and will have better throughput.

[zhyncs]

If we're talking performance, can someone explain how sglang is 50-100% faster than vLLM while using vLLM as a backend? Their code base looks like a pure Python vLLM wrapper. How are they that much faster?

Hi @w013nad I am a member of the SGLang Team. We are delighted that you have taken an interest in our project, SGLang https://github.com/sgl-project/sglang. We recently published a blog post introducing the performance advantages of v0.2 https://lmsys.org/blog/2024-07-25-sglang-llama3/. The performance data from the blog shows that on the Llama 3 model, from 8B to 70B to 405B, and from BF16 to FP8, on both A100 and H100, SGLang is far ahead in terms of performance due to its extensive engineering optimizations such as efficient scheduling, CUDA Graphs, Torch Compile etc. In fact, currently SGLang does not use vLLM as a backend, everything from Attention to scheduling and various optimizations are implemented by SGLang itself. At present we only utilize some linear classes from vLLM. Hope this answers your question. We greatly appreciate and thank vLLM for the excellent work, looking forward to continuing outstanding work on LLM inference acceleration, cheers.

[Ying1123]

@w013nad It is incorrect to say that "SGLang is a pure Python vLLM wrapper." Otherwise, it cannot explain the speedup you mentioned.

1. SGLang has its own efficient memory management and batch scheduler. Essentially, it only imports some fused kernels and layers from vLLM. SGLang uses vLLM as a kernel library, similar to how it uses FlashInfer. All other upper-level code in vLLM is irrelevant to SGLang.
2. What @youkaichao mentioned is an optimization we implemented for 405B model. However, it is not used in any of the benchmark results we have released. We always make fair comparisons. In our blog post, we use dummy weights with 8 heads for all frameworks to ensure fairness.

SGLang has a clean and well-designed architecture. It is also a fully open community with many experienced inference optimization developers. I encourage everyone to give it a try and run the benchmark. Both SGLang and vLLM are great projects and they can learn from each other.

[w013nad]

If we're talking performance, can someone explain how sglang is 50-100% faster than vLLM while using vLLM as a backend? Their code base looks like a pure Python vLLM wrapper. How are they that much faster?

Hi @w013nad I am a member of the SGLang Team. We are delighted that you have taken an interest in our project, SGLang https://github.com/sgl-project/sglang. We recently published a blog post introducing the performance advantages of v0.2 https://lmsys.org/blog/2024-07-25-sglang-llama3/. The performance data from the blog shows that on the Llama 3 model, from 8B to 70B to 405B, and from BF16 to FP8, on both A100 and H100, SGLang is far ahead in terms of performance due to its extensive engineering optimizations such as efficient scheduling, CUDA Graphs, Torch Compile etc. In fact, currently SGLang does not use vLLM as a backend, everything from Attention to scheduling and various optimizations are implemented by SGLang itself. At present we only utilize some linear classes from vLLM. Hope this answers your question. We greatly appreciate and thank vLLM for the excellent work, looking forward to continuing outstanding work on LLM inference acceleration, cheers.

Admittedly I didn't look too much through your code base. I just saw the github code overview that showed 99% Python and that you were importing a lot of vLLM components in your model executor. I was very impressed by your project though. I did a few of my own tests on my A100s and found that your codebase was much faster across the board

https://www.reddit.com/r/LocalLLaMA/comments/1ebztye/the_milestone_release_of_sglang_runtime_v02/leynwa2/

[Ying1123]

@w013nad Great to hear that you got good numbers from SGLang.

1. For llama-8b model, you can add --enable-torch-compile, which will give you an even better speedup.
2. speculative decoding is on our roadmap
3. tokenization endpoint should be easy to add. Your contribution is also welcome!
4. outlines support is there and also much faster than vllm. This is an example on how you can use it with OpenAI-compatible API.

As you said, SGLang is 99% Python. It reflects our philosophy of modular and minimal design.

[youkaichao]

What @youkaichao mentioned is an optimization we implemented for 405B model. However, it is not used in any of the benchmark results we have released. We always make fair comparisons. In our blog post, we use dummy weights with 8 heads for all frameworks to ensure fairness.

Thanks for the clarification! Really hope HF can fix the kv heads issue so that the community can benefit from it without using hacks.

[SolitaryThinker]

@simon-mo please update with rfc for multistep #6854