Skip to content

[Bugfix] Fix TP inference for Flex attention backend #19657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 16, 2025
Merged

[Bugfix] Fix TP inference for Flex attention backend #19657

merged 4 commits into from
Jun 16, 2025

Conversation

Isotr0py
Copy link
Collaborator

@Isotr0py Isotr0py commented Jun 15, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

  • Flex Attention doesn't work with tensor parallel currently because num_gpu_blocks is not updated in cache_config properly:
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]   File "/kaggle/working/vllm/vllm/v1/worker/gpu_model_runner.py", line 1211, in execute_model
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]     spec_decode_metadata) = (self._prepare_inputs(scheduler_output))
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]   File "/kaggle/working/vllm/vllm/v1/worker/gpu_model_runner.py", line 706, in _prepare_inputs
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]     attn_metadata_i = (builder.build(
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]                        ^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]   File "/kaggle/working/vllm/vllm/v1/attention/backends/flex_attention.py", line 301, in build
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]     total_cache_tokens = (self.runner.cache_config.num_gpu_blocks *
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=2540) ERROR 06-15 05:58:54 [multiproc_executor.py:527] TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
  • This PR fixes this issue to make Flex Attention work with TP.

Test Plan

VLLM_ATTENTION_BACKEND=FLEX_ATTENTION python examples/offline_inference/basic/generate.py --model facebook/opt-125m -tp 2

Test Result

(VllmWorker rank=1 pid=5600) INFO 06-15 06:06:11 [gpu_model_runner.py:2083] Graph capturing finished in 26 secs, took 0.20 GiB
(VllmWorker rank=0 pid=5599) INFO 06-15 06:06:11 [gpu_model_runner.py:2083] Graph capturing finished in 26 secs, took 0.20 GiB
INFO 06-15 06:06:11 [core.py:173] init engine (profile, create kv cache, warmup model) took 48.53 seconds
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 776.08it/s]
Processed prompts: 100%|████████████████████████████████████████████████████████████████████| 4/4 [00:11<00:00,  2.92s/it, est. speed input: 2.23 toks/s, output: 5.49 toks/s]
--------------------------------------------------
Prompt: 'Hello, my name is'
Generated text: ' Joel, I am a BYU lecturer and I would love to talk about your class'
--------------------------------------------------
Prompt: 'The president of the United States is'
Generated text: " stuck painting the country like it's 70 years old again, with behind-the"
--------------------------------------------------
Prompt: 'The capital of France is'
Generated text: " directly behind Germany's winning bid for elections as French business and French defence leaders fired"
--------------------------------------------------
Prompt: 'The future of AI is'
Generated text: ' predicted according to the 2020 report by a economists\nAI stand to be in the'
--------------------------------------------------

(Optional) Documentation Update

no need

Isotr0py added 2 commits June 15, 2025 01:56
@Isotr0py
fix tp with flex attn
67199af 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
disable compile for tp
55d41b7 
Signed-off-by: Isotr0py <2037008807@qq.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 15, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @Isotr0py, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical bug that prevented the Flex Attention backend from functioning correctly when Tensor Parallelism is enabled. The core issue was the failure to initialize cache configuration parameters (num_gpu_blocks, num_cpu_blocks) on the worker processes in a distributed setup. The fix involves explicitly sending this configuration to the workers. Additionally, a temporary measure is included to handle a separate CUDA error encountered with the compiled block mask function under TP, falling back to a non-compiled version.

Highlights

  • Bug Fix: Addresses a TypeError in the Flex Attention backend when used with Tensor Parallelism (TP > 1). The error occurred because cache_config.num_gpu_blocks was None on the workers.
  • Cache Configuration Propagation: Ensures that the calculated num_gpu_blocks and num_cpu_blocks are correctly propagated to the worker processes via RPC calls during engine initialization.
  • Conditional Block Mask Creation: Introduces a temporary workaround in the Flex Attention backend to use the non-compiled create_block_mask function when TP > 1, due to a noted CUDA error with the compiled version (create_block_mask_compiled) in this configuration. A FIXME comment is added for future investigation.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@mergify mergify bot added v1 tpu Related to Google TPUs labels Jun 15, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 15, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses two key issues preventing FlexAttention from functioning correctly with tensor parallelism (TP).

  1. It resolves a TypeError caused by num_gpu_blocks being None in the cache_config on worker processes. This is fixed by introducing an RPC call (initialize_cache) from the EngineCore to all workers, ensuring their cache_config is properly updated with the number of GPU and CPU blocks.
  2. It includes a workaround for a CUDA error: an illegal memory access that occurs when using the compiled version of create_block_mask with TP > 1. The fix involves conditionally using the non-compiled create_block_mask function in such scenarios.

The changes are well-targeted, and the provided test results demonstrate successful operation with TP enabled. The codebase is updated in EngineCore, GPU/TPU workers, and the FlexAttention backend itself.

vllm/v1/attention/backends/flex_attention.py
Comment on lines +240 to +241
# FIXME: With TP>1, create_block_mask_compiled will raise
# CUDA error: an illegal memory access was encountered
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The FIXME comment clearly explains the issue with create_block_mask_compiled when the tensor parallel world size is greater than 1. To ensure this is addressed in the future, consider creating a GitHub issue to track this underlying CUDA error if one doesn't exist already. This would help in eventually enabling the compiled version universally.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full trace back of the illegal memory error:

Log 
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/vllm/v1/attention/backends/flex_attention.py", line 262, in __post_init__
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     self.block_mask = self.build_block_mask()
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]                       ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/vllm/v1/attention/backends/flex_attention.py", line 246, in build_block_mask
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return create_block_mask_fn(
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return fn(*args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/nn/attention/flex_attention.py", line 824, in create_block_mask
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     def create_block_mask(
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return fn(*args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return compiled_fn(full_args)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     all_outs = call_func_at_runtime_with_args(
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     out = normalize_as_list(f(args))
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]                             ^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     outs = compiled_fn(args)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return compiled_fn(runtime_args)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 460, in __call__
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return self.current_callable(inputs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1372, in run
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return compiled_fn(new_inputs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 387, in deferred_cudagraphify
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     fn, out = cudagraphify(model, inputs, new_static_input_idxs, *args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 448, in cudagraphify
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return manager.add_function(
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 2308, in add_function
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return fn, fn(inputs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]                ^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 1997, in run
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     out = self._run(new_inputs, function_id)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 2104, in _run
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     out = self.run_eager(new_inputs, function_id)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 2269, in run_eager
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return node.run(new_inputs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py", line 668, in run
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     out = self.wrapped_function.model(new_inputs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/root/.cache/vllm/torch_compile_cache/26b5568570/rank_0_0/inductor_cache/4o/c4osf7wcdszj5dy7kaxakhrrucni4ac5aiyysa63j3fmz37p6jxn.py", line 561, in call
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     triton_per_fused__to_copy_sum_7.run(buf18, buf22, 5718, triton_per_fused__to_copy_sum_7_r0_numel, stream=stream0)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 909, in run
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     self.autotune_to_one_config(*args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 763, in autotune_to_one_config
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     timings = self.benchmark_all_configs(*args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 738, in benchmark_all_configs
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     launcher: self.bench(launcher, *args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 616, in bench
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return benchmarker.benchmark_gpu(kernel_call, rep=40)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/benchmarking.py", line 39, in wrapper
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     return fn(self, *args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/benchmarking.py", line 243, in benchmark_gpu
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     _callable()
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 601, in kernel_call
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     launcher(
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "<string>", line 5, in launcher
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]   File "/kaggle/working/vllm/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 444, in __call__
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527]     self.launch(*args, **kwargs)
(VllmWorker rank=0 pid=13527) ERROR 06-15 06:22:40 [multiproc_executor.py:527] RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered

@drisspg Any idea about this error?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @zou3519 for torch.compile related issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey so I actually just noticed this too, this was not the cause until pretty recently, going to create an issue + tracking for this

@Isotr0py Isotr0py requested a review from houseroad June 15, 2025 06:29
houseroad
houseroad reviewed Jun 15, 2025
View reviewed changes
Copy link
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can add some unittest, that will be great.

vllm/v1/worker/gpu_worker.py
@@ -112,6 +112,11 @@ def wake_up(self, tags: Optional[list[str]] = None) -> None:
buffer.data.copy_(self._sleep_saved_buffers[name].data)
self._sleep_saved_buffers = {}

def initialize_cache(self, num_gpu_blocks: int,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sounds more like "setting_cache_size" instead of initialize_cache?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, cache_config's num_gpu_blocks and num_cpu_blocks are updated in initialize_cache for worker in v0, which is a base class method:

vllm/vllm/worker/worker_base.py

Lines 65 to 69 in 3d330c4

def initialize_cache(self, num_gpu_blocks: int,
num_cpu_blocks: int) -> None:
"""Initialize the KV cache with the given size in blocks.
"""
raise NotImplementedError

vllm/vllm/worker/worker.py

Lines 312 to 325 in 3d330c4

def initialize_cache(self, num_gpu_blocks: int,
num_cpu_blocks: int) -> None:
"""Allocate GPU and CPU KV cache with the specified number of blocks.
This also warms up the model, which may record CUDA graphs.
"""
raise_if_cache_size_invalid(
num_gpu_blocks, self.cache_config.block_size,
self.cache_config.is_attention_free,
self.model_config.max_model_len,
self.parallel_config.pipeline_parallel_size)
self.cache_config.num_gpu_blocks = num_gpu_blocks
self.cache_config.num_cpu_blocks = num_cpu_blocks

Although this method not used by v1 before this PR, I think using this method shared by v0 can keep the worker implementation consistent.

vllm/v1/attention/backends/flex_attention.py
Comment on lines +240 to +241
# FIXME: With TP>1, create_block_mask_compiled will raise
# CUDA error: an illegal memory access was encountered
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @zou3519 for torch.compile related issue.

vllm/v1/engine/core.py
@@ -84,6 +84,8 @@ def __init__(self,

vllm_config.cache_config.num_gpu_blocks = num_gpu_blocks
vllm_config.cache_config.num_cpu_blocks = num_cpu_blocks
self.collective_rpc("initialize_cache",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering why only TP + FlexAttention needs this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because FlexAttention needs num_gpu_blocks for calculation while other attention backends don't need it.

Not sure if this is intended, but in V1, only engine core's cache_config has updated num_gpu_blocks, and worker in different process (TP situation) won't have num_gpu_blocks updated without collective_rpc calling.

Therefore, in distributed inference, worker's num_gpu_blocks is still None, which caused the error in PR description.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Could you check if we need to add some condition to only call this function if tp > 1.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For single-process, we use UniProcExecutor instead of MultiprocExecutor:

vllm/vllm/v1/executor/abstract.py

Lines 44 to 48 in 91b2c17

elif distributed_executor_backend == "mp":
from vllm.v1.executor.multiproc_executor import MultiprocExecutor
executor_class = MultiprocExecutor
elif distributed_executor_backend == "uni":
executor_class = UniProcExecutor

Given that it also has collective_rpc impplemented properly, it's safe to call collective_rpc as well, especially we only update cache_config here, though it has been done in unified process with previous lines before:

vllm/vllm/executor/uniproc_executor.py

Lines 50 to 58 in 91b2c17

def collective_rpc(self,
method: Union[str, Callable],
timeout: Optional[float] = None,
args: Tuple = (),
kwargs: Optional[Dict] = None) -> List[Any]:
if kwargs is None:
kwargs = {}
answer = run_method(self.driver_worker, method, args, kwargs)
return [answer]

Have checked TP=1 can still work currently.

Isotr0py added 2 commits June 15, 2025 16:38
@Isotr0py
add engine core tp test
b4f110f 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
check cpu num blocks as well
30b4f92 
Signed-off-by: Isotr0py <2037008807@qq.com>
houseroad
houseroad approved these changes Jun 15, 2025
View reviewed changes
Copy link
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Left one comment. :-)

@Isotr0py Isotr0py enabled auto-merge (squash) June 16, 2025 03:56
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 16, 2025
@Isotr0py Isotr0py merged commit 1173804 into vllm-project:main Jun 16, 2025
78 checks passed
@Isotr0py Isotr0py deleted the flex-tp branch June 16, 2025 12:03
amogkam added a commit to character-tech/vllm that referenced this pull request Jun 16, 2025
@amogkam @youkaichao
@jeejeelee @hmellor @zucchini-nlp @houseroad @NickLucche @googs1025 @simon-mo @SorenDreano @SorenNumind @reidliu41 @varun-sundar-rabindranath @sarckk @mgoin @Yikun @heheda12345 @ekagra-ranjan @chaunceyjiang @robertgshaw2-redhat @PeaBrane @Jeffwan @russellb @markmc @DarkLight1337 @bigPYJ1151 @lgeiger @vadiklyutiy @calvin0327 @kaixih @WoosukKwon @princepride @lsy323 @eicherseiji @Isotr0py @noooop @Xu-Wenqing @IwakuraRein @jmswen @tlrmchlsmth @kebe7jun @njhill @yangw-dev @huydhn @gemini-code-assist @vllmellm @22quinn @gcalmettes @patrickvonplaten @dubcyfor3 @p12tic @dsikka @vegaluisjose @jerryzh168 @benchislett @yaochengji @xu-song @aarnphm @KingsleyZhang123 @zou3519 @cryptopic @yuguo68 @npanpaliya @Shafi-Hussain @Adolfo-Karim @Chenyaaang @Alexei-V-Ivanov-AMD @ElizaWszola @QiliangCui @AaruniAggarwal @drisspg @draftbk @pramenku @ProExpertProg @Akashcodes732 @jennyyyyzhen @KsuParkhamchuk @cr7258 @conroy-cheers @windsonsea @yinghai @kylesayrs @liusiqian-tal @pavanimajety @yeqcharlotte @gty111 @vanbasten23 @Potabk @Copilot @annapendleton @louie-tsai @YUNQIUGUO @gshtras @py-andy-c @2niuhe @0xjunhao @leopardracer @artetaout @Ximingwang-09 @runzhen @davidxia @bnellnm @rasmith @andyxning @b8zhong @strutive07 @yewentao256 @mobicham @SageMoore @kouroshHakha @zixi-qi @Zerohertz @BoyuanFeng @qscqesze @Concurrensee @sahelib25 @jiahanc @kzawora-intel @maobaolong @ilmarkov @quanliu1991 @f14-bertolotti @francesco-bertolotti @SzymonOzog @Navanit-git @shawntan
Pull Upstream main 6/16 (#7)
add726d 
* [doc] clarify windows support (vllm-project#19088)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [CI/Build] Remove V0 LoRA test (vllm-project#19066)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* Fix underscores in dict keys passed via CLI (vllm-project#19030)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Bugfix] disable processor cache  (vllm-project#19068)

Signed-off-by: raushan <raushan@huggingface.co>

* [Doc] Improve the Pull Request template with key components (vllm-project#19086)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Misc] Add missing `_Backend` enums (vllm-project#19081)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [Misc] fix: add miss best_of param validation (vllm-project#18555)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [Misc] Add SPDX-FileCopyrightText  (vllm-project#19100)

Signed-off-by: simon-mo <simon.mo@hey.com>

* [Doc] Readme standardization (vllm-project#18695)

Co-authored-by: Soren Dreano <soren@numind.ai>

* [doc] update docker version (vllm-project#19074)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Kernel] DeepEP dispatch-combine kernel integration (vllm-project#18434)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [V1] Support cross-layer KV sharing (vllm-project#18212)

Signed-off-by: Yong Hoon Shin <yhshin@meta.com>

* [Perf] Tune `scaled_fp8_quant` by increasing vectorization (vllm-project#18844)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix interaction between `Optional` and `Annotated` in CLI typing (vllm-project#19093)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>

* [v1] Re-init input batch for multiple kv cache groups (vllm-project#18654)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [V1][Spec Decode][Ngram] 1.35x gain -> 1.95x gain on InstructCoder with prompt fix (vllm-project#18971)

* [Bugfix] get_num_blocks_to_allocate with null_block (vllm-project#19031)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled (vllm-project#19075)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix][P/D] Fix Prefix Cache Bug (vllm-project#18411)

Signed-off-by: nicklucche <nlucches@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

* [Bugfix] Max concurrency estimation and check_enough_kv_cache_memory for models with sliding window layers (vllm-project#19029)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* feat: add data parallel rank to KVEventBatch (vllm-project#18925)

* [Misc] Fix path and python alias errors in disagg_prefill exmaples (vllm-project#18919)

* [Docs] Add developer doc about CI failures (vllm-project#18782)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [CPU] V1 support for the CPU backend (vllm-project#16441)

* [Core] Cast multimodal input in hf processor (vllm-project#18862)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* [KERNEL] Sampler. CUDA kernel for applying repetition penalty (vllm-project#18437)

* [Cleanup][v1]:remote guided-decoding-backend for example (vllm-project#19059)

Signed-off-by: calvin chen <120380290@qq.com>

* [NVIDIA] Add Cutlass MLA backend (vllm-project#17625)

* [Bugfix] Fix FA3 full cuda graph correctness (vllm-project#19106)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* Fix vllm-project#19130 (vllm-project#19132)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [TPU] Skip hanging tests (vllm-project#19115)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* Fix ValueError: Missing value for tag key(s): model_name,engine. (vllm-project#19113)

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

* [Misc] Add packages for benchmark as extra dependency (vllm-project#19089)

Signed-off-by: Isotr0py <2037008807@qq.com>

* Improve the output precision of embedding models (vllm-project#19092)

* [CI/Build][Bugfix] Ensure compatibility with transformers 4.52 (vllm-project#18678)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add DeepSeek-R1-0528 function call chat template (vllm-project#18874)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* Sm100 blockwise fp8 swap ab (vllm-project#18564)

* [Doc] Update V1 Guide for embedding models (vllm-project#19141)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Allow AsyncLLMEngine.generate to target a specific DP rank (vllm-project#19102)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Bugfix][EP+DP] Fix internode check (vllm-project#19112)

Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>

* [Perf] Tunings for SM100 FP8 CUTLASS kernel (vllm-project#18778)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [TPU] Update dynamo dump file name in compilation test (vllm-project#19108)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [Bugfix] fix v1 cpu worker fails on macOS (vllm-project#19121)

* [Kernel] Integrate batched/masked deepgemm kernel (vllm-project#19111)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Misc] refactor: simplify EngineCoreClient.make_async_mp_client in AsyncLLM (vllm-project#18817)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [P/D] Heterogeneous TP (vllm-project#18833)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [doc] small fix (vllm-project#19167)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix][Nixl] Fix full prefix cache hit bug (vllm-project#18632)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix port handling in make_zmq_path (vllm-project#19117)

* [Torch Nightly]add missing dependency (vllm-project#18770)

Signed-off-by: Yang Wang <elainewy@meta.com>

* Handle non-serializable objects when dumping benchmark results (vllm-project#19114)

* [BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (vllm-project#19171)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Bugfix]: Fix the incompatibility issue with stream when Thinking is disabled (vllm-project#19135)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Build] Annotate wheel and container path for release workflow (vllm-project#19162)

Signed-off-by: simon-mo <simon.mo@hey.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Misc] Remove unnecessary fallback to prefill-decode attention (vllm-project#19138)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

* [Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly (vllm-project#19105)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Frontend] improve vllm run-batch --help display (vllm-project#19187)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] properly catch PIL-related errors for vision models when incorrect data urls are provided (vllm-project#19202)

Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>

* [mistral_common] Add v11 tokenizer (vllm-project#19193)

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 (vllm-project#19205)

* [Hardware][NVIDIA] FP4 MoE kernel optimization (vllm-project#19110)

Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>

* [MISC][Bugfix] Use less CPU when message queue has been empty for some time (vllm-project#16226)

Signed-off-by: Povilas Kanapickas <povilas@radix.lt>

* [P/D][NixlConnector] Enable FlashInfer backend (vllm-project#19090)

* [Quantization] Skip Fp4 Test for `compressed-tensors` (vllm-project#19217)

* [V1] Use FlashInfer by default on Blackwell GPUs (vllm-project#19118)

* [Model] NemotronH support (vllm-project#18863)

Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>

* Fix AOPerModuleConfig name changes (vllm-project#18869)

Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>

* [Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (vllm-project#19033)

Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>

* [v1] Hybrid Memory Allocator (vllm-project#17996)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [TPU] update torch_xla pin (vllm-project#19231)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* Support allowed_token_ids in ChatCompletionRequest (vllm-project#19143)

Signed-off-by: Xu Song <xusong.vip@gmail.com>

* [Chore] update CODEOWNERS (vllm-project#19247)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [v1][P/D] Fix a edge case in kv cache schedule (vllm-project#19182)

Co-authored-by: jinghui <jinghui@fb.com>

* [TPU] fix kv cache dtype in model runner (vllm-project#19244)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* [Quantization] Bump compressed-tensors version; update NVFP4A16 test model (vllm-project#19224)

Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>

* [Docs] Improve V1 KVConnector interface documentation (vllm-project#19172)

Signed-off-by: Nick Hill <nhill@redhat.com>

* Fix CompilationConfig repr (vllm-project#19091)

Signed-off-by: rzou <zou3519@gmail.com>

* Unit Test for run_dp_sharded_vision_model (vllm-project#19103)

Signed-off-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Siqi Yan <siqi@meta.com>

* [Model] Optimize nemotron_h implementation (vllm-project#19249)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Raise when non-multi-instance DP clients target a DP rank (vllm-project#19227)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* improve logits bias (vllm-project#19041)

* Fixed ppc build when it runs on non-RHEL based linux distros (vllm-project#18422)

Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>

* [BugFix] Fix MultiConnector test after HMA changes (vllm-project#19291)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix][Core] Update cancellation logic in `generate()` to handle Generator exits (vllm-project#19225)

Co-authored-by: Adolfo Victoria <adovi@meta.com>

* [Core] Fix abrupt request abort (vllm-project#18485)

Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>

Co-authored-by: Nick Hill <nhill@redhat.com>

* [BugFix] Fix tpu_model_runner block_id concatenation (vllm-project#19228)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Misc][Tools][Benchmark] Fix and improve auto tune script (vllm-project#19163)

Signed-off-by: Chenyaaang <chenyangli@google.com>

* [Build][ROCm] Update Dockerfile.rocm (vllm-project#19296)

Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

* [Easy][Test] Simplify test_function_tool_use with multiple parametrizes (vllm-project#19269)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Integrate CUTLASS MoE kernel with PPLX (vllm-project#18762)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

* [TPU][Test] Add script to run benchmark on TPU for buildkite (vllm-project#19039)

Signed-off-by: Qiliang Cui <derrhein@gmail.com>

* [CI][PowerPC] Use a more appropriate way to select testcase in tests/models/language/pooling/test_embedding.py (vllm-project#19253)

Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>

* Add FlexAttention to V1 (vllm-project#16078)

Signed-off-by: drisspg <drisspguessous@gmail.com>

* [Misc] refactor context extension (vllm-project#19246)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CI/Build] Improve Llama GGUF test robustness (vllm-project#19287)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Nit][Benchmark]Fix example in benchmark_serving_structured_output.py (vllm-project#19311)

Signed-off-by: Lifan Shen <lifans@meta.com>

* [AMD] Update compatible packaging version (vllm-project#19309)

Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>

* [BugFix][V1] Fix memory profiling bug (vllm-project#18974)

Signed-off-by: luka <luka@neuralmagic.com>

* [Bugfix]: Fix TypeError: 'float' object cannot be interpreted as an integer (vllm-project#19283)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix] Re-enable use_cudagraph in vLLM v1 (vllm-project#19299)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [Misc] Change tests/compile to use VLLM_V1 by default (vllm-project#19302)

Signed-off-by: rzou <zou3519@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B (vllm-project#19315)

Signed-off-by: Xu Wenqing <xuwq1993@qq.com>

* [Hardware][POWER] Add IBM POWER11 Support to CPU Extension Detection (vllm-project#19082)

Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>

* [Quantization] Add compressed-tensors NVFP4 support (vllm-project#18312)

* [Multi Modal] Add an env var for message queue max chunk bytes  (vllm-project#19242)

Signed-off-by: yZhen <yZhen@fb.com>
Co-authored-by: yZhen <yZhen@fb.com>

* [Bugfix] model_max_length should consider max_model_len in tokenizer_config (vllm-project#19201)

* [Deprecation] Remove `inputs` arg fallback in Engine classes (vllm-project#18799)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Add documentation update reminder to PR template (vllm-project#19289)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Frontend] Remove unreachable code from llm.py (vllm-project#19288)

Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>

* [Misc] Cleanup compilation tests (vllm-project#19343)

Signed-off-by: rzou <zou3519@gmail.com>

* [doc] improve ci doc (vllm-project#19307)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Doc] Fix description in the Automatic Prefix Caching design doc (vllm-project#19333)

Signed-off-by: cr7258 <chengzw258@163.com>

* [CI/Build] Fix LoRA test (vllm-project#19350)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Fix] Allow kernel compilation for CUDA capability 8.7 (vllm-project#19328)

Signed-off-by: Conroy Cheers <conroy@corncheese.org>

* [CI] Introduce rules for llama auto-label (vllm-project#19323)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Docs] Fix a bullet list in usage/security.md (vllm-project#19358)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* [full_graph] Fix query_start_loc padding (vllm-project#19321)

Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>

* [v1] Add fp32 support to v1 engine through flex attn (vllm-project#19319)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (vllm-project#19298)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Bugfix][Core] Prevent token lengths exceeding `max_model_len` in V0 (vllm-project#19348)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Quantization] Bump compressed-tensors version (vllm-project#19295)

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* [Frontend] Make TIMEOUT_KEEP_ALIVE configurable through env var (vllm-project#18472)

Signed-off-by: liusiqian <liusiqian@tal.com>

* [TPU]Fix KV cache sharing tests (vllm-project#19371)

* [HOT-FIX] Add `kv_sharing_target_layer_name` argument to cutlass_mla backend (vllm-project#19374)

Signed-off-by: Pavani Majety <pmajety@nvidia.com>

* [Misc] Fix a config typo in disable_hybrid_kv_cache_manager configuration (vllm-project#19383)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (vllm-project#19312)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [Bugfix] Fix benchmark_moe.py (vllm-project#19016)

Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>

* Use xla flag to improve the quantized model performance (vllm-project#19303)

Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

* Fix docs/mkdocs/hooks/remove_announcement.py (vllm-project#19382)

* [Frontend] Add tqdm_leave_pbar to control progress bar visibility (vllm-project#19357)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Core] Use tuple for kv cache group block ids (vllm-project#19175)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix modelscope token passed in (vllm-project#19389)

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Batch multi modal input using pinned memory (vllm-project#19169)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add security warning to bug report template (vllm-project#19365)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [Misc] refactor neuron_multimodal and profiling (vllm-project#19397)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* Add clear documentation around the impact of debugging flag (vllm-project#19369)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (vllm-project#17930)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>

* Revert "[v1] Add fp32 support to v1 engine through flex attn" (vllm-project#19404)

* [BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword `use_irope` (vllm-project#19134)

Signed-off-by: Yunqiu Guo <guorachel@meta.com>

* [BugFix][CPU] Fix CPU CI by ignore collecting test_pixtral (vllm-project#19411)

Signed-off-by: jiang.li <jiang1.li@intel.com>

* Simplify ep kernels installation (vllm-project#19412)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [Misc] Slight improvement of the BNB  (vllm-project#19418)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Docs] Note that alternative structured output backends are supported (vllm-project#19426)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (vllm-project#19440)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [Model] use AutoWeightsLoader for commandr (vllm-project#19399)

Signed-off-by: py-andy-c <pychen1017@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B-FP8 (vllm-project#19401)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* [BugFix] Allow use_cudagraph to work with dynamic VLLM_USE_V1 (vllm-project#19390)

Signed-off-by: rzou <zou3519@gmail.com>

* [New Model]: Support Qwen3 Embedding & Reranker  (vllm-project#19260)

* [BugFix] Fix docker build cpu-dev image error (vllm-project#19394)

Signed-off-by: niu_he <carlton2tang@gmail.com>

* Fix test_max_model_len in tests/entrypoints/llm/test_generate.py (vllm-project#19451)

Signed-off-by: Lu Fang <lufang@fb.com>

* [CI] Disable failing GGUF model test (vllm-project#19454)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Misc] Remove unused `MultiModalHasher.hash_prompt_mm_data` (vllm-project#19422)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add fused MOE config for Qwen3 30B A3B on B200 (vllm-project#19455)

Signed-off-by: Junhao Li <junhao@ubicloud.com>

* Fix Typo in Documentation and Function Name (vllm-project#19442)

* [ROCm] Add rules to automatically label ROCm related PRs (vllm-project#19405)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Support deep_gemm for linear methods (vllm-project#19085)

Signed-off-by: artetaout <lulala341@gmail.com>

* [Doc] Update V1 User Guide for Hardware and Models (vllm-project#19474)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Fix quantization link titles (vllm-project#19478)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Support "important" and "announcement" admonitions (vllm-project#19479)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Reduce warning message introduced in env_override (vllm-project#19476)

Signed-off-by: Lu Fang <lufang@fb.com>

* Support non-string values in JSON keys from CLI (vllm-project#19471)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add cache to cuda get_device_capability (vllm-project#19436)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix some typo (vllm-project#19475)

Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>

* Support no privileged mode on CPU for docker and kubernetes deployments (vllm-project#19241)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>

* [Bugfix] Update the example code, make it work with the latest lmcache (vllm-project#19453)

Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>

* [CI] Update FlashInfer to 0.2.6.post1 (vllm-project#19297)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [doc] fix "Other AI accelerators" getting started page (vllm-project#19457)

Signed-off-by: David Xia <david@davidxia.com>

* [Misc] Fix  misleading ROCm warning (vllm-project#19486)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Docs] Remove WIP features in V1 guide (vllm-project#19498)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Kernels] Add activation chunking logic to FusedMoEModularKernel (vllm-project#19168)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (vllm-project#17331)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [UX] Add Feedback During CUDAGraph Capture (vllm-project#19501)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

* [CI/Build] Fix torch nightly CI dependencies (vllm-project#19505)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [CI] change spell checker from codespell to typos (vllm-project#18711)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [BugFix] Force registration of w8a8_block_fp8_matmul_deepgemm via lazy import (vllm-project#19514)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* Add Triton Fused MoE kernel config for E=16 on B200 (vllm-project#19518)

Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>

* [Frontend] Improve error message in tool_choice validation (vllm-project#19239)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [BugFix] Work-around incremental detokenization edge case error (vllm-project#19449)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [BugFix] Handle missing sep_token for Qwen3-Reranker in Score API (vllm-project#19522)

Signed-off-by: strutive07 <strutive07@gmail.com>

* [AMD][Kernel][BugFix] fix test_rocm_compressed_tensors_w8a8 for rocm (vllm-project#19509)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* Fix typo (vllm-project#19525)

Signed-off-by: 2niuhe <carlton2tang@gmail.com>

* [Security] Prevent new imports of (cloud)pickle (vllm-project#18018)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Bugfix][V1] Allow manual FlashAttention for Blackwell (vllm-project#19492)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Bugfix] Respect num-gpu-blocks-override in v1 (vllm-project#19503)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Quantization] Improve AWQ logic (vllm-project#19431)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Doc] Add V1 column to supported models list (vllm-project#19523)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [V1][NixlConnector] Drop `num_blocks` check  (vllm-project#19532)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [Perf] Vectorize static / dynamic INT8 quant kernels (vllm-project#19233)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Fix TorchAOConfig skip layers (vllm-project#19265)

Signed-off-by: mobicham <hicham@mobiuslabs.com>

* [torch.compile][ROCm] Fuse quantization onto attention using a torch.compile pass (vllm-project#16756)

Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>

* [doc] Make top navigation sticky (vllm-project#19540)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Spec Decode][Benchmark] Generalize spec decode offline benchmark to more methods and datasets (vllm-project#18847)

* [Misc] Turn MOE_DP_CHUNK_SIZE into an env var (vllm-project#19506)

* [Bugfix] Enforce contiguous input for dynamic_per_token FP8/INT8 quant (vllm-project#19452)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Doc] Unify structured outputs examples (vllm-project#18196)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [V1] Resolve failed concurrent structured output requests (vllm-project#19565)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Revert "[Build/CI] Add tracing deps to vllm container image (vllm-project#15224)" (vllm-project#19378)

* [BugFix] : Fix Batched DeepGemm Experts (vllm-project#19515)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [Bugfix] Fix EAGLE vocab embedding for multimodal target model (vllm-project#19570)

Signed-off-by: qizixi <qizixi@meta.com>

* [Doc] uses absolute links for structured outputs (vllm-project#19582)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [doc] fix incorrect link (vllm-project#19586)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Correct broken docs link (vllm-project#19553)

Signed-off-by: Zerohertz <ohg3417@gmail.com>

* [CPU] Refine default config for the CPU backend (vllm-project#19539)

Signed-off-by: jiang1.li <jiang1.li@intel.com>

* [Fix] bump mistral common to support magistral (vllm-project#19533)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [Fix] The zip function in Python 3.9 does not have the strict argument (vllm-project#19549)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* use base version for version comparison (vllm-project#19587)

Signed-off-by: Boyuan Feng <boyuan@meta.com>

* [torch.compile] reorganize the cache directory to support compiling multiple models (vllm-project#19064)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [BugFix] Honor `enable_caching` in connector-delayed kvcache load case (vllm-project#19435)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Model] Fix minimax model cache & lm_head precision (vllm-project#19592)

Signed-off-by: qingjun <qingjun@minimaxi.com>

* [Refactor] Remove unused variables in `moe_permute_unpermute_kernel.inl` (vllm-project#19573)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [doc][mkdocs] fix the  duplicate Supported features sections in GPU docs (vllm-project#19606)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CUDA] Enable full cudagraph for FlashMLA (vllm-project#18581)

Signed-off-by: luka <luka@neuralmagic.com>

* [Doc] Add troubleshooting section to k8s deployment (vllm-project#19377)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* [torch.compile] Use custom ops when use_inductor=False (vllm-project#19618)

* Adding "AMD: Multi-step Tests" to amdproduction. (vllm-project#19508)

Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [BugFix] Fix DP Coordinator incorrect debug log message (vllm-project#19624)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [V1][Metrics] Deprecate metrics with gpu_ prefix for non GPU specific metrics. (vllm-project#18354)

Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>

* [Bugfix] Fix the speculative decoding test by setting the target dtype (vllm-project#19633)

* [Misc] Modularize CLI Argument Parsing in Benchmark Scripts (vllm-project#19593)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] Fix auto dtype casting for BatchFeature (vllm-project#19316)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization (vllm-project#19500)

* Only build CUTLASS MoE kernels on Hopper (vllm-project#19648)

* [Bugfix] Don't attempt to use triton if no driver is active (vllm-project#19561)

* [Fix] Convert kv_transfer_config from dict to KVTransferConfig (vllm-project#19262)

* [Perf] Further tunings for SM100 FP8 CUTLASS kernel (vllm-project#19566)

* [Bugfix][2/n] Fix speculative decoding CI - Fix test_ngram_e2e_greedy_correctness (vllm-project#19644)

* [Kernel] Raise verbose error and consolidate `num_heads/num_kv_heads` divisibility check (vllm-project#19339)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Benchmark] Refactor benchmark script for fp8 & int8 (vllm-project#19627)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Enable prefix caching with full cuda graphs (vllm-project#19617)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [CI/Build] Fix torch nightly CI dependencies part 2 (vllm-project#19589)

* [Misc] Remove duplicate multiproc method setting for CPU platform (vllm-project#19649)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] Remove unused variableds in C++ (vllm-project#19609)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Bugfix][Core] Prefix caching causes incorrect outputs due to outdated ComputedBlocksTracker (vllm-project#18957)

Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>

* [Misc][Frontend] passthrough `bad_words` (vllm-project#19564)

Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Misc] Fix skipped max-model-len validation when deriving max model length from tokenizer config (vllm-project#19660)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [TPU] support attention head dim smaller than 128 (vllm-project#19620)

Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [MISC] typo fix (vllm-project#19672)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [CI] Add mteb testing for rerank models (vllm-project#19344)

* [Docs] Move multiproc doc to v1 dir (vllm-project#19651)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Kernel] GGUF MMVQ kernel for multiple input vectors (vllm-project#18754)

Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>

* [BugFix] Don't catch BaseException when dumping execute_model errors (vllm-project#19626)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [DOC] Add reasoning capability to vLLM streamlit code (vllm-project#19557)

* [Feature]:Allow for Granite MoE Hybrid models with _only_ shared experts. (vllm-project#19652)

Signed-off-by: Shawn Tan <shawntan@ibm.com>

* [Bugfix] Fix TP inference for Flex attention backend (vllm-project#19657)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] bump huggingface_hub pkg to 0.33.0 (vllm-project#19547)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [Bugfix] fix missing 'finish_reason': null in streaming chat (vllm-project#19662)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Kernels] Use empty for modular MoE workspaces (vllm-project#19667)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [Model] Add support for MiniMaxM1ForCausalLM (shares architecture with MiniMaxText01ForCausalLM) (vllm-project#19677)

Signed-off-by: QscQ <qscqesze@gmail.com>

* [V1] Change return type on get_multimodal_embeddings() (vllm-project#19446)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: raushan <raushan@huggingface.co>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: googs1025 <googs1025@gmail.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Varun <vsundarr@redhat.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: calvin chen <120380290@qq.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Yang Wang <elainewy@meta.com>
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Signed-off-by: Povilas Kanapickas <povilas@radix.lt>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Signed-off-by: Xu Song <xusong.vip@gmail.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: Siqi Yan <siqi@meta.com>
Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Chenyaaang <chenyangli@google.com>
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Qiliang Cui <derrhein@gmail.com>
Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Lifan Shen <lifans@meta.com>
Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>
Signed-off-by: luka <luka@neuralmagic.com>
Signed-off-by: Richard Zou <zou3519@gmail.com>
Signed-off-by: Xu Wenqing <xuwq1993@qq.com>
Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Signed-off-by: yZhen <yZhen@fb.com>
Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>
Signed-off-by: cr7258 <chengzw258@163.com>
Signed-off-by: Conroy Cheers <conroy@corncheese.org>
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: liusiqian <liusiqian@tal.com>
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Anna Pendleton <pendleton@google.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: Yunqiu Guo <guorachel@meta.com>
Signed-off-by: jiang.li <jiang1.li@intel.com>
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: py-andy-c <pychen1017@gmail.com>
Signed-off-by: niu_he <carlton2tang@gmail.com>
Signed-off-by: Junhao Li <junhao@ubicloud.com>
Signed-off-by: artetaout <lulala341@gmail.com>
Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>
Signed-off-by: David Xia <david@davidxia.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Signed-off-by: strutive07 <strutive07@gmail.com>
Signed-off-by: 2niuhe <carlton2tang@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: mobicham <hicham@mobiuslabs.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: qizixi <qizixi@meta.com>
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: qingjun <qingjun@minimaxi.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>
Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>
Signed-off-by: Shawn Tan <shawntan@ibm.com>
Signed-off-by: QscQ <qscqesze@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: CYJiang <86391540+googs1025@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: SorenDreano <71752785+SorenDreano@users.noreply.github.com>
Co-authored-by: Soren Dreano <soren@numind.ai>
Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Jiaxin Shan <seedjeffwan@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Co-authored-by: Lukas Geiger <lukas.geiger94@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Calvin Chen <45745657+calvin0327@users.noreply.github.com>
Co-authored-by: Kaixi Hou <kaixih@nvidia.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>
Co-authored-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Xu Wenqing <121550081+Xu-Wenqing@users.noreply.github.com>
Co-authored-by: Lain <fusiyuan2000@hotmail.com>
Co-authored-by: jmswen <jmswen@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Kebe <mail@kebe7jun.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yang Wang <elainewy@meta.com>
Co-authored-by: Huy Do <huydhn@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Chiyue Wei <92623189+dubcyfor3@users.noreply.github.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Povilas Kanapickas <povilas@radix.lt>
Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>
Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Jerry Zhang <jerryzh168@gmail.com>
Co-authored-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: Xu Song <xusong.vip@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Jinghui Zhang <jinghuizhang0804@gmail.com>
Co-authored-by: jinghui <jinghui@fb.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
Co-authored-by: Siqi Yan <ysq0807@hotmail.com>
Co-authored-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Yu Guo <82124926+yuguo68@users.noreply.github.com>
Co-authored-by: Nishidha <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Co-authored-by: Adolfo Victoria <adolfokarim@gmail.com>
Co-authored-by: Adolfo Victoria <adovi@meta.com>
Co-authored-by: Chenyaaang <42742451+Chenyaaang@users.noreply.github.com>
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: ElizaWszola <ewszola@redhat.com>
Co-authored-by: QiliangCui <derrhein@gmail.com>
Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com>
Co-authored-by: Driss Guessous <32754868+drisspg@users.noreply.github.com>
Co-authored-by: Lifans <draftbks@gmail.com>
Co-authored-by: pramenku <7664080+pramenku@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Akash kaothalkar <61960177+Akashcodes732@users.noreply.github.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: jennyyyyzhen <47012288+jennyyyyzhen@users.noreply.github.com>
Co-authored-by: yZhen <yZhen@fb.com>
Co-authored-by: Kseniya Parkhamchuk <43078183+KsuParkhamchuk@users.noreply.github.com>
Co-authored-by: Se7en <chengzw258@163.com>
Co-authored-by: Conroy Cheers <conroy@corncheese.org>
Co-authored-by: Michael Yao <haifeng.yao@daocloud.io>
Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: liusiqian-tal <141730978+liusiqian-tal@users.noreply.github.com>
Co-authored-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: Ye (Charlotte) Qi <yeq@meta.com>
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Co-authored-by: XiongfeiWei <isaacwxf23@gmail.com>
Co-authored-by: Li Wang <wangli858794774@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Anna Pendleton <pendleton@google.com>
Co-authored-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>
Co-authored-by: Rachel Guo <35738743+YUNQIUGUO@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: py-andy-c <37168711+py-andy-c@users.noreply.github.com>
Co-authored-by: niu_he <carlton2tang@gmail.com>
Co-authored-by: Junhao Li <junhao@ubicloud.com>
Co-authored-by: leopardracer <136604165+leopardracer@users.noreply.github.com>
Co-authored-by: artetaout <128046886+artetaout@users.noreply.github.com>
Co-authored-by: Ximingwang-09 <72070413+Ximingwang-09@users.noreply.github.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: runzhen <wangrunzhen@gmail.com>
Co-authored-by: David Xia <david@davidxia.com>
Co-authored-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: rasmith <Randall.Smith@amd.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: wonjun Jang <strutive07@gmail.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mobicham <37179323+mobicham@users.noreply.github.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: kourosh hakhamaneshi <31483498+kouroshHakha@users.noreply.github.com>
Co-authored-by: qizixi <22851944+zixi-qi@users.noreply.github.com>
Co-authored-by: Hyogeun Oh (오효근) <ohg3417@gmail.com>
Co-authored-by: Boyuan Feng <fby.1994@gmail.com>
Co-authored-by: qscqesze <qingjun@minimaxi.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Saheli Bhattacharjee <47847054+sahelib25@users.noreply.github.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: maobaolong <baoloongmao@tencent.com>
Co-authored-by: Ilya Markov <markovilya197@gmail.com>
Co-authored-by: quanliu <33453350+quanliu1991@users.noreply.github.com>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: Francesco Bertolotti <f14.bertolotti@gmail.com>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Szymon Ożóg <58388001+SzymonOzog@users.noreply.github.com>
Co-authored-by: Navanit Dubey <98005188+Navanit-git@users.noreply.github.com>
Co-authored-by: Shawn Tan <shawntan@ibm.com>
Co-authored-by: qscqesze <qscqesze@gmail.com>
amogkam added a commit to character-tech/vllm that referenced this pull request Jun 16, 2025
@amogkam @zucchini-nlp
@houseroad @NickLucche @googs1025 @simon-mo @SorenDreano @SorenNumind @reidliu41 @varun-sundar-rabindranath @sarckk @mgoin @hmellor @Yikun @heheda12345 @ekagra-ranjan @chaunceyjiang @robertgshaw2-redhat @PeaBrane @Jeffwan @russellb @markmc @DarkLight1337 @bigPYJ1151 @lgeiger @vadiklyutiy @calvin0327 @kaixih @WoosukKwon @princepride @lsy323 @eicherseiji @Isotr0py @noooop @Xu-Wenqing @IwakuraRein @jmswen @tlrmchlsmth @kebe7jun @njhill @yangw-dev @huydhn @gemini-code-assist @vllmellm @22quinn @gcalmettes @patrickvonplaten @dubcyfor3 @p12tic @dsikka @vegaluisjose @jerryzh168 @benchislett @yaochengji @xu-song @aarnphm @KingsleyZhang123 @zou3519 @cryptopic @jeejeelee @yuguo68 @npanpaliya @Shafi-Hussain @Adolfo-Karim @Chenyaaang @Alexei-V-Ivanov-AMD @ElizaWszola @QiliangCui @AaruniAggarwal @drisspg @draftbk @pramenku @ProExpertProg @Akashcodes732 @jennyyyyzhen @KsuParkhamchuk @cr7258 @conroy-cheers @windsonsea @yinghai @kylesayrs @liusiqian-tal @pavanimajety @yeqcharlotte @gty111 @vanbasten23 @Potabk @Copilot @annapendleton @louie-tsai @YUNQIUGUO @youkaichao @gshtras @py-andy-c @2niuhe @0xjunhao @leopardracer @artetaout @Ximingwang-09 @runzhen @davidxia @bnellnm @rasmith @andyxning @b8zhong @strutive07 @yewentao256 @mobicham @SageMoore @kouroshHakha @zixi-qi @Zerohertz @BoyuanFeng @qscqesze @Concurrensee @sahelib25 @jiahanc @kzawora-intel @maobaolong @ilmarkov @quanliu1991 @f14-bertolotti @francesco-bertolotti @SzymonOzog @Navanit-git @shawntan
Fix interleaved_sliding_window config (#8)
3def9c4 
* [Bugfix] disable processor cache  (vllm-project#19068)

Signed-off-by: raushan <raushan@huggingface.co>

* [Doc] Improve the Pull Request template with key components (vllm-project#19086)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Misc] Add missing `_Backend` enums (vllm-project#19081)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [Misc] fix: add miss best_of param validation (vllm-project#18555)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [Misc] Add SPDX-FileCopyrightText  (vllm-project#19100)

Signed-off-by: simon-mo <simon.mo@hey.com>

* [Doc] Readme standardization (vllm-project#18695)

Co-authored-by: Soren Dreano <soren@numind.ai>

* [doc] update docker version (vllm-project#19074)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Kernel] DeepEP dispatch-combine kernel integration (vllm-project#18434)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [V1] Support cross-layer KV sharing (vllm-project#18212)

Signed-off-by: Yong Hoon Shin <yhshin@meta.com>

* [Perf] Tune `scaled_fp8_quant` by increasing vectorization (vllm-project#18844)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix interaction between `Optional` and `Annotated` in CLI typing (vllm-project#19093)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>

* [v1] Re-init input batch for multiple kv cache groups (vllm-project#18654)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [V1][Spec Decode][Ngram] 1.35x gain -> 1.95x gain on InstructCoder with prompt fix (vllm-project#18971)

* [Bugfix] get_num_blocks_to_allocate with null_block (vllm-project#19031)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled (vllm-project#19075)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix][P/D] Fix Prefix Cache Bug (vllm-project#18411)

Signed-off-by: nicklucche <nlucches@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

* [Bugfix] Max concurrency estimation and check_enough_kv_cache_memory for models with sliding window layers (vllm-project#19029)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* feat: add data parallel rank to KVEventBatch (vllm-project#18925)

* [Misc] Fix path and python alias errors in disagg_prefill exmaples (vllm-project#18919)

* [Docs] Add developer doc about CI failures (vllm-project#18782)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [CPU] V1 support for the CPU backend (vllm-project#16441)

* [Core] Cast multimodal input in hf processor (vllm-project#18862)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* [KERNEL] Sampler. CUDA kernel for applying repetition penalty (vllm-project#18437)

* [Cleanup][v1]:remote guided-decoding-backend for example (vllm-project#19059)

Signed-off-by: calvin chen <120380290@qq.com>

* [NVIDIA] Add Cutlass MLA backend (vllm-project#17625)

* [Bugfix] Fix FA3 full cuda graph correctness (vllm-project#19106)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* Fix vllm-project#19130 (vllm-project#19132)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [TPU] Skip hanging tests (vllm-project#19115)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* Fix ValueError: Missing value for tag key(s): model_name,engine. (vllm-project#19113)

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

* [Misc] Add packages for benchmark as extra dependency (vllm-project#19089)

Signed-off-by: Isotr0py <2037008807@qq.com>

* Improve the output precision of embedding models (vllm-project#19092)

* [CI/Build][Bugfix] Ensure compatibility with transformers 4.52 (vllm-project#18678)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add DeepSeek-R1-0528 function call chat template (vllm-project#18874)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* Sm100 blockwise fp8 swap ab (vllm-project#18564)

* [Doc] Update V1 Guide for embedding models (vllm-project#19141)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Allow AsyncLLMEngine.generate to target a specific DP rank (vllm-project#19102)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Bugfix][EP+DP] Fix internode check (vllm-project#19112)

Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>

* [Perf] Tunings for SM100 FP8 CUTLASS kernel (vllm-project#18778)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [TPU] Update dynamo dump file name in compilation test (vllm-project#19108)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [Bugfix] fix v1 cpu worker fails on macOS (vllm-project#19121)

* [Kernel] Integrate batched/masked deepgemm kernel (vllm-project#19111)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Misc] refactor: simplify EngineCoreClient.make_async_mp_client in AsyncLLM (vllm-project#18817)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [P/D] Heterogeneous TP (vllm-project#18833)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [doc] small fix (vllm-project#19167)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix][Nixl] Fix full prefix cache hit bug (vllm-project#18632)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix port handling in make_zmq_path (vllm-project#19117)

* [Torch Nightly]add missing dependency (vllm-project#18770)

Signed-off-by: Yang Wang <elainewy@meta.com>

* Handle non-serializable objects when dumping benchmark results (vllm-project#19114)

* [BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (vllm-project#19171)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Bugfix]: Fix the incompatibility issue with stream when Thinking is disabled (vllm-project#19135)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Build] Annotate wheel and container path for release workflow (vllm-project#19162)

Signed-off-by: simon-mo <simon.mo@hey.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Misc] Remove unnecessary fallback to prefill-decode attention (vllm-project#19138)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

* [Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly (vllm-project#19105)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Frontend] improve vllm run-batch --help display (vllm-project#19187)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] properly catch PIL-related errors for vision models when incorrect data urls are provided (vllm-project#19202)

Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>

* [mistral_common] Add v11 tokenizer (vllm-project#19193)

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 (vllm-project#19205)

* [Hardware][NVIDIA] FP4 MoE kernel optimization (vllm-project#19110)

Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>

* [MISC][Bugfix] Use less CPU when message queue has been empty for some time (vllm-project#16226)

Signed-off-by: Povilas Kanapickas <povilas@radix.lt>

* [P/D][NixlConnector] Enable FlashInfer backend (vllm-project#19090)

* [Quantization] Skip Fp4 Test for `compressed-tensors` (vllm-project#19217)

* [V1] Use FlashInfer by default on Blackwell GPUs (vllm-project#19118)

* [Model] NemotronH support (vllm-project#18863)

Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>

* Fix AOPerModuleConfig name changes (vllm-project#18869)

Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>

* [Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (vllm-project#19033)

Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>

* [v1] Hybrid Memory Allocator (vllm-project#17996)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [TPU] update torch_xla pin (vllm-project#19231)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* Support allowed_token_ids in ChatCompletionRequest (vllm-project#19143)

Signed-off-by: Xu Song <xusong.vip@gmail.com>

* [Chore] update CODEOWNERS (vllm-project#19247)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [v1][P/D] Fix a edge case in kv cache schedule (vllm-project#19182)

Co-authored-by: jinghui <jinghui@fb.com>

* [TPU] fix kv cache dtype in model runner (vllm-project#19244)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* [Quantization] Bump compressed-tensors version; update NVFP4A16 test model (vllm-project#19224)

Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>

* [Docs] Improve V1 KVConnector interface documentation (vllm-project#19172)

Signed-off-by: Nick Hill <nhill@redhat.com>

* Fix CompilationConfig repr (vllm-project#19091)

Signed-off-by: rzou <zou3519@gmail.com>

* Unit Test for run_dp_sharded_vision_model (vllm-project#19103)

Signed-off-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Siqi Yan <siqi@meta.com>

* [Model] Optimize nemotron_h implementation (vllm-project#19249)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Raise when non-multi-instance DP clients target a DP rank (vllm-project#19227)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* improve logits bias (vllm-project#19041)

* Fixed ppc build when it runs on non-RHEL based linux distros (vllm-project#18422)

Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>

* [BugFix] Fix MultiConnector test after HMA changes (vllm-project#19291)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix][Core] Update cancellation logic in `generate()` to handle Generator exits (vllm-project#19225)

Co-authored-by: Adolfo Victoria <adovi@meta.com>

* [Core] Fix abrupt request abort (vllm-project#18485)

Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>

Co-authored-by: Nick Hill <nhill@redhat.com>

* [BugFix] Fix tpu_model_runner block_id concatenation (vllm-project#19228)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Misc][Tools][Benchmark] Fix and improve auto tune script (vllm-project#19163)

Signed-off-by: Chenyaaang <chenyangli@google.com>

* [Build][ROCm] Update Dockerfile.rocm (vllm-project#19296)

Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

* [Easy][Test] Simplify test_function_tool_use with multiple parametrizes (vllm-project#19269)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Integrate CUTLASS MoE kernel with PPLX (vllm-project#18762)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

* [TPU][Test] Add script to run benchmark on TPU for buildkite (vllm-project#19039)

Signed-off-by: Qiliang Cui <derrhein@gmail.com>

* [CI][PowerPC] Use a more appropriate way to select testcase in tests/models/language/pooling/test_embedding.py (vllm-project#19253)

Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>

* Add FlexAttention to V1 (vllm-project#16078)

Signed-off-by: drisspg <drisspguessous@gmail.com>

* [Misc] refactor context extension (vllm-project#19246)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CI/Build] Improve Llama GGUF test robustness (vllm-project#19287)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Nit][Benchmark]Fix example in benchmark_serving_structured_output.py (vllm-project#19311)

Signed-off-by: Lifan Shen <lifans@meta.com>

* [AMD] Update compatible packaging version (vllm-project#19309)

Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>

* [BugFix][V1] Fix memory profiling bug (vllm-project#18974)

Signed-off-by: luka <luka@neuralmagic.com>

* [Bugfix]: Fix TypeError: 'float' object cannot be interpreted as an integer (vllm-project#19283)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix] Re-enable use_cudagraph in vLLM v1 (vllm-project#19299)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [Misc] Change tests/compile to use VLLM_V1 by default (vllm-project#19302)

Signed-off-by: rzou <zou3519@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B (vllm-project#19315)

Signed-off-by: Xu Wenqing <xuwq1993@qq.com>

* [Hardware][POWER] Add IBM POWER11 Support to CPU Extension Detection (vllm-project#19082)

Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>

* [Quantization] Add compressed-tensors NVFP4 support (vllm-project#18312)

* [Multi Modal] Add an env var for message queue max chunk bytes  (vllm-project#19242)

Signed-off-by: yZhen <yZhen@fb.com>
Co-authored-by: yZhen <yZhen@fb.com>

* [Bugfix] model_max_length should consider max_model_len in tokenizer_config (vllm-project#19201)

* [Deprecation] Remove `inputs` arg fallback in Engine classes (vllm-project#18799)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Add documentation update reminder to PR template (vllm-project#19289)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Frontend] Remove unreachable code from llm.py (vllm-project#19288)

Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>

* [Misc] Cleanup compilation tests (vllm-project#19343)

Signed-off-by: rzou <zou3519@gmail.com>

* [doc] improve ci doc (vllm-project#19307)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Doc] Fix description in the Automatic Prefix Caching design doc (vllm-project#19333)

Signed-off-by: cr7258 <chengzw258@163.com>

* [CI/Build] Fix LoRA test (vllm-project#19350)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Fix] Allow kernel compilation for CUDA capability 8.7 (vllm-project#19328)

Signed-off-by: Conroy Cheers <conroy@corncheese.org>

* [CI] Introduce rules for llama auto-label (vllm-project#19323)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Docs] Fix a bullet list in usage/security.md (vllm-project#19358)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* [full_graph] Fix query_start_loc padding (vllm-project#19321)

Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>

* [v1] Add fp32 support to v1 engine through flex attn (vllm-project#19319)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (vllm-project#19298)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Bugfix][Core] Prevent token lengths exceeding `max_model_len` in V0 (vllm-project#19348)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Quantization] Bump compressed-tensors version (vllm-project#19295)

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* [Frontend] Make TIMEOUT_KEEP_ALIVE configurable through env var (vllm-project#18472)

Signed-off-by: liusiqian <liusiqian@tal.com>

* [TPU]Fix KV cache sharing tests (vllm-project#19371)

* [HOT-FIX] Add `kv_sharing_target_layer_name` argument to cutlass_mla backend (vllm-project#19374)

Signed-off-by: Pavani Majety <pmajety@nvidia.com>

* [Misc] Fix a config typo in disable_hybrid_kv_cache_manager configuration (vllm-project#19383)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (vllm-project#19312)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [Bugfix] Fix benchmark_moe.py (vllm-project#19016)

Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>

* Use xla flag to improve the quantized model performance (vllm-project#19303)

Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

* Fix docs/mkdocs/hooks/remove_announcement.py (vllm-project#19382)

* [Frontend] Add tqdm_leave_pbar to control progress bar visibility (vllm-project#19357)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Core] Use tuple for kv cache group block ids (vllm-project#19175)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix modelscope token passed in (vllm-project#19389)

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Batch multi modal input using pinned memory (vllm-project#19169)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add security warning to bug report template (vllm-project#19365)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [Misc] refactor neuron_multimodal and profiling (vllm-project#19397)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* Add clear documentation around the impact of debugging flag (vllm-project#19369)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (vllm-project#17930)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>

* Revert "[v1] Add fp32 support to v1 engine through flex attn" (vllm-project#19404)

* [BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword `use_irope` (vllm-project#19134)

Signed-off-by: Yunqiu Guo <guorachel@meta.com>

* [BugFix][CPU] Fix CPU CI by ignore collecting test_pixtral (vllm-project#19411)

Signed-off-by: jiang.li <jiang1.li@intel.com>

* Simplify ep kernels installation (vllm-project#19412)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [Misc] Slight improvement of the BNB  (vllm-project#19418)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Docs] Note that alternative structured output backends are supported (vllm-project#19426)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (vllm-project#19440)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [Model] use AutoWeightsLoader for commandr (vllm-project#19399)

Signed-off-by: py-andy-c <pychen1017@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B-FP8 (vllm-project#19401)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* [BugFix] Allow use_cudagraph to work with dynamic VLLM_USE_V1 (vllm-project#19390)

Signed-off-by: rzou <zou3519@gmail.com>

* [New Model]: Support Qwen3 Embedding & Reranker  (vllm-project#19260)

* [BugFix] Fix docker build cpu-dev image error (vllm-project#19394)

Signed-off-by: niu_he <carlton2tang@gmail.com>

* Fix test_max_model_len in tests/entrypoints/llm/test_generate.py (vllm-project#19451)

Signed-off-by: Lu Fang <lufang@fb.com>

* [CI] Disable failing GGUF model test (vllm-project#19454)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Misc] Remove unused `MultiModalHasher.hash_prompt_mm_data` (vllm-project#19422)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add fused MOE config for Qwen3 30B A3B on B200 (vllm-project#19455)

Signed-off-by: Junhao Li <junhao@ubicloud.com>

* Fix Typo in Documentation and Function Name (vllm-project#19442)

* [ROCm] Add rules to automatically label ROCm related PRs (vllm-project#19405)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Support deep_gemm for linear methods (vllm-project#19085)

Signed-off-by: artetaout <lulala341@gmail.com>

* [Doc] Update V1 User Guide for Hardware and Models (vllm-project#19474)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Fix quantization link titles (vllm-project#19478)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Support "important" and "announcement" admonitions (vllm-project#19479)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Reduce warning message introduced in env_override (vllm-project#19476)

Signed-off-by: Lu Fang <lufang@fb.com>

* Support non-string values in JSON keys from CLI (vllm-project#19471)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add cache to cuda get_device_capability (vllm-project#19436)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix some typo (vllm-project#19475)

Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>

* Support no privileged mode on CPU for docker and kubernetes deployments (vllm-project#19241)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>

* [Bugfix] Update the example code, make it work with the latest lmcache (vllm-project#19453)

Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>

* [CI] Update FlashInfer to 0.2.6.post1 (vllm-project#19297)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [doc] fix "Other AI accelerators" getting started page (vllm-project#19457)

Signed-off-by: David Xia <david@davidxia.com>

* [Misc] Fix  misleading ROCm warning (vllm-project#19486)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Docs] Remove WIP features in V1 guide (vllm-project#19498)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Kernels] Add activation chunking logic to FusedMoEModularKernel (vllm-project#19168)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (vllm-project#17331)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [UX] Add Feedback During CUDAGraph Capture (vllm-project#19501)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

* [CI/Build] Fix torch nightly CI dependencies (vllm-project#19505)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [CI] change spell checker from codespell to typos (vllm-project#18711)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [BugFix] Force registration of w8a8_block_fp8_matmul_deepgemm via lazy import (vllm-project#19514)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* Add Triton Fused MoE kernel config for E=16 on B200 (vllm-project#19518)

Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>

* [Frontend] Improve error message in tool_choice validation (vllm-project#19239)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [BugFix] Work-around incremental detokenization edge case error (vllm-project#19449)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [BugFix] Handle missing sep_token for Qwen3-Reranker in Score API (vllm-project#19522)

Signed-off-by: strutive07 <strutive07@gmail.com>

* [AMD][Kernel][BugFix] fix test_rocm_compressed_tensors_w8a8 for rocm (vllm-project#19509)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* Fix typo (vllm-project#19525)

Signed-off-by: 2niuhe <carlton2tang@gmail.com>

* [Security] Prevent new imports of (cloud)pickle (vllm-project#18018)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Bugfix][V1] Allow manual FlashAttention for Blackwell (vllm-project#19492)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Bugfix] Respect num-gpu-blocks-override in v1 (vllm-project#19503)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Quantization] Improve AWQ logic (vllm-project#19431)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Doc] Add V1 column to supported models list (vllm-project#19523)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [V1][NixlConnector] Drop `num_blocks` check  (vllm-project#19532)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [Perf] Vectorize static / dynamic INT8 quant kernels (vllm-project#19233)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Fix TorchAOConfig skip layers (vllm-project#19265)

Signed-off-by: mobicham <hicham@mobiuslabs.com>

* [torch.compile][ROCm] Fuse quantization onto attention using a torch.compile pass (vllm-project#16756)

Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>

* [doc] Make top navigation sticky (vllm-project#19540)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Spec Decode][Benchmark] Generalize spec decode offline benchmark to more methods and datasets (vllm-project#18847)

* [Misc] Turn MOE_DP_CHUNK_SIZE into an env var (vllm-project#19506)

* [Bugfix] Enforce contiguous input for dynamic_per_token FP8/INT8 quant (vllm-project#19452)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Doc] Unify structured outputs examples (vllm-project#18196)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [V1] Resolve failed concurrent structured output requests (vllm-project#19565)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Revert "[Build/CI] Add tracing deps to vllm container image (vllm-project#15224)" (vllm-project#19378)

* [BugFix] : Fix Batched DeepGemm Experts (vllm-project#19515)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [Bugfix] Fix EAGLE vocab embedding for multimodal target model (vllm-project#19570)

Signed-off-by: qizixi <qizixi@meta.com>

* [Doc] uses absolute links for structured outputs (vllm-project#19582)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [doc] fix incorrect link (vllm-project#19586)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Correct broken docs link (vllm-project#19553)

Signed-off-by: Zerohertz <ohg3417@gmail.com>

* [CPU] Refine default config for the CPU backend (vllm-project#19539)

Signed-off-by: jiang1.li <jiang1.li@intel.com>

* [Fix] bump mistral common to support magistral (vllm-project#19533)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [Fix] The zip function in Python 3.9 does not have the strict argument (vllm-project#19549)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* use base version for version comparison (vllm-project#19587)

Signed-off-by: Boyuan Feng <boyuan@meta.com>

* [torch.compile] reorganize the cache directory to support compiling multiple models (vllm-project#19064)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [BugFix] Honor `enable_caching` in connector-delayed kvcache load case (vllm-project#19435)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Model] Fix minimax model cache & lm_head precision (vllm-project#19592)

Signed-off-by: qingjun <qingjun@minimaxi.com>

* [Refactor] Remove unused variables in `moe_permute_unpermute_kernel.inl` (vllm-project#19573)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [doc][mkdocs] fix the  duplicate Supported features sections in GPU docs (vllm-project#19606)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CUDA] Enable full cudagraph for FlashMLA (vllm-project#18581)

Signed-off-by: luka <luka@neuralmagic.com>

* [Doc] Add troubleshooting section to k8s deployment (vllm-project#19377)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* [torch.compile] Use custom ops when use_inductor=False (vllm-project#19618)

* Adding "AMD: Multi-step Tests" to amdproduction. (vllm-project#19508)

Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [BugFix] Fix DP Coordinator incorrect debug log message (vllm-project#19624)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [V1][Metrics] Deprecate metrics with gpu_ prefix for non GPU specific metrics. (vllm-project#18354)

Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>

* [Bugfix] Fix the speculative decoding test by setting the target dtype (vllm-project#19633)

* [Misc] Modularize CLI Argument Parsing in Benchmark Scripts (vllm-project#19593)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] Fix auto dtype casting for BatchFeature (vllm-project#19316)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization (vllm-project#19500)

* Only build CUTLASS MoE kernels on Hopper (vllm-project#19648)

* [Bugfix] Don't attempt to use triton if no driver is active (vllm-project#19561)

* [Fix] Convert kv_transfer_config from dict to KVTransferConfig (vllm-project#19262)

* [Perf] Further tunings for SM100 FP8 CUTLASS kernel (vllm-project#19566)

* [Bugfix][2/n] Fix speculative decoding CI - Fix test_ngram_e2e_greedy_correctness (vllm-project#19644)

* [Kernel] Raise verbose error and consolidate `num_heads/num_kv_heads` divisibility check (vllm-project#19339)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Benchmark] Refactor benchmark script for fp8 & int8 (vllm-project#19627)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Enable prefix caching with full cuda graphs (vllm-project#19617)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [CI/Build] Fix torch nightly CI dependencies part 2 (vllm-project#19589)

* [Misc] Remove duplicate multiproc method setting for CPU platform (vllm-project#19649)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] Remove unused variableds in C++ (vllm-project#19609)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Bugfix][Core] Prefix caching causes incorrect outputs due to outdated ComputedBlocksTracker (vllm-project#18957)

Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>

* [Misc][Frontend] passthrough `bad_words` (vllm-project#19564)

Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Misc] Fix skipped max-model-len validation when deriving max model length from tokenizer config (vllm-project#19660)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [TPU] support attention head dim smaller than 128 (vllm-project#19620)

Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [MISC] typo fix (vllm-project#19672)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [CI] Add mteb testing for rerank models (vllm-project#19344)

* [Docs] Move multiproc doc to v1 dir (vllm-project#19651)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Kernel] GGUF MMVQ kernel for multiple input vectors (vllm-project#18754)

Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>

* [BugFix] Don't catch BaseException when dumping execute_model errors (vllm-project#19626)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [DOC] Add reasoning capability to vLLM streamlit code (vllm-project#19557)

* [Feature]:Allow for Granite MoE Hybrid models with _only_ shared experts. (vllm-project#19652)

Signed-off-by: Shawn Tan <shawntan@ibm.com>

* [Bugfix] Fix TP inference for Flex attention backend (vllm-project#19657)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] bump huggingface_hub pkg to 0.33.0 (vllm-project#19547)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [Bugfix] fix missing 'finish_reason': null in streaming chat (vllm-project#19662)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Kernels] Use empty for modular MoE workspaces (vllm-project#19667)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [Model] Add support for MiniMaxM1ForCausalLM (shares architecture with MiniMaxText01ForCausalLM) (vllm-project#19677)

Signed-off-by: QscQ <qscqesze@gmail.com>

* [V1] Change return type on get_multimodal_embeddings() (vllm-project#19446)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* fix

Signed-off-by: Amog Kamsetty <amogkamsetty@gmail.com>

* remove logging

Signed-off-by: Amog Kamsetty <amogkamsetty@gmail.com>

---------

Signed-off-by: raushan <raushan@huggingface.co>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: googs1025 <googs1025@gmail.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Varun <vsundarr@redhat.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: calvin chen <120380290@qq.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Yang Wang <elainewy@meta.com>
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Signed-off-by: Povilas Kanapickas <povilas@radix.lt>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Signed-off-by: Xu Song <xusong.vip@gmail.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: Siqi Yan <siqi@meta.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Chenyaaang <chenyangli@google.com>
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Qiliang Cui <derrhein@gmail.com>
Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Lifan Shen <lifans@meta.com>
Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>
Signed-off-by: luka <luka@neuralmagic.com>
Signed-off-by: Richard Zou <zou3519@gmail.com>
Signed-off-by: Xu Wenqing <xuwq1993@qq.com>
Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Signed-off-by: yZhen <yZhen@fb.com>
Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>
Signed-off-by: cr7258 <chengzw258@163.com>
Signed-off-by: Conroy Cheers <conroy@corncheese.org>
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: liusiqian <liusiqian@tal.com>
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Anna Pendleton <pendleton@google.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: Yunqiu Guo <guorachel@meta.com>
Signed-off-by: jiang.li <jiang1.li@intel.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: py-andy-c <pychen1017@gmail.com>
Signed-off-by: niu_he <carlton2tang@gmail.com>
Signed-off-by: Junhao Li <junhao@ubicloud.com>
Signed-off-by: artetaout <lulala341@gmail.com>
Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>
Signed-off-by: David Xia <david@davidxia.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Signed-off-by: strutive07 <strutive07@gmail.com>
Signed-off-by: 2niuhe <carlton2tang@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: mobicham <hicham@mobiuslabs.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: qizixi <qizixi@meta.com>
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: qingjun <qingjun@minimaxi.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>
Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>
Signed-off-by: Shawn Tan <shawntan@ibm.com>
Signed-off-by: QscQ <qscqesze@gmail.com>
Signed-off-by: Amog Kamsetty <amogkamsetty@gmail.com>
Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: CYJiang <86391540+googs1025@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: SorenDreano <71752785+SorenDreano@users.noreply.github.com>
Co-authored-by: Soren Dreano <soren@numind.ai>
Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Jiaxin Shan <seedjeffwan@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Co-authored-by: Lukas Geiger <lukas.geiger94@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Calvin Chen <45745657+calvin0327@users.noreply.github.com>
Co-authored-by: Kaixi Hou <kaixih@nvidia.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>
Co-authored-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Xu Wenqing <121550081+Xu-Wenqing@users.noreply.github.com>
Co-authored-by: Lain <fusiyuan2000@hotmail.com>
Co-authored-by: jmswen <jmswen@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Kebe <mail@kebe7jun.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yang Wang <elainewy@meta.com>
Co-authored-by: Huy Do <huydhn@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Chiyue Wei <92623189+dubcyfor3@users.noreply.github.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Povilas Kanapickas <povilas@radix.lt>
Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>
Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Jerry Zhang <jerryzh168@gmail.com>
Co-authored-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: Xu Song <xusong.vip@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Jinghui Zhang <jinghuizhang0804@gmail.com>
Co-authored-by: jinghui <jinghui@fb.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
Co-authored-by: Siqi Yan <ysq0807@hotmail.com>
Co-authored-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Yu Guo <82124926+yuguo68@users.noreply.github.com>
Co-authored-by: Nishidha <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Co-authored-by: Adolfo Victoria <adolfokarim@gmail.com>
Co-authored-by: Adolfo Victoria <adovi@meta.com>
Co-authored-by: Chenyaaang <42742451+Chenyaaang@users.noreply.github.com>
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: ElizaWszola <ewszola@redhat.com>
Co-authored-by: QiliangCui <derrhein@gmail.com>
Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com>
Co-authored-by: Driss Guessous <32754868+drisspg@users.noreply.github.com>
Co-authored-by: Lifans <draftbks@gmail.com>
Co-authored-by: pramenku <7664080+pramenku@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Akash kaothalkar <61960177+Akashcodes732@users.noreply.github.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: jennyyyyzhen <47012288+jennyyyyzhen@users.noreply.github.com>
Co-authored-by: yZhen <yZhen@fb.com>
Co-authored-by: Kseniya Parkhamchuk <43078183+KsuParkhamchuk@users.noreply.github.com>
Co-authored-by: Se7en <chengzw258@163.com>
Co-authored-by: Conroy Cheers <conroy@corncheese.org>
Co-authored-by: Michael Yao <haifeng.yao@daocloud.io>
Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: liusiqian-tal <141730978+liusiqian-tal@users.noreply.github.com>
Co-authored-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: Ye (Charlotte) Qi <yeq@meta.com>
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Co-authored-by: XiongfeiWei <isaacwxf23@gmail.com>
Co-authored-by: Li Wang <wangli858794774@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Anna Pendleton <pendleton@google.com>
Co-authored-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>
Co-authored-by: Rachel Guo <35738743+YUNQIUGUO@users.noreply.github.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: py-andy-c <37168711+py-andy-c@users.noreply.github.com>
Co-authored-by: niu_he <carlton2tang@gmail.com>
Co-authored-by: Junhao Li <junhao@ubicloud.com>
Co-authored-by: leopardracer <136604165+leopardracer@users.noreply.github.com>
Co-authored-by: artetaout <128046886+artetaout@users.noreply.github.com>
Co-authored-by: Ximingwang-09 <72070413+Ximingwang-09@users.noreply.github.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: runzhen <wangrunzhen@gmail.com>
Co-authored-by: David Xia <david@davidxia.com>
Co-authored-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: rasmith <Randall.Smith@amd.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: wonjun Jang <strutive07@gmail.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mobicham <37179323+mobicham@users.noreply.github.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: kourosh hakhamaneshi <31483498+kouroshHakha@users.noreply.github.com>
Co-authored-by: qizixi <22851944+zixi-qi@users.noreply.github.com>
Co-authored-by: Hyogeun Oh (오효근) <ohg3417@gmail.com>
Co-authored-by: Boyuan Feng <fby.1994@gmail.com>
Co-authored-by: qscqesze <qingjun@minimaxi.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Saheli Bhattacharjee <47847054+sahelib25@users.noreply.github.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: maobaolong <baoloongmao@tencent.com>
Co-authored-by: Ilya Markov <markovilya197@gmail.com>
Co-authored-by: quanliu <33453350+quanliu1991@users.noreply.github.com>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: Francesco Bertolotti <f14.bertolotti@gmail.com>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Szymon Ożóg <58388001+SzymonOzog@users.noreply.github.com>
Co-authored-by: Navanit Dubey <98005188+Navanit-git@users.noreply.github.com>
Co-authored-by: Shawn Tan <shawntan@ibm.com>
Co-authored-by: qscqesze <qscqesze@gmail.com>
amogkam added a commit to character-tech/vllm that referenced this pull request Jun 16, 2025
@amogkam @zucchini-nlp
@houseroad @NickLucche @googs1025 @simon-mo @SorenDreano @SorenNumind @reidliu41 @varun-sundar-rabindranath @sarckk @mgoin @hmellor @Yikun @heheda12345 @ekagra-ranjan @chaunceyjiang @robertgshaw2-redhat @PeaBrane @Jeffwan @russellb @markmc @DarkLight1337 @bigPYJ1151 @lgeiger @vadiklyutiy @calvin0327 @kaixih @WoosukKwon @princepride @lsy323 @eicherseiji @Isotr0py @noooop @Xu-Wenqing @IwakuraRein @jmswen @tlrmchlsmth @kebe7jun @njhill @yangw-dev @huydhn @gemini-code-assist @vllmellm @22quinn @gcalmettes @patrickvonplaten @dubcyfor3 @p12tic @dsikka @vegaluisjose @jerryzh168 @benchislett @yaochengji @xu-song @aarnphm @KingsleyZhang123 @zou3519 @cryptopic @jeejeelee @yuguo68 @npanpaliya @Shafi-Hussain @Adolfo-Karim @Chenyaaang @Alexei-V-Ivanov-AMD @ElizaWszola @QiliangCui @AaruniAggarwal @drisspg @draftbk @pramenku @ProExpertProg @Akashcodes732 @jennyyyyzhen @KsuParkhamchuk @cr7258 @conroy-cheers @windsonsea @yinghai @kylesayrs @liusiqian-tal @pavanimajety @yeqcharlotte @gty111 @vanbasten23 @Potabk @Copilot @annapendleton @louie-tsai @YUNQIUGUO @youkaichao @gshtras @py-andy-c @2niuhe @0xjunhao @leopardracer @artetaout @Ximingwang-09 @runzhen @davidxia @bnellnm @rasmith @andyxning @b8zhong @strutive07 @yewentao256 @mobicham @SageMoore @kouroshHakha @zixi-qi @Zerohertz @BoyuanFeng @qscqesze @Concurrensee @sahelib25 @jiahanc @kzawora-intel @maobaolong @ilmarkov @quanliu1991 @f14-bertolotti @francesco-bertolotti @SzymonOzog @Navanit-git @shawntan
Fix (#9)
2d162c3 
* [Bugfix] disable processor cache  (vllm-project#19068)

Signed-off-by: raushan <raushan@huggingface.co>

* [Doc] Improve the Pull Request template with key components (vllm-project#19086)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Misc] Add missing `_Backend` enums (vllm-project#19081)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [Misc] fix: add miss best_of param validation (vllm-project#18555)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [Misc] Add SPDX-FileCopyrightText  (vllm-project#19100)

Signed-off-by: simon-mo <simon.mo@hey.com>

* [Doc] Readme standardization (vllm-project#18695)

Co-authored-by: Soren Dreano <soren@numind.ai>

* [doc] update docker version (vllm-project#19074)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Kernel] DeepEP dispatch-combine kernel integration (vllm-project#18434)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [V1] Support cross-layer KV sharing (vllm-project#18212)

Signed-off-by: Yong Hoon Shin <yhshin@meta.com>

* [Perf] Tune `scaled_fp8_quant` by increasing vectorization (vllm-project#18844)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix interaction between `Optional` and `Annotated` in CLI typing (vllm-project#19093)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>

* [v1] Re-init input batch for multiple kv cache groups (vllm-project#18654)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [V1][Spec Decode][Ngram] 1.35x gain -> 1.95x gain on InstructCoder with prompt fix (vllm-project#18971)

* [Bugfix] get_num_blocks_to_allocate with null_block (vllm-project#19031)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled (vllm-project#19075)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix][P/D] Fix Prefix Cache Bug (vllm-project#18411)

Signed-off-by: nicklucche <nlucches@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

* [Bugfix] Max concurrency estimation and check_enough_kv_cache_memory for models with sliding window layers (vllm-project#19029)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* feat: add data parallel rank to KVEventBatch (vllm-project#18925)

* [Misc] Fix path and python alias errors in disagg_prefill exmaples (vllm-project#18919)

* [Docs] Add developer doc about CI failures (vllm-project#18782)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [CPU] V1 support for the CPU backend (vllm-project#16441)

* [Core] Cast multimodal input in hf processor (vllm-project#18862)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* [KERNEL] Sampler. CUDA kernel for applying repetition penalty (vllm-project#18437)

* [Cleanup][v1]:remote guided-decoding-backend for example (vllm-project#19059)

Signed-off-by: calvin chen <120380290@qq.com>

* [NVIDIA] Add Cutlass MLA backend (vllm-project#17625)

* [Bugfix] Fix FA3 full cuda graph correctness (vllm-project#19106)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* Fix vllm-project#19130 (vllm-project#19132)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [TPU] Skip hanging tests (vllm-project#19115)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* Fix ValueError: Missing value for tag key(s): model_name,engine. (vllm-project#19113)

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

* [Misc] Add packages for benchmark as extra dependency (vllm-project#19089)

Signed-off-by: Isotr0py <2037008807@qq.com>

* Improve the output precision of embedding models (vllm-project#19092)

* [CI/Build][Bugfix] Ensure compatibility with transformers 4.52 (vllm-project#18678)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add DeepSeek-R1-0528 function call chat template (vllm-project#18874)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* Sm100 blockwise fp8 swap ab (vllm-project#18564)

* [Doc] Update V1 Guide for embedding models (vllm-project#19141)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Allow AsyncLLMEngine.generate to target a specific DP rank (vllm-project#19102)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Bugfix][EP+DP] Fix internode check (vllm-project#19112)

Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>

* [Perf] Tunings for SM100 FP8 CUTLASS kernel (vllm-project#18778)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [TPU] Update dynamo dump file name in compilation test (vllm-project#19108)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [Bugfix] fix v1 cpu worker fails on macOS (vllm-project#19121)

* [Kernel] Integrate batched/masked deepgemm kernel (vllm-project#19111)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Misc] refactor: simplify EngineCoreClient.make_async_mp_client in AsyncLLM (vllm-project#18817)

Signed-off-by: googs1025 <googs1025@gmail.com>

* [P/D] Heterogeneous TP (vllm-project#18833)

Signed-off-by: nicklucche <nlucches@redhat.com>

* [doc] small fix (vllm-project#19167)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix][Nixl] Fix full prefix cache hit bug (vllm-project#18632)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix port handling in make_zmq_path (vllm-project#19117)

* [Torch Nightly]add missing dependency (vllm-project#18770)

Signed-off-by: Yang Wang <elainewy@meta.com>

* Handle non-serializable objects when dumping benchmark results (vllm-project#19114)

* [BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (vllm-project#19171)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Bugfix]: Fix the incompatibility issue with stream when Thinking is disabled (vllm-project#19135)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Build] Annotate wheel and container path for release workflow (vllm-project#19162)

Signed-off-by: simon-mo <simon.mo@hey.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Misc] Remove unnecessary fallback to prefill-decode attention (vllm-project#19138)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

* [Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly (vllm-project#19105)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Frontend] improve vllm run-batch --help display (vllm-project#19187)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] properly catch PIL-related errors for vision models when incorrect data urls are provided (vllm-project#19202)

Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>

* [mistral_common] Add v11 tokenizer (vllm-project#19193)

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 (vllm-project#19205)

* [Hardware][NVIDIA] FP4 MoE kernel optimization (vllm-project#19110)

Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>

* [MISC][Bugfix] Use less CPU when message queue has been empty for some time (vllm-project#16226)

Signed-off-by: Povilas Kanapickas <povilas@radix.lt>

* [P/D][NixlConnector] Enable FlashInfer backend (vllm-project#19090)

* [Quantization] Skip Fp4 Test for `compressed-tensors` (vllm-project#19217)

* [V1] Use FlashInfer by default on Blackwell GPUs (vllm-project#19118)

* [Model] NemotronH support (vllm-project#18863)

Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>

* Fix AOPerModuleConfig name changes (vllm-project#18869)

Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>

* [Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (vllm-project#19033)

Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>

* [v1] Hybrid Memory Allocator (vllm-project#17996)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [TPU] update torch_xla pin (vllm-project#19231)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* Support allowed_token_ids in ChatCompletionRequest (vllm-project#19143)

Signed-off-by: Xu Song <xusong.vip@gmail.com>

* [Chore] update CODEOWNERS (vllm-project#19247)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [v1][P/D] Fix a edge case in kv cache schedule (vllm-project#19182)

Co-authored-by: jinghui <jinghui@fb.com>

* [TPU] fix kv cache dtype in model runner (vllm-project#19244)

Signed-off-by: Chengji Yao <chengjiyao@google.com>

* [Quantization] Bump compressed-tensors version; update NVFP4A16 test model (vllm-project#19224)

Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>

* [Docs] Improve V1 KVConnector interface documentation (vllm-project#19172)

Signed-off-by: Nick Hill <nhill@redhat.com>

* Fix CompilationConfig repr (vllm-project#19091)

Signed-off-by: rzou <zou3519@gmail.com>

* Unit Test for run_dp_sharded_vision_model (vllm-project#19103)

Signed-off-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Siqi Yan <siqi@meta.com>

* [Model] Optimize nemotron_h implementation (vllm-project#19249)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Raise when non-multi-instance DP clients target a DP rank (vllm-project#19227)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* improve logits bias (vllm-project#19041)

* Fixed ppc build when it runs on non-RHEL based linux distros (vllm-project#18422)

Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>

* [BugFix] Fix MultiConnector test after HMA changes (vllm-project#19291)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix][Core] Update cancellation logic in `generate()` to handle Generator exits (vllm-project#19225)

Co-authored-by: Adolfo Victoria <adovi@meta.com>

* [Core] Fix abrupt request abort (vllm-project#18485)

Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>

Co-authored-by: Nick Hill <nhill@redhat.com>

* [BugFix] Fix tpu_model_runner block_id concatenation (vllm-project#19228)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Misc][Tools][Benchmark] Fix and improve auto tune script (vllm-project#19163)

Signed-off-by: Chenyaaang <chenyangli@google.com>

* [Build][ROCm] Update Dockerfile.rocm (vllm-project#19296)

Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

* [Easy][Test] Simplify test_function_tool_use with multiple parametrizes (vllm-project#19269)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Integrate CUTLASS MoE kernel with PPLX (vllm-project#18762)

Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

* [TPU][Test] Add script to run benchmark on TPU for buildkite (vllm-project#19039)

Signed-off-by: Qiliang Cui <derrhein@gmail.com>

* [CI][PowerPC] Use a more appropriate way to select testcase in tests/models/language/pooling/test_embedding.py (vllm-project#19253)

Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>

* Add FlexAttention to V1 (vllm-project#16078)

Signed-off-by: drisspg <drisspguessous@gmail.com>

* [Misc] refactor context extension (vllm-project#19246)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CI/Build] Improve Llama GGUF test robustness (vllm-project#19287)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Nit][Benchmark]Fix example in benchmark_serving_structured_output.py (vllm-project#19311)

Signed-off-by: Lifan Shen <lifans@meta.com>

* [AMD] Update compatible packaging version (vllm-project#19309)

Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>

* [BugFix][V1] Fix memory profiling bug (vllm-project#18974)

Signed-off-by: luka <luka@neuralmagic.com>

* [Bugfix]: Fix TypeError: 'float' object cannot be interpreted as an integer (vllm-project#19283)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Bugfix] Re-enable use_cudagraph in vLLM v1 (vllm-project#19299)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [Misc] Change tests/compile to use VLLM_V1 by default (vllm-project#19302)

Signed-off-by: rzou <zou3519@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B (vllm-project#19315)

Signed-off-by: Xu Wenqing <xuwq1993@qq.com>

* [Hardware][POWER] Add IBM POWER11 Support to CPU Extension Detection (vllm-project#19082)

Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>

* [Quantization] Add compressed-tensors NVFP4 support (vllm-project#18312)

* [Multi Modal] Add an env var for message queue max chunk bytes  (vllm-project#19242)

Signed-off-by: yZhen <yZhen@fb.com>
Co-authored-by: yZhen <yZhen@fb.com>

* [Bugfix] model_max_length should consider max_model_len in tokenizer_config (vllm-project#19201)

* [Deprecation] Remove `inputs` arg fallback in Engine classes (vllm-project#18799)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Add documentation update reminder to PR template (vllm-project#19289)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [Frontend] Remove unreachable code from llm.py (vllm-project#19288)

Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>

* [Misc] Cleanup compilation tests (vllm-project#19343)

Signed-off-by: rzou <zou3519@gmail.com>

* [doc] improve ci doc (vllm-project#19307)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Doc] Fix description in the Automatic Prefix Caching design doc (vllm-project#19333)

Signed-off-by: cr7258 <chengzw258@163.com>

* [CI/Build] Fix LoRA test (vllm-project#19350)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Fix] Allow kernel compilation for CUDA capability 8.7 (vllm-project#19328)

Signed-off-by: Conroy Cheers <conroy@corncheese.org>

* [CI] Introduce rules for llama auto-label (vllm-project#19323)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Docs] Fix a bullet list in usage/security.md (vllm-project#19358)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* [full_graph] Fix query_start_loc padding (vllm-project#19321)

Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>

* [v1] Add fp32 support to v1 engine through flex attn (vllm-project#19319)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (vllm-project#19298)

Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>

* [Bugfix][Core] Prevent token lengths exceeding `max_model_len` in V0 (vllm-project#19348)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Quantization] Bump compressed-tensors version (vllm-project#19295)

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

* [Frontend] Make TIMEOUT_KEEP_ALIVE configurable through env var (vllm-project#18472)

Signed-off-by: liusiqian <liusiqian@tal.com>

* [TPU]Fix KV cache sharing tests (vllm-project#19371)

* [HOT-FIX] Add `kv_sharing_target_layer_name` argument to cutlass_mla backend (vllm-project#19374)

Signed-off-by: Pavani Majety <pmajety@nvidia.com>

* [Misc] Fix a config typo in disable_hybrid_kv_cache_manager configuration (vllm-project#19383)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (vllm-project#19312)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [Bugfix] Fix benchmark_moe.py (vllm-project#19016)

Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>

* Use xla flag to improve the quantized model performance (vllm-project#19303)

Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

* Fix docs/mkdocs/hooks/remove_announcement.py (vllm-project#19382)

* [Frontend] Add tqdm_leave_pbar to control progress bar visibility (vllm-project#19357)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Core] Use tuple for kv cache group block ids (vllm-project#19175)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix modelscope token passed in (vllm-project#19389)

Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

* [Core] Batch multi modal input using pinned memory (vllm-project#19169)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add security warning to bug report template (vllm-project#19365)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [Misc] refactor neuron_multimodal and profiling (vllm-project#19397)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* Add clear documentation around the impact of debugging flag (vllm-project#19369)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (vllm-project#17930)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>

* Revert "[v1] Add fp32 support to v1 engine through flex attn" (vllm-project#19404)

* [BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword `use_irope` (vllm-project#19134)

Signed-off-by: Yunqiu Guo <guorachel@meta.com>

* [BugFix][CPU] Fix CPU CI by ignore collecting test_pixtral (vllm-project#19411)

Signed-off-by: jiang.li <jiang1.li@intel.com>

* Simplify ep kernels installation (vllm-project#19412)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [Misc] Slight improvement of the BNB  (vllm-project#19418)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Docs] Note that alternative structured output backends are supported (vllm-project#19426)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (vllm-project#19440)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [Model] use AutoWeightsLoader for commandr (vllm-project#19399)

Signed-off-by: py-andy-c <pychen1017@gmail.com>

* Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B-FP8 (vllm-project#19401)

Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>

* [BugFix] Allow use_cudagraph to work with dynamic VLLM_USE_V1 (vllm-project#19390)

Signed-off-by: rzou <zou3519@gmail.com>

* [New Model]: Support Qwen3 Embedding & Reranker  (vllm-project#19260)

* [BugFix] Fix docker build cpu-dev image error (vllm-project#19394)

Signed-off-by: niu_he <carlton2tang@gmail.com>

* Fix test_max_model_len in tests/entrypoints/llm/test_generate.py (vllm-project#19451)

Signed-off-by: Lu Fang <lufang@fb.com>

* [CI] Disable failing GGUF model test (vllm-project#19454)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Misc] Remove unused `MultiModalHasher.hash_prompt_mm_data` (vllm-project#19422)

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>

* Add fused MOE config for Qwen3 30B A3B on B200 (vllm-project#19455)

Signed-off-by: Junhao Li <junhao@ubicloud.com>

* Fix Typo in Documentation and Function Name (vllm-project#19442)

* [ROCm] Add rules to automatically label ROCm related PRs (vllm-project#19405)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Kernel] Support deep_gemm for linear methods (vllm-project#19085)

Signed-off-by: artetaout <lulala341@gmail.com>

* [Doc] Update V1 User Guide for Hardware and Models (vllm-project#19474)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Fix quantization link titles (vllm-project#19478)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Doc] Support "important" and "announcement" admonitions (vllm-project#19479)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Reduce warning message introduced in env_override (vllm-project#19476)

Signed-off-by: Lu Fang <lufang@fb.com>

* Support non-string values in JSON keys from CLI (vllm-project#19471)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Add cache to cuda get_device_capability (vllm-project#19436)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Fix some typo (vllm-project#19475)

Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>

* Support no privileged mode on CPU for docker and kubernetes deployments (vllm-project#19241)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>

* [Bugfix] Update the example code, make it work with the latest lmcache (vllm-project#19453)

Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>

* [CI] Update FlashInfer to 0.2.6.post1 (vllm-project#19297)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [doc] fix "Other AI accelerators" getting started page (vllm-project#19457)

Signed-off-by: David Xia <david@davidxia.com>

* [Misc] Fix  misleading ROCm warning (vllm-project#19486)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Docs] Remove WIP features in V1 guide (vllm-project#19498)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Kernels] Add activation chunking logic to FusedMoEModularKernel (vllm-project#19168)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (vllm-project#17331)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [UX] Add Feedback During CUDAGraph Capture (vllm-project#19501)

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

* [CI/Build] Fix torch nightly CI dependencies (vllm-project#19505)

Signed-off-by: Richard Zou <zou3519@gmail.com>

* [CI] change spell checker from codespell to typos (vllm-project#18711)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [BugFix] Force registration of w8a8_block_fp8_matmul_deepgemm via lazy import (vllm-project#19514)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* Add Triton Fused MoE kernel config for E=16 on B200 (vllm-project#19518)

Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>

* [Frontend] Improve error message in tool_choice validation (vllm-project#19239)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [BugFix] Work-around incremental detokenization edge case error (vllm-project#19449)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [BugFix] Handle missing sep_token for Qwen3-Reranker in Score API (vllm-project#19522)

Signed-off-by: strutive07 <strutive07@gmail.com>

* [AMD][Kernel][BugFix] fix test_rocm_compressed_tensors_w8a8 for rocm (vllm-project#19509)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* Fix typo (vllm-project#19525)

Signed-off-by: 2niuhe <carlton2tang@gmail.com>

* [Security] Prevent new imports of (cloud)pickle (vllm-project#18018)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Bugfix][V1] Allow manual FlashAttention for Blackwell (vllm-project#19492)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Bugfix] Respect num-gpu-blocks-override in v1 (vllm-project#19503)

Signed-off-by: Jon Swenson <jmswen@gmail.com>

* [Quantization] Improve AWQ logic (vllm-project#19431)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Doc] Add V1 column to supported models list (vllm-project#19523)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [V1][NixlConnector] Drop `num_blocks` check  (vllm-project#19532)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [Perf] Vectorize static / dynamic INT8 quant kernels (vllm-project#19233)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Fix TorchAOConfig skip layers (vllm-project#19265)

Signed-off-by: mobicham <hicham@mobiuslabs.com>

* [torch.compile][ROCm] Fuse quantization onto attention using a torch.compile pass (vllm-project#16756)

Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>

* [doc] Make top navigation sticky (vllm-project#19540)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Spec Decode][Benchmark] Generalize spec decode offline benchmark to more methods and datasets (vllm-project#18847)

* [Misc] Turn MOE_DP_CHUNK_SIZE into an env var (vllm-project#19506)

* [Bugfix] Enforce contiguous input for dynamic_per_token FP8/INT8 quant (vllm-project#19452)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Doc] Unify structured outputs examples (vllm-project#18196)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [V1] Resolve failed concurrent structured output requests (vllm-project#19565)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Revert "[Build/CI] Add tracing deps to vllm container image (vllm-project#15224)" (vllm-project#19378)

* [BugFix] : Fix Batched DeepGemm Experts (vllm-project#19515)

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

* [Bugfix] Fix EAGLE vocab embedding for multimodal target model (vllm-project#19570)

Signed-off-by: qizixi <qizixi@meta.com>

* [Doc] uses absolute links for structured outputs (vllm-project#19582)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [doc] fix incorrect link (vllm-project#19586)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Correct broken docs link (vllm-project#19553)

Signed-off-by: Zerohertz <ohg3417@gmail.com>

* [CPU] Refine default config for the CPU backend (vllm-project#19539)

Signed-off-by: jiang1.li <jiang1.li@intel.com>

* [Fix] bump mistral common to support magistral (vllm-project#19533)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* [Fix] The zip function in Python 3.9 does not have the strict argument (vllm-project#19549)

Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

* use base version for version comparison (vllm-project#19587)

Signed-off-by: Boyuan Feng <boyuan@meta.com>

* [torch.compile] reorganize the cache directory to support compiling multiple models (vllm-project#19064)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [BugFix] Honor `enable_caching` in connector-delayed kvcache load case (vllm-project#19435)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Model] Fix minimax model cache & lm_head precision (vllm-project#19592)

Signed-off-by: qingjun <qingjun@minimaxi.com>

* [Refactor] Remove unused variables in `moe_permute_unpermute_kernel.inl` (vllm-project#19573)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* [doc][mkdocs] fix the  duplicate Supported features sections in GPU docs (vllm-project#19606)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [CUDA] Enable full cudagraph for FlashMLA (vllm-project#18581)

Signed-off-by: luka <luka@neuralmagic.com>

* [Doc] Add troubleshooting section to k8s deployment (vllm-project#19377)

Signed-off-by: Anna Pendleton <pendleton@google.com>

* [torch.compile] Use custom ops when use_inductor=False (vllm-project#19618)

* Adding "AMD: Multi-step Tests" to amdproduction. (vllm-project#19508)

Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [BugFix] Fix DP Coordinator incorrect debug log message (vllm-project#19624)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [V1][Metrics] Deprecate metrics with gpu_ prefix for non GPU specific metrics. (vllm-project#18354)

Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>

* [Bugfix] Fix the speculative decoding test by setting the target dtype (vllm-project#19633)

* [Misc] Modularize CLI Argument Parsing in Benchmark Scripts (vllm-project#19593)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] Fix auto dtype casting for BatchFeature (vllm-project#19316)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* [Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization (vllm-project#19500)

* Only build CUTLASS MoE kernels on Hopper (vllm-project#19648)

* [Bugfix] Don't attempt to use triton if no driver is active (vllm-project#19561)

* [Fix] Convert kv_transfer_config from dict to KVTransferConfig (vllm-project#19262)

* [Perf] Further tunings for SM100 FP8 CUTLASS kernel (vllm-project#19566)

* [Bugfix][2/n] Fix speculative decoding CI - Fix test_ngram_e2e_greedy_correctness (vllm-project#19644)

* [Kernel] Raise verbose error and consolidate `num_heads/num_kv_heads` divisibility check (vllm-project#19339)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

* [Benchmark] Refactor benchmark script for fp8 & int8 (vllm-project#19627)

Signed-off-by: yewentao256 <zhyanwentao@126.com>

* Enable prefix caching with full cuda graphs (vllm-project#19617)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [CI/Build] Fix torch nightly CI dependencies part 2 (vllm-project#19589)

* [Misc] Remove duplicate multiproc method setting for CPU platform (vllm-project#19649)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] Remove unused variableds in C++ (vllm-project#19609)

Signed-off-by: Lu Fang <lufang@fb.com>

* [Bugfix][Core] Prefix caching causes incorrect outputs due to outdated ComputedBlocksTracker (vllm-project#18957)

Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>

* [Misc][Frontend] passthrough `bad_words` (vllm-project#19564)

Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>

* [Misc] Fix skipped max-model-len validation when deriving max model length from tokenizer config (vllm-project#19660)

Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

* [TPU] support attention head dim smaller than 128 (vllm-project#19620)

Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [MISC] typo fix (vllm-project#19672)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [CI] Add mteb testing for rerank models (vllm-project#19344)

* [Docs] Move multiproc doc to v1 dir (vllm-project#19651)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Kernel] GGUF MMVQ kernel for multiple input vectors (vllm-project#18754)

Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>

* [BugFix] Don't catch BaseException when dumping execute_model errors (vllm-project#19626)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [DOC] Add reasoning capability to vLLM streamlit code (vllm-project#19557)

* [Feature]:Allow for Granite MoE Hybrid models with _only_ shared experts. (vllm-project#19652)

Signed-off-by: Shawn Tan <shawntan@ibm.com>

* [Bugfix] Fix TP inference for Flex attention backend (vllm-project#19657)

Signed-off-by: Isotr0py <2037008807@qq.com>

* [MISC] bump huggingface_hub pkg to 0.33.0 (vllm-project#19547)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [Bugfix] fix missing 'finish_reason': null in streaming chat (vllm-project#19662)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Kernels] Use empty for modular MoE workspaces (vllm-project#19667)

Signed-off-by: Bill Nell <bnell@redhat.com>

* [Model] Add support for MiniMaxM1ForCausalLM (shares architecture with MiniMaxText01ForCausalLM) (vllm-project#19677)

Signed-off-by: QscQ <qscqesze@gmail.com>

* [V1] Change return type on get_multimodal_embeddings() (vllm-project#19446)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* fix

Signed-off-by: Amog Kamsetty <amogkamsetty@gmail.com>

---------

Signed-off-by: raushan <raushan@huggingface.co>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: googs1025 <googs1025@gmail.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Varun <vsundarr@redhat.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: calvin chen <120380290@qq.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 许文卿 <xwq391974@alibaba-inc.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Yang Wang <elainewy@meta.com>
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Chiyue Wei <chiyuew@nvidia.com>
Signed-off-by: Povilas Kanapickas <povilas@radix.lt>
Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Signed-off-by: Xu Song <xusong.vip@gmail.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: Siqi Yan <siqi@meta.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Signed-off-by: npanpaliya <nishidha.panpaliya@partner.ibm.com>
Signed-off-by: Chenyaaang <chenyangli@google.com>
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Qiliang Cui <derrhein@gmail.com>
Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Lifan Shen <lifans@meta.com>
Signed-off-by: pramkuma <Pramendra.Kumar@amd.com>
Signed-off-by: luka <luka@neuralmagic.com>
Signed-off-by: Richard Zou <zou3519@gmail.com>
Signed-off-by: Xu Wenqing <xuwq1993@qq.com>
Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Signed-off-by: yZhen <yZhen@fb.com>
Signed-off-by: KsuParkhamchuk <k.parkhamchuk@gmail.com>
Signed-off-by: cr7258 <chengzw258@163.com>
Signed-off-by: Conroy Cheers <conroy@corncheese.org>
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: liusiqian <liusiqian@tal.com>
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Signed-off-by: wangli <wangli858794774@gmail.com>
Signed-off-by: Anna Pendleton <pendleton@google.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: Yunqiu Guo <guorachel@meta.com>
Signed-off-by: jiang.li <jiang1.li@intel.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: py-andy-c <pychen1017@gmail.com>
Signed-off-by: niu_he <carlton2tang@gmail.com>
Signed-off-by: Junhao Li <junhao@ubicloud.com>
Signed-off-by: artetaout <lulala341@gmail.com>
Signed-off-by: ximing.wxm <ximing.wxm@antgroup.com>
Signed-off-by: Runzhen Wang <wangrunzhen@gmail.com>
Signed-off-by: David Xia <david@davidxia.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Signed-off-by: strutive07 <strutive07@gmail.com>
Signed-off-by: 2niuhe <carlton2tang@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: mobicham <hicham@mobiuslabs.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: qizixi <qizixi@meta.com>
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: qingjun <qingjun@minimaxi.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>
Signed-off-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Signed-off-by: SzymonOzog <szymon.ozog@gmail.com>
Signed-off-by: Shawn Tan <shawntan@ibm.com>
Signed-off-by: QscQ <qscqesze@gmail.com>
Signed-off-by: Amog Kamsetty <amogkamsetty@gmail.com>
Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: CYJiang <86391540+googs1025@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: SorenDreano <71752785+SorenDreano@users.noreply.github.com>
Co-authored-by: Soren Dreano <soren@numind.ai>
Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Jiaxin Shan <seedjeffwan@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Co-authored-by: Lukas Geiger <lukas.geiger94@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Calvin Chen <45745657+calvin0327@users.noreply.github.com>
Co-authored-by: Kaixi Hou <kaixih@nvidia.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>
Co-authored-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Xu Wenqing <121550081+Xu-Wenqing@users.noreply.github.com>
Co-authored-by: Lain <fusiyuan2000@hotmail.com>
Co-authored-by: jmswen <jmswen@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Kebe <mail@kebe7jun.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yang Wang <elainewy@meta.com>
Co-authored-by: Huy Do <huydhn@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Guillaume Calmettes <gcalmettes@scaleway.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Chiyue Wei <92623189+dubcyfor3@users.noreply.github.com>
Co-authored-by: Chiyue Wei <chiyuew@nvidia.com>
Co-authored-by: Povilas Kanapickas <povilas@radix.lt>
Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>
Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com>
Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com>
Co-authored-by: Jerry Zhang <jerryzh168@gmail.com>
Co-authored-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: Xu Song <xusong.vip@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Jinghui Zhang <jinghuizhang0804@gmail.com>
Co-authored-by: jinghui <jinghui@fb.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
Co-authored-by: Siqi Yan <ysq0807@hotmail.com>
Co-authored-by: Siqi Yan <siqi@meta.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Yu Guo <82124926+yuguo68@users.noreply.github.com>
Co-authored-by: Nishidha <nishidha.panpaliya@partner.ibm.com>
Co-authored-by: Md. Shafi Hussain <Md.Shafi.Hussain@ibm.com>
Co-authored-by: Adolfo Victoria <adolfokarim@gmail.com>
Co-authored-by: Adolfo Victoria <adovi@meta.com>
Co-authored-by: Chenyaaang <42742451+Chenyaaang@users.noreply.github.com>
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: ElizaWszola <ewszola@redhat.com>
Co-authored-by: QiliangCui <derrhein@gmail.com>
Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com>
Co-authored-by: Driss Guessous <32754868+drisspg@users.noreply.github.com>
Co-authored-by: Lifans <draftbks@gmail.com>
Co-authored-by: pramenku <7664080+pramenku@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Akash kaothalkar <61960177+Akashcodes732@users.noreply.github.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: jennyyyyzhen <47012288+jennyyyyzhen@users.noreply.github.com>
Co-authored-by: yZhen <yZhen@fb.com>
Co-authored-by: Kseniya Parkhamchuk <43078183+KsuParkhamchuk@users.noreply.github.com>
Co-authored-by: Se7en <chengzw258@163.com>
Co-authored-by: Conroy Cheers <conroy@corncheese.org>
Co-authored-by: Michael Yao <haifeng.yao@daocloud.io>
Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: liusiqian-tal <141730978+liusiqian-tal@users.noreply.github.com>
Co-authored-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: Ye (Charlotte) Qi <yeq@meta.com>
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Co-authored-by: XiongfeiWei <isaacwxf23@gmail.com>
Co-authored-by: Li Wang <wangli858794774@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Anna Pendleton <pendleton@google.com>
Co-authored-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>
Co-authored-by: Rachel Guo <35738743+YUNQIUGUO@users.noreply.github.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: py-andy-c <37168711+py-andy-c@users.noreply.github.com>
Co-authored-by: niu_he <carlton2tang@gmail.com>
Co-authored-by: Junhao Li <junhao@ubicloud.com>
Co-authored-by: leopardracer <136604165+leopardracer@users.noreply.github.com>
Co-authored-by: artetaout <128046886+artetaout@users.noreply.github.com>
Co-authored-by: Ximingwang-09 <72070413+Ximingwang-09@users.noreply.github.com>
Co-authored-by: ximing.wxm <ximing.wxm@antgroup.com>
Co-authored-by: runzhen <wangrunzhen@gmail.com>
Co-authored-by: David Xia <david@davidxia.com>
Co-authored-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: rasmith <Randall.Smith@amd.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: wonjun Jang <strutive07@gmail.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mobicham <37179323+mobicham@users.noreply.github.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: kourosh hakhamaneshi <31483498+kouroshHakha@users.noreply.github.com>
Co-authored-by: qizixi <22851944+zixi-qi@users.noreply.github.com>
Co-authored-by: Hyogeun Oh (오효근) <ohg3417@gmail.com>
Co-authored-by: Boyuan Feng <fby.1994@gmail.com>
Co-authored-by: qscqesze <qingjun@minimaxi.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Saheli Bhattacharjee <47847054+sahelib25@users.noreply.github.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: maobaolong <baoloongmao@tencent.com>
Co-authored-by: Ilya Markov <markovilya197@gmail.com>
Co-authored-by: quanliu <33453350+quanliu1991@users.noreply.github.com>
Co-authored-by: 刘全 <quan.liu2@dbappsecurity.com.cn>
Co-authored-by: Francesco Bertolotti <f14.bertolotti@gmail.com>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Szymon Ożóg <58388001+SzymonOzog@users.noreply.github.com>
Co-authored-by: Navanit Dubey <98005188+Navanit-git@users.noreply.github.com>
Co-authored-by: Shawn Tan <shawntan@ibm.com>
Co-authored-by: qscqesze <qscqesze@gmail.com>
yeqcharlotte pushed a commit to yeqcharlotte/vllm that referenced this pull request Jun 22, 2025
@Isotr0py @yeqcharlotte
[Bugfix] Fix TP inference for Flex attention backend (vllm-project#19657
d89265a 
)

Signed-off-by: Isotr0py <2037008807@qq.com>
minpeter pushed a commit to minpeter/vllm that referenced this pull request Jun 24, 2025
@Isotr0py @minpeter
[Bugfix] Fix TP inference for Flex attention backend (vllm-project#19657
ddb08da 
)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: minpeter <kali2005611@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drisspg drisspg drisspg left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

@houseroad houseroad houseroad approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed tpu Related to Google TPUs v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Isotr0py @houseroad @drisspg