[Bug]: V1 engine fails with offline batched inference code in V0 engine

[vForce825]

Your current environment

The output of `python collect_env.py`

INFO 02-08 09:51:29 __init__.py:190] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Alibaba Group Enterprise Linux Server 7.2 (Paladin) (x86_64)
GCC version: (GCC) 10.5.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.32

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.9.151-015.ali3000.alios7.x86_64-x86_64-with-glibc2.32
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB

Nvidia driver version: 470.82.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 106
Model name:            Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz
Stepping:              6
CPU MHz:               3499.859
CPU max MHz:           3500.0000
CPU min MHz:           800.0000
BogoMIPS:              5800.00
Virtualization:        VT-x
L1d cache:             48K
L1i cache:             32K
L2 cache:              1280K
L3 cache:              49152K
NUMA node0 CPU(s):     0-31,64-95
NUMA node1 CPU(s):     32-63,96-127
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] flake8==7.1.1
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] sentence-transformers==3.4.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.2
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.3                    pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] sentence-transformers     3.4.1                    pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchaudio                2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.48.2                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      NV12    0-31,64-95      0
GPU1    NV12     X      0-31,64-95      0

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=GPU-5294a6a7-80ee-6c4f-6dee-f115cffa4c48,GPU-a22ea803-549c-7751-b818-7786ff518963
LD_LIBRARY_PATH=:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64
NVIDIA_DRIVER_CAPABILITIES=all
NCCL_NVLS_ENABLE=0
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I'm inferencing multiple prompts (inconsistent length) with local vllm in python code. If changed to VLLM V1 engine, it fails. If reversed to V0 engine, it can finish without any error. The startup code is simplified like this:

model_path = "/nfs/llm_models/DeepSeek-R1-Distill-Qwen-32B"
stop_token_ids = [151645, 151643]
llm = LLM(model=model_path, tokenizer=None, tensor_parallel_size=torch.cuda.device_count(), trust_remote_code=True, gpu_memory_utilization=0.8)
tokenizer = AutoTokenizer.from_pretrained(model_path)
input_texts = []
for one_prompt in prompts:
    messages = [
        {"role": "system", "content": ""},
        {"role": "user", "content": one_prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    input_texts.append(text)
output_strs = llm.generate(input_texts, sampling_params, use_tqdm=True)

Here's the log and traceback:

Processed prompts:   0%|                                                                                                 | 4/3549 [00:50<9:04:40,  9.22s/it, est. speed input: 48.56 toks/s, output: 12.39 toks/s](VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     sampler_output = self.model.sample(
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 46, in forward
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     logits = self.apply_penalties(logits, sampling_metadata)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     logits = apply_all_penalties(
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return apply_penalties(logits, prompt_token_ids, output_tokens_t,
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     _, prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374]     bin_counts.scatter_add_(1, tokens, torch.ones_like(tokens))
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] RuntimeError: Expected index [668, 651] to be smaller than self [667, 152065] apart from dimension 1 and to be smaller size than src [668, 651]
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     output = func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     sampler_output = self.model.sample(
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 46, in forward
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     logits = self.apply_penalties(logits, sampling_metadata)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     logits = apply_all_penalties(
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     return apply_penalties(logits, prompt_token_ids, output_tokens_t,
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     _, prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]   File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374]     bin_counts.scatter_add_(1, tokens, torch.ones_like(tokens))
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] RuntimeError: Expected index [668, 651] to be smaller than self [667, 152065] apart from dimension 1 and to be smaller size than src [668, 651]
ERROR 02-08 09:31:59 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 203, in run_engine_core
ERROR 02-08 09:31:59 core.py:210]     engine_core.run_busy_loop()
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 243, in run_busy_loop
ERROR 02-08 09:31:59 core.py:210]     outputs = self.step()
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 129, in step
ERROR 02-08 09:31:59 core.py:210]     output = self.model_executor.execute_model(scheduler_output)
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 77, in execute_model
ERROR 02-08 09:31:59 core.py:210]     output = self.collective_rpc("execute_model",
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 133, in collective_rpc
ERROR 02-08 09:31:59 core.py:210]     raise e
ERROR 02-08 09:31:59 core.py:210]   File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 122, in collective_rpc
ERROR 02-08 09:31:59 core.py:210]     raise result

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[YorkSu]

Encountered the same problem. But I'm using vllm serve to deploy DeepSeek-R1-AWQ.

Environment

• Image: cuda:12.6.0-cudnn-devel-ubuntu22.04
• GPUs: A800 x 8
• Python 3.10
• vLLM 0.7.2
• torch 2.5.1

Deploy scripts

#!/bin/bash

export VLLM_USE_V1=1
export VLLM_CUDA_MEM_ALIGN_KV_CACHE=1
export VLLM_USE_TRITON_AWQ=1

vllm serve \
/path/to/DeepSeek-R1-AWQ \
--served-model-name "DeepSeek-R1" \
--trust-remote-code \
--host "0.0.0.0" \
--port 20000 \
--max-model-len 16384 \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size 8 \
--swap-space 32 \
--quantization moe_wna16 \
--enforce-eager \
--enable-reasoning \
--reasoning-parser deepseek_r1

[YangZeyu95]

What's the difference between v0 and v1?

[YorkSu]

After reading the source code, I have the following findings.

V0 Implement

Function get_token_bin_counts_and_mask(vllm.model_executor.layers.utils.py, line8), the shape of bin_counts is [num_seq, vocab_size + 1]
The num_seqs is logits.shape[0]
The tokens is sampling_tensors.prompt_tokens, created from SamplingTensors.from_sampling_metadata. The prompt_tokens is not empty only when do_penalties is True, and the shape of prompt_tokens is [len(sampling_metadata.seq_groups), ...]
The sampling_metadata is vllm.model_executor.sampling_metadata.SamplingMetadata

V1 Implement
The sampling_metadata is vllm.v1.sample.metadata.SamplingMetadata
The tokens is sampling_metadata.prompt_token_ids, which created from vllm.v1.worker.gpu_input_batch.InputBatch._make_prompt_token_ids_tensor. The shape of it is [InputBatch.num_reqs, max_prompt_len]
The InputBatch.num_reqs is property, equals len(InputBatch.req_id_to_index)

Could it be that the inconsistency between logits.shape[0] and len(InputBatch.req_id_to_index) is causing this issue?
More precisely, could it be that during the sampling phase, new requests are entering, leading to inconsistency in seqs?

[YorkSu]

I can apply a temporary fix by directly modifying the code to address this error. However, I am uncertain about the correctness of this approach, so this suggestion is provided for reference only.

vllm/vllm/v1/sample/ops/penalties.py

Line 28 in 3243158

def apply_all_penalties(

def apply_all_penalties(
    logits: torch.Tensor,
    prompt_token_ids: torch.Tensor,
    presence_penalties: torch.Tensor,
    frequency_penalties: torch.Tensor,
    repetition_penalties: torch.Tensor,
    output_token_ids: List[List[int]],
) -> torch.Tensor:
    """
    Applies presence, frequency and repetition penalties to the logits.
    """
    num_seqs, vocab_size = logits.shape
    output_tokens_t = _convert_to_tensors(output_token_ids, vocab_size,
                                          logits.device)
    return apply_penalties(logits, prompt_token_ids[:num_seqs], output_tokens_t,
                           presence_penalties, frequency_penalties,
                           repetition_penalties)

[robertgshaw2-redhat]

Thanks for the report

[robertgshaw2-redhat]

Can you share a working repro script:

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3.1-8B-Instruct"
stop_token_ids = [11469, 11]
llm = LLM(model=model_path,
          tensor_parallel_size=1,
          trust_remote_code=True,
          gpu_memory_utilization=0.8)
tokenizer = AutoTokenizer.from_pretrained(model_path)

prompts = [
    "Hello. Who is the best company supporting vLLM?",
    "Who is the market leader in enterprise Linux and Kubernetes?"
]
input_texts = []
for one_prompt in prompts:
    messages = [
        {"role": "system", "content": ""},
        {"role": "user", "content": one_prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    input_texts.append(text)

sampling_params = SamplingParams(stop_token_ids=stop_token_ids,
                                 max_tokens=100,
                                 temperature=0)
outputs = llm.generate(input_texts, sampling_params, use_tqdm=True)
print(outputs)

[JaheimLee]

same problem when there are multiple requests at the same time.

[weekend-ai]

we observed a same issue when deploy llama 3.3
ERROR 02-17 01:53:55 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 203, in run_engine_core
ERROR 02-17 01:53:55 core.py:210] engine_core.run_busy_loop()
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 243, in run_busy_loop
ERROR 02-17 01:53:55 core.py:210] outputs = self.step()
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 129, in step
ERROR 02-17 01:53:55 core.py:210] output = self.model_executor.execute_model(scheduler_output)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 77, in execute_model
ERROR 02-17 01:53:55 core.py:210] output = self.collective_rpc("execute_model",
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 51, in collective_rpc
ERROR 02-17 01:53:55 core.py:210] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2220, in run_method
ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
ERROR 02-17 01:53:55 core.py:210] output = self.model_runner.execute_model(scheduler_output)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
ERROR 02-17 01:53:55 core.py:210] sampler_output = self.model.sample(
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 557, in sample
ERROR 02-17 01:53:55 core.py:210] next_tokens = self.sampler(logits, sampling_metadata)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 02-17 01:53:55 core.py:210] return self._call_impl(*args, **kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 02-17 01:53:55 core.py:210] return forward_call(*args, **kwargs)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 46, in forward
ERROR 02-17 01:53:55 core.py:210] logits = self.apply_penalties(logits, sampling_metadata)
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
ERROR 02-17 01:53:55 core.py:210] logits = apply_all_penalties(
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
ERROR 02-17 01:53:55 core.py:210] return apply_penalties(logits, prompt_token_ids, output_tokens_t,
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
ERROR 02-17 01:53:55 core.py:210] , prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
ERROR 02-17 01:53:55 core.py:210] bin_counts.scatter_add(1, tokens, torch.ones_like(tokens))
ERROR 02-17 01:53:55 core.py:210] RuntimeError: Expected index [4, 1193] to be smaller than self [3, 128257] apart from dimension 1 and to be smaller size than src [4, 1193]

[Fy-]

same here (with multiple requests at the same time)

[weekend-ai]

@Fy- I think the issue is fixed with latest code. #13076 I bake the main branch and no issues

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!