[Bug]: Hang During CUDA Graph Capture on ROCM in 0.19

[depuhitv]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.1 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e)
CMake version                : version 3.31.10
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+git8514f05
Is debug build               : False
CUDA used to build PyTorch   : N/A
ROCM used to build PyTorch   : 7.2.53211

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.19.10-200.fc43.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   :
GPU models and configuration :  (gfx1201)
Nvidia driver version        : Could not collect
cuDNN version                : Could not collect
HIP runtime version          : 7.2.53211
MIOpen runtime version       : 3.5.1
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  32
On-line CPU(s) list:                     0-31
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen 9 9950X3D 16-Core Processor
CPU family:                              26
Model:                                   68
Thread(s) per core:                      2
Core(s) per socket:                      16
Socket(s):                               1
Stepping:                                0
Frequency boost:                         enabled
CPU max MHz:                             5756.4521
CPU min MHz:                             624.1940
BogoMIPS:                                8583.31
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze
Virtualization:                          AMD-V
L1d cache:                               768 KiB (16 instances)
L1i cache:                               512 KiB (16 instances)
L2 cache:                                16 MiB (16 instances)
L3 cache:                                128 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-31
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; IBPB on VMEXIT only
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB on VMEXIT

==============================
Versions of relevant libraries
==============================
[pip3] conch-triton-kernels==1.2.1
[pip3] numpy==2.1.3
[pip3] onnx==1.19.0
[pip3] onnx-ir==0.2.0
[pip3] onnxscript==0.6.2
[pip3] onnxslim==0.1.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+git8514f05
[pip3] torchaudio==2.9.0+eaa9e4e
[pip3] torchvision==0.24.1+d801a34
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[pip3] triton_kernels==1.0.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : 7.2.53211-e1a6bc5663
vLLM Version                 : 0.19.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
  ============================ ROCm System Management Interface ============================
================================ Weight between two GPUs =================================
       GPU0         GPU1
GPU0   0            40
GPU1   40           0

================================= Hops between two GPUs ==================================
       GPU0         GPU1
GPU0   0            2
GPU1   2            0

=============================== Link Type between two GPUs ===============================
       GPU0         GPU1
GPU0   0            PCIE
GPU1   PCIE         0

======================================= Numa Nodes =======================================
GPU[0]          : (Topology) Numa Node: 0
GPU[0]          : (Topology) Numa Affinity: -1
GPU[1]          : (Topology) Numa Node: 0
GPU[1]          : (Topology) Numa Affinity: -1
================================== End of ROCm SMI Log ===================================

==============================
     Environment Variables
==============================
PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
OMP_NUM_THREADS=8
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

Attempting to serve any model (tested with devstral small 2) fails and stops at
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████ | 36/51 [00:22<00:01, 11.70it/s]
After this message there is a continuous shm_broadcast message but the load never proceeds.
[shm_broadcast.py:681] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

I am using the latest (v0.19.0) docker image and the command used was

docker run --rm --group-add=video --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri --ipc=host --shm-size=16gb --env "OMP_NUM_THREADS=8" -v /mnt/LinuxStorage/vllm:/root/.cache/huggingface  --env "HF_TOKEN=token" --env "HIP_VISIBLE_DEVICES=0,1"  -p 8000:8000 vllm/vllm-openai-rocm:latest --model mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice  --max-model-len 128000 --quantization fp8 --tensor-parallel-size 2 --gpu_memory_utilization=0.9 --kv_cache_dtype="fp8_e4m3"  --enable-prefix-caching

using --enfore-eager only loaded the model, but the requests made to the api would timeout with the same shm_broadcast.py logs message.

The full logs are

WARNING 04-05 03:14:16 [argparse_utils.py:191] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.0
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]   █▄█▀ █     █     █     █  model   mistralai/Devstral-Small-2-24B-Instruct-2512
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:299]
(APIServer pid=1) INFO 04-05 03:14:16 [utils.py:233] non-default args: {'model_tag': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'max_model_len': 128000, 'quantization': 'fp8', 'tensor_parallel_size': 2, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True}
Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00,  5.09it/s]
(APIServer pid=1) INFO 04-05 03:14:18 [config.py:288] Inferred from consolidated*.safetensors files torch.bfloat16 dtype.
(APIServer pid=1) INFO 04-05 03:14:23 [model.py:549] Resolved architecture: PixtralForConditionalGeneration
(APIServer pid=1) INFO 04-05 03:14:23 [model.py:1678] Using max model len 128000
(APIServer pid=1) [aiter] import [module_aiter_enum] under /usr/local/lib/python3.12/dist-packages/aiter/jit/module_aiter_enum.so
(APIServer pid=1) INFO 04-05 03:14:24 [cache.py:227] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
(APIServer pid=1) INFO 04-05 03:14:24 [vllm.py:790] Asynchronous scheduling is enabled.
(EngineCore pid=104) INFO 04-05 03:14:29 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='mistralai/Devstral-Small-2-24B-Instruct-2512', speculative_config=None, tokenizer='mistralai/Devstral-Small-2-24B-Instruct-2512', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=mistralai/Devstral-Small-2-24B-Instruct-2512, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=104) INFO 04-05 03:14:29 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.2 (local), world_size=2, local_world_size=2
(Worker pid=136) INFO 04-05 03:14:34 [parallel_state.py:1400] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:52449 backend=nccl
(Worker pid=137) INFO 04-05 03:14:34 [parallel_state.py:1400] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:52449 backend=nccl
(Worker pid=136) INFO 04-05 03:14:34 [pynccl.py:111] vLLM is using nccl==2.27.7
(Worker pid=136) INFO 04-05 03:14:39 [parallel_state.py:1716] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker_TP0 pid=136) INFO 04-05 03:14:39 [gpu_model_runner.py:4735] Starting to load model mistralai/Devstral-Small-2-24B-Instruct-2512...
(Worker_TP0 pid=136) INFO 04-05 03:14:40 [vllm.py:790] Asynchronous scheduling is enabled.
(Worker_TP0 pid=136) INFO 04-05 03:14:40 [__init__.py:261] Selected PerTensorTorchFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP0 pid=136) INFO 04-05 03:14:40 [rocm.py:496] Using ROCM_ATTN backend out of potential backends: ['ROCM_ATTN', 'TRITON_ATTN'].
(Worker_TP0 pid=136) WARNING 04-05 03:14:40 [compilation.py:1220] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:01<00:01,  1.81s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.25s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.34s/it]
(Worker_TP0 pid=136)
(Worker_TP0 pid=136) INFO 04-05 03:14:44 [default_loader.py:384] Loading weights took 2.75 seconds
(Worker_TP0 pid=136) WARNING 04-05 03:14:44 [kv_cache.py:94] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=136) WARNING 04-05 03:14:44 [kv_cache.py:108] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=136) INFO 04-05 03:14:45 [gpu_model_runner.py:4820] Model loading took 15.6 GiB memory and 4.872822 seconds
(Worker_TP0 pid=136) INFO 04-05 03:14:46 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 3025 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=136) INFO 04-05 03:14:51 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/bfe186540c/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=136) INFO 04-05 03:14:51 [backends.py:1111] Dynamo bytecode transform time: 4.81 s
(Worker_TP0 pid=136) INFO 04-05 03:14:57 [backends.py:372] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=136) INFO 04-05 03:15:02 [backends.py:390] Compiling a graph for compile range (1, 2048) takes 10.74 s
(Worker_TP0 pid=136) INFO 04-05 03:15:04 [decorators.py:640] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4c2881b4ad445e3cc0159e01cf6e6ac0f7b584db6faa9f898ccdf234cde36d53/rank_0_0/model
(Worker_TP0 pid=136) INFO 04-05 03:15:04 [monitor.py:48] torch.compile took 17.40 s in total
(Worker_TP0 pid=136) INFO 04-05 03:15:05 [monitor.py:76] Initial profiling/warmup run took 0.93 s
(Worker_TP0 pid=136) INFO 04-05 03:15:08 [gpu_worker.py:436] Available KV cache memory: 11.87 GiB
(EngineCore pid=104) INFO 04-05 03:15:08 [kv_cache_utils.py:1319] GPU KV cache size: 311,264 tokens
(EngineCore pid=104) INFO 04-05 03:15:08 [kv_cache_utils.py:1324] Maximum concurrency for 128,000 tokens per request: 2.43x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████   | 36/51 [00:22<00:01, 11.70it/s](EngineCore pid=104) INFO 04-05 03:16:09 [shm_broadcast.py:681] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=104) INFO 04-05 03:17:09 [shm_broadcast.py:681] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm for ROCm-related issue

[github-actions]

Hi @depuhitv, thanks for reporting this ROCm issue!

To help us investigate, please make sure the following information is included:

• A minimal reproducer (code snippet or script that triggers the issue)
• The full error message or traceback
• How you installed vLLM (Docker image name, pip install, or build from source steps)
• The command you used to launch vLLM (e.g., vllm serve ... or the Python script)
• Your GPU model and GFX architecture (e.g., MI300X / gfx942) — run rocminfo | grep gfx

Please provide any unchecked items above. This will help us reproduce and resolve the issue faster. Thank you!

[depuhitv]

This seems to be a regression since the same run command works as expected with v0.18.1

WARNING 04-05 03:04:25 [gpt_oss_triton_kernels_moe.py:56] Using legacy triton_kernels on ROCm
WARNING 04-05 03:04:27 [argparse_utils.py:193] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.18.1
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]   █▄█▀ █     █     █     █  model   mistralai/Devstral-Small-2-24B-Instruct-2512
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:297]
(APIServer pid=1) INFO 04-05 03:04:27 [utils.py:233] non-default args: {'model_tag': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'max_model_len': 128000, 'quantization': 'fp8', 'tensor_parallel_size': 2, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True}
Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00,  2.63it/s]
(APIServer pid=1) INFO 04-05 03:04:29 [config.py:279] Inferred from consolidated*.safetensors files torch.bfloat16 dtype.
(APIServer pid=1) INFO 04-05 03:04:34 [model.py:533] Resolved architecture: PixtralForConditionalGeneration
(APIServer pid=1) INFO 04-05 03:04:34 [model.py:1582] Using max model len 128000
(APIServer pid=1) [aiter] start build [module_aiter_enum] under /usr/local/lib/python3.12/dist-packages/aiter/jit/build/module_aiter_enum
(APIServer pid=1) [aiter] finish build [module_aiter_enum], cost 6.3s
(APIServer pid=1) [aiter] import [module_aiter_enum] under /usr/local/lib/python3.12/dist-packages/aiter/jit/module_aiter_enum.so
(APIServer pid=1) INFO 04-05 03:04:41 [cache.py:212] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
(APIServer pid=1) INFO 04-05 03:04:41 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=1) INFO 04-05 03:04:41 [vllm.py:775] Asynchronous scheduling is enabled.
WARNING 04-05 03:04:46 [gpt_oss_triton_kernels_moe.py:56] Using legacy triton_kernels on ROCm
(EngineCore pid=366) INFO 04-05 03:04:47 [core.py:103] Initializing a V1 LLM engine (v0.18.1) with config: model='mistralai/Devstral-Small-2-24B-Instruct-2512', speculative_config=None, tokenizer='mistralai/Devstral-Small-2-24B-Instruct-2512', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=mistralai/Devstral-Small-2-24B-Instruct-2512, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=366) INFO 04-05 03:04:47 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.2 (local), world_size=2, local_world_size=2
WARNING 04-05 03:04:50 [gpt_oss_triton_kernels_moe.py:56] Using legacy triton_kernels on ROCm
WARNING 04-05 03:04:50 [gpt_oss_triton_kernels_moe.py:56] Using legacy triton_kernels on ROCm
(Worker pid=398) INFO 04-05 03:04:52 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:34289 backend=nccl
(Worker pid=397) INFO 04-05 03:04:52 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:34289 backend=nccl
(Worker pid=397) INFO 04-05 03:04:52 [pynccl.py:111] vLLM is using nccl==2.26.6
(Worker pid=397) INFO 04-05 03:04:55 [parallel_state.py:1717] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker pid=398) INFO 04-05 03:04:55 [parallel_state.py:1717] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A
(Worker_TP0 pid=397) INFO 04-05 03:04:55 [gpu_model_runner.py:4481] Starting to load model mistralai/Devstral-Small-2-24B-Instruct-2512...
(Worker_TP0 pid=397) INFO 04-05 03:04:55 [vllm.py:775] Asynchronous scheduling is enabled.
(Worker_TP0 pid=397) INFO 04-05 03:04:55 [__init__.py:255] Selected PerTensorTorchFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP1 pid=398) INFO 04-05 03:04:55 [vllm.py:775] Asynchronous scheduling is enabled.
(Worker_TP0 pid=397) INFO 04-05 03:04:56 [rocm.py:496] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN'].
(Worker_TP0 pid=397) WARNING 04-05 03:04:56 [compilation.py:1136] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
(Worker_TP1 pid=398) WARNING 04-05 03:04:56 [compilation.py:1136] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:02<00:02,  2.09s/it]
(Worker_TP1 pid=398) WARNING 04-05 03:05:00 [kv_cache.py:94] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP1 pid=398) WARNING 04-05 03:05:00 [kv_cache.py:108] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.27s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.39s/it]
(Worker_TP0 pid=397)
(Worker_TP0 pid=397) INFO 04-05 03:05:00 [default_loader.py:384] Loading weights took 2.84 seconds
(Worker_TP0 pid=397) WARNING 04-05 03:05:00 [kv_cache.py:94] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=397) WARNING 04-05 03:05:00 [kv_cache.py:108] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=397) INFO 04-05 03:05:01 [gpu_model_runner.py:4566] Model loading took 15.59 GiB memory and 5.320500 seconds
(Worker_TP0 pid=397) INFO 04-05 03:05:02 [gpu_model_runner.py:5488] Encoder cache will be initialized with a budget of 3025 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP1 pid=398) INFO 04-05 03:05:02 [gpu_model_runner.py:5488] Encoder cache will be initialized with a budget of 3025 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=397) INFO 04-05 03:05:06 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/26fe58f3f9/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=397) INFO 04-05 03:05:06 [backends.py:1048] Dynamo bytecode transform time: 4.25 s
(Worker_TP1 pid=398) INFO 04-05 03:05:12 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=397) INFO 04-05 03:05:12 [backends.py:371] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=397) INFO 04-05 03:05:18 [backends.py:387] Compiling a graph for compile range (1, 2048) takes 10.93 s
(Worker_TP0 pid=397) INFO 04-05 03:05:20 [monitor.py:48] torch.compile and initial profiling/warmup run together took 17.47 s in total
(Worker_TP0 pid=397) INFO 04-05 03:05:23 [gpu_worker.py:456] Available KV cache memory: 11.84 GiB
(EngineCore pid=366) INFO 04-05 03:05:23 [kv_cache_utils.py:1316] GPU KV cache size: 310,352 tokens
(EngineCore pid=366) INFO 04-05 03:05:23 [kv_cache_utils.py:1321] Maximum concurrency for 128,000 tokens per request: 2.42x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 10.28it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 35/35 [00:06<00:00,  5.74it/s]
(Worker_TP0 pid=397) INFO 04-05 03:05:35 [gpu_model_runner.py:5746] Graph capturing finished in 12 secs, took 0.48 GiB
(EngineCore pid=366) INFO 04-05 03:05:35 [core.py:281] init engine (profile, create kv cache, warmup model) took 33.86 seconds
(EngineCore pid=366) INFO 04-05 03:05:37 [vllm.py:775] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 04-05 03:05:37 [api_server.py:576] Supported tasks: ['generate']
(APIServer pid=1) INFO 04-05 03:05:37 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) WARNING 04-05 03:05:37 [model.py:1376] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.15}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 04-05 03:05:37 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 04-05 03:05:37 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 04-05 03:05:38 [base.py:216] Multi-modal warmup completed in 0.077s
(APIServer pid=1) INFO 04-05 03:05:38 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 04-05 03:05:38 [api_server.py:580] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:37] Available routes are:
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 04-05 03:05:38 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.

[vllmellm]

@depuhitv It could be related to this issue. #38587

Can you try the solution in the issue?

To see if downgrading rccl works
wget -q " https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb " -O /tmp/rccl.deb && sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix

[depuhitv]

@vllmellm That did work for me.

root@e1c41c085b7e:/app# wget -q "https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb" -O /tmp/rccl.deb
root@e1c41c085b7e:/app# sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix
root@e1c41c085b7e:/app# vllm serve --model mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice  --max-model-len 128000 --quantization
 fp8 --tensor-parallel-size 2 --gpu_memory_utilization=0.9 --kv_cache_dtype="fp8_e4m3"  --enable-prefix-caching
WARNING 04-05 07:07:22 [argparse_utils.py:191] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.0
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]   █▄█▀ █     █     █     █  model   mistralai/Devstral-Small-2-24B-Instruct-2512
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:299]
(APIServer pid=134) INFO 04-05 07:07:22 [utils.py:233] non-default args: {'model_tag': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'max_model_len': 128000, 'quantization': 'fp8', 'tensor_parallel_size': 2, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True}
Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.41it/s]
(APIServer pid=134) INFO 04-05 07:07:24 [config.py:288] Inferred from consolidated*.safetensors files torch.bfloat16 dtype.
(APIServer pid=134) INFO 04-05 07:07:29 [model.py:549] Resolved architecture: PixtralForConditionalGeneration
(APIServer pid=134) INFO 04-05 07:07:29 [model.py:1678] Using max model len 128000
(APIServer pid=134) [aiter] import [module_aiter_enum] under /usr/local/lib/python3.12/dist-packages/aiter/jit/module_aiter_enum.so
(APIServer pid=134) INFO 04-05 07:07:30 [cache.py:227] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
(APIServer pid=134) INFO 04-05 07:07:30 [vllm.py:790] Asynchronous scheduling is enabled.
(EngineCore pid=232) INFO 04-05 07:07:35 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='mistralai/Devstral-Small-2-24B-Instruct-2512', speculative_config=None, tokenizer='mistralai/Devstral-Small-2-24B-Instruct-2512', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=mistralai/Devstral-Small-2-24B-Instruct-2512, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=232) INFO 04-05 07:07:35 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.2 (local), world_size=2, local_world_size=2
(Worker pid=264) INFO 04-05 07:07:40 [parallel_state.py:1400] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:34259 backend=nccl
(Worker pid=265) INFO 04-05 07:07:40 [parallel_state.py:1400] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:34259 backend=nccl
(Worker pid=264) INFO 04-05 07:07:40 [pynccl.py:111] vLLM is using nccl==2.27.7
(Worker pid=264) INFO 04-05 07:07:47 [parallel_state.py:1716] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker_TP0 pid=264) INFO 04-05 07:07:47 [gpu_model_runner.py:4735] Starting to load model mistralai/Devstral-Small-2-24B-Instruct-2512...
(Worker_TP0 pid=264) INFO 04-05 07:07:48 [vllm.py:790] Asynchronous scheduling is enabled.
(Worker_TP0 pid=264) INFO 04-05 07:07:48 [__init__.py:261] Selected PerTensorTorchFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP0 pid=264) INFO 04-05 07:07:48 [rocm.py:496] Using ROCM_ATTN backend out of potential backends: ['ROCM_ATTN', 'TRITON_ATTN'].
(Worker_TP0 pid=264) WARNING 04-05 07:07:48 [compilation.py:1220] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:02<00:02,  2.02s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.22s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.34s/it]
(Worker_TP0 pid=264)
(Worker_TP0 pid=264) INFO 04-05 07:07:52 [default_loader.py:384] Loading weights took 2.75 seconds
(Worker_TP0 pid=264) WARNING 04-05 07:07:53 [kv_cache.py:94] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=264) WARNING 04-05 07:07:53 [kv_cache.py:108] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=264) INFO 04-05 07:07:53 [gpu_model_runner.py:4820] Model loading took 15.6 GiB memory and 5.266937 seconds
(Worker_TP0 pid=264) INFO 04-05 07:07:54 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 3025 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=264) INFO 04-05 07:07:59 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/bfe186540c/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=264) INFO 04-05 07:07:59 [backends.py:1111] Dynamo bytecode transform time: 4.41 s
(Worker_TP0 pid=264) INFO 04-05 07:08:04 [backends.py:372] Cache the graph of compile range (1, 2048) for later use
(Worker_TP0 pid=264) INFO 04-05 07:08:09 [backends.py:390] Compiling a graph for compile range (1, 2048) takes 10.61 s
(Worker_TP0 pid=264) INFO 04-05 07:08:11 [decorators.py:640] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4c2881b4ad445e3cc0159e01cf6e6ac0f7b584db6faa9f898ccdf234cde36d53/rank_0_0/model
(Worker_TP0 pid=264) INFO 04-05 07:08:11 [monitor.py:48] torch.compile took 16.88 s in total
(Worker_TP0 pid=264) INFO 04-05 07:08:12 [monitor.py:76] Initial profiling/warmup run took 0.90 s
(Worker_TP0 pid=264) INFO 04-05 07:08:15 [gpu_worker.py:436] Available KV cache memory: 11.87 GiB
(EngineCore pid=232) INFO 04-05 07:08:15 [kv_cache_utils.py:1319] GPU KV cache size: 311,264 tokens
(EngineCore pid=232) INFO 04-05 07:08:15 [kv_cache_utils.py:1324] Maximum concurrency for 128,000 tokens per request: 2.43x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:04<00:00, 10.68it/s]
Capturing CUDA graphs (decode, FULL):   0%|                                                                                                                 | 0/35 [00:00<?, ?it/s](Worker_TP0 pid=264) WARNING 04-05 07:08:21 [chunked_prefill_paged_decode.py:398] Cannot use ROCm custom paged attention kernel, falling back to Triton implementation.
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:03<00:00, 10.93it/s]
(Worker_TP0 pid=264) INFO 04-05 07:08:24 [gpu_model_runner.py:6046] Graph capturing finished in 9 secs, took 3.69 GiB
(EngineCore pid=232) INFO 04-05 07:08:24 [core.py:283] init engine (profile, create kv cache, warmup model) took 30.87 seconds
(EngineCore pid=232) INFO 04-05 07:08:26 [vllm.py:790] Asynchronous scheduling is enabled.
(APIServer pid=134) INFO 04-05 07:08:26 [api_server.py:590] Supported tasks: ['generate']
(APIServer pid=134) INFO 04-05 07:08:26 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=134) WARNING 04-05 07:08:26 [model.py:1435] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.15}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=134) INFO 04-05 07:08:27 [base.py:231] Multi-modal warmup completed in 0.078s
(APIServer pid=134) INFO 04-05 07:08:28 [api_server.py:594] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:37] Available routes are:
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=134) INFO 04-05 07:08:28 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=134) INFO:     Started server process [134]
(APIServer pid=134) INFO:     Waiting for application startup.
(APIServer pid=134) INFO:     Application startup complete.
(APIServer pid=134) INFO:     192.168.0.71:35050 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=134) INFO:     192.168.0.71:35058 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=134) INFO:     192.168.0.71:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=134) INFO:     192.168.0.71:43848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=134) INFO 04-05 07:08:58 [loggers.py:259] Engine 000: Avg prompt throughput: 31.4 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[depuhitv]

I also tried with the 7.2 version and that works as well,

except both 7.1.1 and 7.2 have a warning about falling back to Triton paged attention from rocm which you dont get on v0.18.1.

(Worker_TP0 pid=901) WARNING 04-05 08:17:02 [chunked_prefill_paged_decode.py:398] Cannot use ROCm custom paged attention kernel, falling back to Triton implementation.

wget -q "https://repo.radeon.com/rocm/apt/7.2/pool/main/r/rccl/rccl_2.27.7.70200-43~22.04_amd64.deb" -O /tmp/rccl.deb && sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.2.0/lib/librccl.so.1.0.70200 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70200 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix

root@e1c41c085b7e:/app# vllm serve --model mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice  --max-model-len 128000 --quantization fp8 --tensor-parallel-size 2 --gpu_memory_utilization=0.9 --kv_cache_dtype="fp8_e4m3"  --enable-prefix-caching
WARNING 04-05 08:16:28 [argparse_utils.py:191] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.0
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]   █▄█▀ █     █     █     █  model   mistralai/Devstral-Small-2-24B-Instruct-2512
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:299]
(APIServer pid=798) INFO 04-05 08:16:28 [utils.py:233] non-default args: {'model_tag': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'max_model_len': 128000, 'quantization': 'fp8', 'tensor_parallel_size': 2, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True}
Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.84it/s]
(APIServer pid=798) INFO 04-05 08:16:30 [config.py:288] Inferred from consolidated*.safetensors files torch.bfloat16 dtype.
(APIServer pid=798) INFO 04-05 08:16:31 [model.py:549] Resolved architecture: PixtralForConditionalGeneration
(APIServer pid=798) INFO 04-05 08:16:31 [model.py:1678] Using max model len 128000
(APIServer pid=798) [aiter] import [module_aiter_enum] under /usr/local/lib/python3.12/dist-packages/aiter/jit/module_aiter_enum.so
(APIServer pid=798) INFO 04-05 08:16:31 [cache.py:227] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
(APIServer pid=798) INFO 04-05 08:16:31 [vllm.py:790] Asynchronous scheduling is enabled.
(EngineCore pid=869) INFO 04-05 08:16:36 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='mistralai/Devstral-Small-2-24B-Instruct-2512', speculative_config=None, tokenizer='mistralai/Devstral-Small-2-24B-Instruct-2512', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=mistralai/Devstral-Small-2-24B-Instruct-2512, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=869) INFO 04-05 08:16:36 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.2 (local), world_size=2, local_world_size=2
(Worker pid=901) INFO 04-05 08:16:41 [parallel_state.py:1400] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:32915 backend=nccl
(Worker pid=902) INFO 04-05 08:16:41 [parallel_state.py:1400] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:32915 backend=nccl
(Worker pid=901) INFO 04-05 08:16:41 [pynccl.py:111] vLLM is using nccl==2.27.7
(Worker pid=901) INFO 04-05 08:16:46 [parallel_state.py:1716] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker_TP0 pid=901) INFO 04-05 08:16:46 [gpu_model_runner.py:4735] Starting to load model mistralai/Devstral-Small-2-24B-Instruct-2512...
(Worker_TP0 pid=901) INFO 04-05 08:16:46 [vllm.py:790] Asynchronous scheduling is enabled.
(Worker_TP0 pid=901) INFO 04-05 08:16:46 [__init__.py:261] Selected PerTensorTorchFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP0 pid=901) INFO 04-05 08:16:47 [rocm.py:496] Using ROCM_ATTN backend out of potential backends: ['ROCM_ATTN', 'TRITON_ATTN'].
(Worker_TP0 pid=901) WARNING 04-05 08:16:47 [compilation.py:1220] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:02<00:02,  2.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.22s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.35s/it]
(Worker_TP0 pid=901)
(Worker_TP0 pid=901) INFO 04-05 08:16:51 [default_loader.py:384] Loading weights took 2.75 seconds
(Worker_TP0 pid=901) WARNING 04-05 08:16:51 [kv_cache.py:94] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=901) WARNING 04-05 08:16:51 [kv_cache.py:108] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=901) INFO 04-05 08:16:52 [gpu_model_runner.py:4820] Model loading took 15.6 GiB memory and 5.032809 seconds
(Worker_TP0 pid=901) INFO 04-05 08:16:52 [gpu_model_runner.py:5753] Encoder cache will be initialized with a budget of 3025 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=901) INFO 04-05 08:16:55 [backends.py:1051] Using cache directory: /root/.cache/vllm/torch_compile_cache/bfe186540c/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=901) INFO 04-05 08:16:55 [backends.py:1111] Dynamo bytecode transform time: 1.85 s
(Worker_TP0 pid=901) INFO 04-05 08:16:56 [backends.py:285] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 0.736 s
(Worker_TP0 pid=901) INFO 04-05 08:16:56 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4c2881b4ad445e3cc0159e01cf6e6ac0f7b584db6faa9f898ccdf234cde36d53/rank_0_0/model
(Worker_TP0 pid=901) INFO 04-05 08:16:56 [monitor.py:48] torch.compile took 2.79 s in total
(Worker_TP1 pid=902) INFO 04-05 08:16:56 [decorators.py:303] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4c2881b4ad445e3cc0159e01cf6e6ac0f7b584db6faa9f898ccdf234cde36d53/rank_1_0/model
(Worker_TP0 pid=901) INFO 04-05 08:16:56 [monitor.py:76] Initial profiling/warmup run took 0.22 s
(Worker_TP0 pid=901) INFO 04-05 08:16:57 [gpu_worker.py:436] Available KV cache memory: 11.94 GiB
(EngineCore pid=869) INFO 04-05 08:16:57 [kv_cache_utils.py:1319] GPU KV cache size: 313,056 tokens
(EngineCore pid=869) INFO 04-05 08:16:57 [kv_cache_utils.py:1324] Maximum concurrency for 128,000 tokens per request: 2.45x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:04<00:00, 10.70it/s]
Capturing CUDA graphs (decode, FULL):   0%|                                                                                                                 | 0/35 [00:00<?, ?it/s](Worker_TP0 pid=901) WARNING 04-05 08:17:02 [chunked_prefill_paged_decode.py:398] Cannot use ROCm custom paged attention kernel, falling back to Triton implementation.
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:02<00:00, 12.67it/s]
(Worker_TP0 pid=901) INFO 04-05 08:17:05 [gpu_model_runner.py:6046] Graph capturing finished in 8 secs, took 3.74 GiB
(EngineCore pid=869) INFO 04-05 08:17:05 [core.py:283] init engine (profile, create kv cache, warmup model) took 13.27 seconds
(EngineCore pid=869) INFO 04-05 08:17:07 [vllm.py:790] Asynchronous scheduling is enabled.
(APIServer pid=798) INFO 04-05 08:17:07 [api_server.py:590] Supported tasks: ['generate']
(APIServer pid=798) INFO 04-05 08:17:07 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=798) WARNING 04-05 08:17:07 [model.py:1435] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.15}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=798) INFO 04-05 08:17:08 [base.py:231] Multi-modal warmup completed in 0.076s
(APIServer pid=798) INFO 04-05 08:17:08 [api_server.py:594] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:37] Available routes are:
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=798) INFO 04-05 08:17:08 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=798) INFO:     Started server process [798]
(APIServer pid=798) INFO:     Waiting for application startup.
(APIServer pid=798) INFO:     Application startup complete.
(APIServer pid=798) INFO:     192.168.0.71:39444 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=798) INFO:     192.168.0.71:56598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=798) INFO 04-05 08:21:09 [loggers.py:259] Engine 000: Avg prompt throughput: 2.0 tokens/s, Avg generation throughput: 1.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[cjrolo]

@depuhitv It could be related to this issue. #38587

Can you try the solution in the issue?

To see if downgrading rccl works
wget -q " https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb " -O /tmp/rccl.deb && sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix

It doesn't work for me. I've tried different models, different env flags. It always get stuck in graph decode. TP4, R9700.

It always get stuck at 9%

(EngineCore pid=117) INFO 04-22 17:36:38 [kv_cache_utils.py:1324] Maximum concurrency for 131,072 tokens per request: 14.88x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 3/33 [00:01<00:09, 3.26it/s]

What information can I provide?

[depuhitv]

@depuhitv It could be related to this issue. #38587
Can you try the solution in the issue?

To see if downgrading rccl works
wget -q " https://repo.radeon.com/rocm/apt/7.1.1/pool/main/r/rccl/rccl_2.27.7.70101-38~22.04_amd64.deb " -O /tmp/rccl.deb && sudo mkdir -p /tmp/rccl_fix && sudo dpkg -x /tmp/rccl.deb /tmp/rccl_fix && sudo rm -f /opt/rocm/lib/librccl.so.1.0.70201 /opt/rocm/lib/librccl.so.1 /opt/rocm/lib/librccl.so && sudo cp /tmp/rccl_fix/opt/rocm-7.1.1/lib/librccl.so.1.0.70101 /opt/rocm/lib/ && sudo ln -sf librccl.so.1.0.70101 /opt/rocm/lib/librccl.so.1 && sudo ln -sf librccl.so.1 /opt/rocm/lib/librccl.so && sudo ln -sf librocm_smi64.so.1 /opt/rocm/lib/librocm_smi64.so.7 2>/dev/null || true && sudo ldconfig && sudo rm -rf /tmp/rccl.deb /tmp/rccl_fix

It doesn't work for me. I've tried different models, different env flags. It always get stuck in graph decode. TP4, R9700.

It always get stuck at 9%

(EngineCore pid=117) INFO 04-22 17:36:38 [kv_cache_utils.py:1324] Maximum concurrency for 131,072 tokens per request: 14.88x Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 3/33 [00:01<00:09, 3.26it/s]

What information can I provide?

If you’d like me to test it out on my setup with TP2 on R9700, please share the entire launch command with flags and env variables so I could replicate it and see if I have the same issue as you.

If it’s for the dev’s probably the same information as the original bug report would be useful for them.

I have personally reverted to the 0.18.1 version for most of my usage (with transformer upgrades for more recent models) until I see the fix deployed since this is pretty much a showstopper.