[Bug]: V1 engine hangs then crashes with "No available shared memory broadcast block found / RPC call to sample_tokens timed out" under chat completions burst load on Qwen3.5-122B-A10B

[natech-persona]

Your current environment

The output of python collect_env.py

uv is set
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.12 (main, Mar  3 2026, 11:56:32) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.0-164-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H200
GPU 1: NVIDIA H200
GPU 2: NVIDIA H200
GPU 3: NVIDIA H200
GPU 4: NVIDIA H200
GPU 5: NVIDIA H200
GPU 6: NVIDIA H200
GPU 7: NVIDIA H200

Nvidia driver version        : 570.211.01
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              INTEL(R) XEON(R) PLATINUM 8568Y+
CPU family:                              6
Model:                                   207
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                2
BogoMIPS:                                4600.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                600 MiB (2 instances)
NUMA node(s):                            4
NUMA node0 CPU(s):                       0-23,96-119
NUMA node1 CPU(s):                       24-47,120-143
NUMA node2 CPU(s):                       48-71,144-167
NUMA node3 CPU(s):                       72-95,168-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.1
[pip3] nvidia-cutlass-dsl-libs-base==4.4.1
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torch-c-dlpack-ext==0.1.5
[pip3] torchaudio==2.10.0
[pip3] torchvision==0.25.0
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     0,2-23,96,98-119        0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0,2-23,96,98-119        0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     PIX     NODE    SYS     SYS     SYS     SYS     SYS     24-47,120-143   1               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     NODE    PIX     SYS     SYS     SYS     SYS     SYS     24-47,120-143   1               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    SYS     SYS     PIX     48-71,144-167   2               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     SYS     SYS     NODE    48-71,144-167   2               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     SYS     PIX     NODE    SYS     72-95,168-191   3               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     SYS     NODE    PIX     SYS     72-95,168-191   3               N/A
NIC0    PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC2    SYS     SYS     PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS     SYS     SYS     SYS
NIC3    SYS     SYS     NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS     PIX
NIC5    SYS     SYS     SYS     SYS     NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS     NODE
NIC6    SYS     SYS     SYS     SYS     SYS     SYS     PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS
NIC7    SYS     SYS     SYS     SYS     SYS     SYS     NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      SYS
NIC8    SYS     SYS     SYS     SYS     PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS     PIX     NODE    SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_7
  NIC6: mlx5_8
  NIC7: mlx5_9
  NIC8: mlx5_bond_0

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_persona

🐛 Describe the bug

We are repeatedly hitting what looks like a V1 engine / multiprocess shared-memory coordination bug under bursty but otherwise normal chat-completions load.

The engine will run normally for a while, then throughput suddenly drops to zero, we start seeing:

No available shared memory broadcast block found in 60 seconds.

repeated every minute, and eventually the engine dies with:

• TimeoutError: RPC call to sample_tokens timed out.

This is happening even though:

• KV cache usage is still low
• /dev/shm is very large (1TB)
• requests are ordinary chat completions
• this reproduces across multiple versions/config tweaks

My launch args:

vllm serve Qwen/Qwen3.5-122B-A10B \
 --tensor-parallel-size 8 \
 --reasoning-parser qwen3 \
 --served-model-name q3.5-122b \
 --trust-remote-code \
 --cudagraph-metrics \
 --enable-prefix-caching \
 --gpu-memory-utilization 0.95 \
 --max-model-len 16384 \
 --language-model-only \

I'm seeing the error under versions:

• 0.17.0
• 0.17.1
• 0.16.0rc2.dev471+g709eadbb0

Tested already with:

• --max-num-batched-tokens 8192 and lower, other args e.g.
• removing the --language-model-only,
• lowering mem util,
• using Ray backend

Using persistent caches for XDG, Torch, Triton & CUDA.

This is an offline inference workload, but sent through the OpenAI-compatible chat/completions API. So from the engines perspective it is a normal bursty chat workload.

• we've observed crashes after 5 mins, 20 mins, 1hr, 2.5 hrs, or 4.5 hrs.
• the failure often happens when a new stage in our pipeline starts and a fresh burst of requests is added

requests are not especially exotic; ordinary chat completions with prompts in the ~100–1000 token range and max_tokens at 1024. The sender has concurrency limiting and we even added a 10 second exponential rampup, but the issue still reproduces.

Symptoms

A typical sequence is:

• Engine is healthy and serving requests.
  . A burst of new requests arrives.
• Logs show normal throughput, then suddenly:
  Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s
  while there are still many running requests.
• Then every 60 seconds:

No available shared memory broadcast block found in 60 seconds.
This typically happens when some processes are hanging or doing some time-consuming work
(e.g. compilation, weight/kv cache quantization).

Eventually the engine aborts with:

TimeoutError: RPC call to sample_tokens timed out.

Then all worker processes are torn down.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[natech-persona]

an example of the outputs right before it crashes:

APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] **CUDAGraph Stats:**
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] 
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | Unpadded Tokens | Padded Tokens | Num Paddings | Runtime Mode | Count |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] |-----------------|---------------|--------------|--------------|-------|
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7963            | 7963          | 0            | NONE         | 2     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7959            | 7959          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7991            | 7991          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7687            | 7687          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7840            | 7840          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7994            | 7994          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7879            | 7879          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7909            | 7909          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 8146            | 8146          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 8046            | 8046          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7862            | 7862          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 8120            | 8120          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 8147            | 8147          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 8076            | 8076          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7799            | 7799          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] | 7970            | 7970          | 0            | NONE         | 1     |
(APIServer pid=2955628) INFO 03-12 20:47:38 [cuda_graph.py:117] 
(APIServer pid=2955628) INFO 03-12 20:47:48 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 66 reqs, Waiting: 425 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
(EngineCore_DP0 pid=2956319) INFO 03-12 20:48:30 [shm_broadcast.py:548] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore_DP0 pid=2956319) INFO 03-12 20:49:30 [shm_broadcast.py:548] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore_DP0 pid=2956319) INFO 03-12 20:50:30 [shm_broadcast.py:548] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore_DP0 pid=2956319) INFO 03-12 20:51:30 [shm_broadcast.py:548] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.16.0rc2.dev471+g709eadbb0) with config: model='Qwen/Qwen3.5-122B-A10B', speculative_config=None, tokenizer='Qwen/Qwen3.5-122B-A10B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=True, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=q3.5-122b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, 
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-9afd64a2dd5773bd-93f72e03,prompt_token_ids_len=1812,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([0, 0, 1401], [0, 0, 8322], [0, 0, 9851], [15227, 10525, 7912]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None), NewRequestData(req_id=chatcmpl-b2d510bf8ef75918-bb9ffdba,prompt_token_ids_len=1856,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([0, 0, 10218], [0, 0, 10283], [0, 0, 8905], [8694, 8046, 6822]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None), NewRequestData(req_id=chatcmpl-b07eac473b5ec5aa-bd6c8320,prompt_token_ids_len=1940,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([0, 0, 15085], [0, 0, 15076], [0, 0, 1194], [8036, 9143, 13449]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None), NewRequestData(req_id=chatcmpl-b85979245451657a-b88619fa,prompt_token_ids_len=1682,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([0, 0, 6006], [0, 0, 4314], [0, 0, 10861], [15136, 13814, 9894]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-92490ff13bace712-a50df0e5', 'chatcmpl-84cd974be493c657-ae6e310b', 'chatcmpl-8adaa12afe66af2a-a65a6131', 'chatcmpl-a45fd01ffc763e7c-80d2f099', 'chatcmpl-9c4f941da7b7856f-90c91820', 'chatcmpl-a3fd97b3d0fc3de4-8c6f7292', 'chatcmpl-84d86416f85b3052-865e1166', 'chatcmpl-a91ee9f0bec318c9-92c57c8f', 'chatcmpl-b1f5a853a9123615-8625e245', 'chatcmpl-97c51c602a986921-bc046dc6', 'chatcmpl-a6a14d329cdfd6eb-9701971f', 'chatcmpl-b47ea863b5e9a0a9-863cd164', 'chatcmpl-9d3e1a0421023fd3-bfc3281f', 'chatcmpl-887848e9abeff6c6-b7d56b75', 'chatcmpl-85e7067fd379df5f-894a499c', 'chatcmpl-b8745c6699337d3a-bec902c9', 'chatcmpl-89a83f7f7ee22760-a6438053', 'chatcmpl-84749002cf5e27a0-92308d38', 'chatcmpl-95911af8977230d2-bbdffba9', 'chatcmpl-a283879d6a90f0a4-925e6b19', 'chatcmpl-a292c073676b24c6-b855395c', 'chatcmpl-a913db4aed0baed2-a07f00af', 'chatcmpl-b41f19fde8649000-91dc7d01', 'chatcmpl-8c5eca1bdefc4e33-97776a78', 'chatcmpl-893d100a3caf28ae-830a0323', 'chatcmpl-ab0bf242e8a6cfe3-af5159f1', 'chatcmpl-a332a902c2b3657b-bcc1c773', 'chatcmpl-98ff828fd4766e66-95275a75', 'chatcmpl-81e6cf6c3870d664-81d6e576', 'chatcmpl-a77ee5e50f42a8c8-b92f7cba', 'chatcmpl-891980cf2a9af89c-a9d91807', 'chatcmpl-81e5ea8acdfafadc-9ca78af1', 'chatcmpl-bdfaafc01825a626-93c9ce92', 'chatcmpl-964dc767b9d0b99c-9f0a74e6', 'chatcmpl-b7e401f5626ba974-9609f657', 'chatcmpl-8df536ffa5b00266-8d814bd8', 'chatcmpl-9add1cf822613be0-84251eaa', 'chatcmpl-a3a4eb0a5381f125-9c739ebc', 'chatcmpl-94b65435ccc3722d-ac9b6945', 'chatcmpl-9af88789cf652e5b-9820cfe7', 'chatcmpl-8ac5e264a8dea244-9121c158', 'chatcmpl-949dded8b4a1897b-95ddfc06', 'chatcmpl-9c900c59669d3b35-bd8fe340', 'chatcmpl-b49834882f194706-96f5718e', 'chatcmpl-9ec4014cfb182e20-9ff06fcf', 'chatcmpl-91d4dc51fb5ca3fb-96a692a0', 'chatcmpl-a3d2af775c8cb881-9a176bcd', 'chatcmpl-be90ca5ad6fa0909-83c748c7', 'chatcmpl-96cb070f27944aa9-82f8c8df', 'chatcmpl-bcdbf6476837afe0-8d8a0a79', 'chatcmpl-a753a5baab538c6f-8cbcad11', 'chatcmpl-a1ac6fe2d1414672-8b1926a2', 'chatcmpl-a81da2caddee3d43-a111143a', 'chatcmpl-9ca34bb79992052b-90db0071', 'chatcmpl-86cb615bbba73e43-a99774ff', 'chatcmpl-9d16444da6b76188-9a120226', 'chatcmpl-a4459bdfbb386a83-817bdea0', 'chatcmpl-92ea3d14b58f3f98-8c65726b', 'chatcmpl-8936686159e308ba-9b1f3891', 'chatcmpl-80b9548d13a26a4b-a12cbddc', 'chatcmpl-9b02fa06c1aceaa3-a4fa64af', 'chatcmpl-b73de05241f950b6-a5a7c053', 'chatcmpl-97ec310c89ee8382-9f745cc8', 'chatcmpl-b05e37e759bbf79d-a5fd5492', 'chatcmpl-85f7b0f76bc80830-a7e008c2', 'chatcmpl-a65b3c1a4adebb35-8c9bb0a9', 'chatcmpl-b2cf706d4dcffa50-98db9920'],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, ([14819], [9460], [15717], [10064]), ([159], [9356], [14115], [1962]), ([13516], [11276], [14662], [1348]), ([0, 742], [0, 2008], [0, 7774], [15649, 11785])],num_computed_tokens=[1608, 2258, 1422, 2000, 1465, 1922, 1719, 1439, 1544, 2277, 2023, 2062, 1740, 2289, 1855, 2031, 1274, 2723, 1921, 1619, 2123, 1920, 1713, 2548, 2231, 2572, 2227, 1899, 1947, 1402, 1981, 1564, 2178, 1952, 1703, 2065, 1938, 2219, 1684, 2226, 1950, 2085, 1957, 2157, 1903, 2148, 1649, 1495, 1612, 2069, 2349, 1664, 2205, 1805, 1548, 2170, 1987, 2037, 1799, 1480, 1953, 1660, 2072, 1584, 2112, 2112, 528],num_output_tokens=[15, 15, 15, 15, 15, 14, 14, 14, 14, 14, 13, 13, 13, 12, 12, 12, 12, 11, 11, 11, 12, 10, 10, 10, 10, 9, 9, 9, 9, 8, 8, 8, 8, 8, 7, 7, 7, 6, 6, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 4, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0]), num_scheduled_tokens={chatcmpl-84cd974be493c657-ae6e310b: 1, chatcmpl-b73de05241f950b6-a5a7c053: 1, chatcmpl-85f7b0f76bc80830-a7e008c2: 121, chatcmpl-9d3e1a0421023fd3-bfc3281f: 1, chatcmpl-a3fd97b3d0fc3de4-8c6f7292: 1, chatcmpl-92490ff13bace712-a50df0e5: 1, chatcmpl-a913db4aed0baed2-a07f00af: 1, chatcmpl-a1ac6fe2d1414672-8b1926a2: 1, chatcmpl-8adaa12afe66af2a-a65a6131: 1, chatcmpl-81e6cf6c3870d664-81d6e576: 1, chatcmpl-b41f19fde8649000-91dc7d01: 1, chatcmpl-a4459bdfbb386a83-817bdea0: 1, chatcmpl-a3d2af775c8cb881-9a176bcd: 1, chatcmpl-85e7067fd379df5f-894a499c: 1, chatcmpl-8ac5e264a8dea244-9121c158: 1, chatcmpl-97ec310c89ee8382-9f745cc8: 1, chatcmpl-949dded8b4a1897b-95ddfc06: 1, chatcmpl-b2cf706d4dcffa50-98db9920: 1056, chatcmpl-96cb070f27944aa9-82f8c8df: 1, chatcmpl-a292c073676b24c6-b855395c: 1, chatcmpl-8df536ffa5b00266-8d814bd8: 1, chatcmpl-84d86416f85b3052-865e1166: 1, chatcmpl-b7e401f5626ba974-9609f657: 1, chatcmpl-9ca34bb79992052b-90db0071: 1, chatcmpl-a81da2caddee3d43-a111143a: 1, chatcmpl-92ea3d14b58f3f98-8c65726b: 1, chatcmpl-b8745c6699337d3a-bec902c9: 1, chatcmpl-84749002cf5e27a0-92308d38: 1, chatcmpl-887848e9abeff6c6-b7d56b75: 1, chatcmpl-a3a4eb0a5381f125-9c739ebc: 1, chatcmpl-b2d510bf8ef75918-bb9ffdba: 1584, chatcmpl-9c4f941da7b7856f-90c91820: 1, chatcmpl-a65b3c1a4adebb35-8c9bb0a9: 112, chatcmpl-8c5eca1bdefc4e33-97776a78: 1, chatcmpl-9ec4014cfb182e20-9ff06fcf: 1, chatcmpl-b05e37e759bbf79d-a5fd5492: 251, chatcmpl-a332a902c2b3657b-bcc1c773: 1, chatcmpl-a6a14d329cdfd6eb-9701971f: 1, chatcmpl-891980cf2a9af89c-a9d91807: 1, chatcmpl-9add1cf822613be0-84251eaa: 1, chatcmpl-89a83f7f7ee22760-a6438053: 1, chatcmpl-bdfaafc01825a626-93c9ce92: 1, chatcmpl-98ff828fd4766e66-95275a75: 1, chatcmpl-9c900c59669d3b35-bd8fe340: 1, chatcmpl-a753a5baab538c6f-8cbcad11: 1, chatcmpl-b47ea863b5e9a0a9-863cd164: 1, chatcmpl-97c51c602a986921-bc046dc6: 1, chatcmpl-81e5ea8acdfafadc-9ca78af1: 1, chatcmpl-9d16444da6b76188-9a120226: 1, chatcmpl-be90ca5ad6fa0909-83c748c7: 1, chatcmpl-9afd64a2dd5773bd-93f72e03: 1584, chatcmpl-b85979245451657a-b88619fa: 1584, chatcmpl-ab0bf242e8a6cfe3-af5159f1: 1, chatcmpl-86cb615bbba73e43-a99774ff: 1, chatcmpl-80b9548d13a26a4b-a12cbddc: 1, chatcmpl-b07eac473b5ec5aa-bd6c8320: 1584, chatcmpl-91d4dc51fb5ca3fb-96a692a0: 1, chatcmpl-94b65435ccc3722d-ac9b6945: 1, chatcmpl-bcdbf6476837afe0-8d8a0a79: 1, chatcmpl-b1f5a853a9123615-8625e245: 1, chatcmpl-8936686159e308ba-9b1f3891: 1, chatcmpl-964dc767b9d0b99c-9f0a74e6: 1, chatcmpl-9af88789cf652e5b-9820cfe7: 1, chatcmpl-9b02fa06c1aceaa3-a4fa64af: 1, chatcmpl-95911af8977230d2-bbdffba9: 1, chatcmpl-a91ee9f0bec318c9-92c57c8f: 1, chatcmpl-a45fd01ffc763e7c-80d2f099: 1, chatcmpl-a77ee5e50f42a8c8-b92f7cba: 1, chatcmpl-893d100a3caf28ae-830a0323: 1, chatcmpl-b49834882f194706-96f5718e: 1, chatcmpl-a283879d6a90f0a4-925e6b19: 1}, total_num_scheduled_tokens=7939, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0], finished_req_ids=['chatcmpl-93cf42faf5095db0-9bf4c50b', 'chatcmpl-9e2b1f4c2eb48ba5-993df6c5', 'chatcmpl-a66f77c9c18e4e6a-b51952bb', 'chatcmpl-b4010f088b81f024-aae2e54b'], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null)
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=71, num_waiting_reqs=424, step_counter=0, current_wave=0, kv_cache_usage=0.03161764705882353, encoder_cache_usage=0.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=6, queries=11487, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] Traceback (most recent call last):
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 352, in get_response
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     status, result = mq.dequeue(
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 622, in dequeue
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     with self.acquire_read(timeout, cancel, indefinite) as buf:
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     return next(self.gen)
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 542, in acquire_read
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     raise TimeoutError
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] TimeoutError
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] 
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] 
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] Traceback (most recent call last):
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1072, in run_engine_core
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1107, in run_busy_loop
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     self._process_engine_step()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1138, in _process_engine_step
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 494, in step_with_batch_queue
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     model_output = future.result()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 81, in result
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     return super().result()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     return self.__get_result()
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     raise self._exception
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 85, in wait_for_response
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     response = self.aggregate(get_response())
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 356, in get_response
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081]     raise TimeoutError(f"RPC call to {method} timed out.") from e
(EngineCore_DP0 pid=2956319) ERROR 03-12 20:52:30 [core.py:1081] TimeoutError: RPC call to sample_tokens timed out.
(Worker_TP2 pid=2956402) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP1 pid=2956401) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP0 pid=2956400) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP3 pid=2956403) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP6 pid=2956406) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP7 pid=2956407) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP5 pid=2956405) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker_TP4 pid=2956404) INFO 03-12 20:52:30 [multiproc_executor.py:732] Parent process exited, terminating worker
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702] AsyncLLM output_handler failed.
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702] Traceback (most recent call last):
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 658, in output_handler
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702]     outputs = await engine_core.get_output_async()
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702]   File "/mnt/weka/cursive-nat/persona-src/sbert/vllm_latest_env_q3_5/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 917, in get_output_async
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702]     raise self._format_exception(outputs) from None
(APIServer pid=2955628) ERROR 03-12 20:52:30 [async_llm.py:702] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

[ZJY0516]

Could you please check your cpu resources, maybe it was compiling jit kernel

[natech-persona]

Thanks for the suggestion -

We enabled PyTorch recompilation logging and confirmed real recompiles on all TP workers (get_masked_input_and_mask in vocab_parallel_embedding.py) due to guard failures on input_.size()[0].

However, the recompiles occur about a minute before the engine stalls; the engine continues making good progress after the recompile event, then later drops abruptly to 0 throughput while retaining ~100 running requests and low KV usage, after which SHM broadcast block acquisition times out repeatedly and sample_tokens RPC times out. This makes us suspect recompilation / dynamic-shape churn may be a trigger or destabilizer, but not the full explanation. Also, enabling very verbose TORCH_LOGS noticeably increased crash frequency, likely due to added CPU/stderr overhead.

Currently I'm trying to get some better CPU monitoring over one of the crashes, will update with more logs.

[natech-persona]

Before crash:

pidstat outputs

# Time        UID       PID    %usr %system  %guest   %wait    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
17:49:06     1001   4050928   11.00    1.00    0.00    0.00   12.00    77   2142.00      0.00 79271524 1668212   0.08      0.00      0.00      0.00       0  vllm
17:49:06     1001   4051679    0.00    0.00    0.00    0.00    0.00   172      0.00      0.00   19404   11124   0.00      0.00      0.00      0.00       0  python3
17:49:06     1001   4051680   79.00   21.00    0.00    0.00  100.00     5     69.00      0.00 56084492 1497020   0.07      0.00      0.00      0.00       0  VLLM::EngineCor
17:49:06     1001   4051761  123.00    2.00    0.00    0.00  125.00    74    987.00      0.00 1091845944 4333800   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051762   98.00    3.00    0.00    0.00  101.00    24   1402.00      0.00 1091705488 4335508   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051763   97.00    3.00    0.00    0.00  100.00     2    960.00      0.00 1091698936 4332932   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051764   99.00    1.00    0.00    0.00  100.00   121    936.00      0.00 1091700088 4330240   0.20      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051765   98.00    2.00    0.00    0.00  100.00    72    995.00      0.00 1091721460 4353924   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051766   98.00    2.00    0.00    0.00  100.00    73    805.00      0.00 1091706364 4334940   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051767   99.00    1.00    0.00    0.00  100.00    96    785.00      0.00 1091706436 4338660   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:06     1001   4051768   99.00    2.00    0.00    0.00  101.00    50    893.00      0.00 1091699328 4331392   0.20      0.00      0.00      0.00       0  VLLM::Worker_TP

I note from this it's basically all usr work, no sys work. Then as the crash happens:

pidstat outputs

# Time        UID       PID    %usr %system  %guest   %wait    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
17:49:11     1001   4050928    0.00    0.00    0.00    0.00    0.00    77     21.00      0.00 79271524 1668212   0.08      0.00      0.00      0.00       0  vllm
17:49:11     1001   4051679    0.00    0.00    0.00    0.00    0.00   172      0.00      0.00   19404   11124   0.00      0.00      0.00      0.00       0  python3
17:49:11     1001   4051680   83.00   17.00    0.00    0.00  100.00     5     51.00      0.00 56084492 1497020   0.07      0.00      0.00      0.00       0  VLLM::EngineCor
17:49:11     1001   4051761   56.00   45.00    0.00    0.00  101.00    74    160.00      0.00 1091845944 4333800   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051762   55.00   46.00    0.00    0.00  101.00   120    149.00      0.00 1091705488 4335508   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051763   59.00   42.00    0.00    0.00  101.00     2    146.00      0.00 1091698936 4332932   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051764   56.00   45.00    0.00    0.00  101.00   121    159.00      0.00 1091700088 4330240   0.20      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051765   53.00   47.00    0.00    0.00  100.00    72    264.00      0.00 1091721460 4353924   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051766   53.00   48.00    0.00    0.00  101.00    73    132.00      0.00 1091706364 4334940   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051767   49.00   52.00    0.00    0.00  101.00    96    133.00      0.00 1091706436 4338660   0.21      0.00      0.00      0.00       0  VLLM::Worker_TP
17:49:11     1001   4051768   53.00   48.00    0.00    0.00  101.00    50    201.00      0.00 1091699328 4331392   0.20      0.00      0.00      0.00       0  VLLM::Worker_TP

And it continues in that 50/50 pattern until the engine crashes 5 mins later, after several No available shared memory broadcast block found in 60 seconds messages.

[wuzixia]

I'm having the same issue with a PD-separated deployment on ds3.1. Have you managed to resolve it?

[natech-persona]

Hi, no still dealing with it. Given up debugging it, current workaround is just to have a separate process that watches for the "No available... " log and then launches a new job when one of them crashes.

[natech-persona]

Fixed now.

Updated to 0.18.0, enabled --gdn-prefill-backend triton, then had to patch compilation/compiler_interface.py with:

if input_fake_mode is not None:
    # In Python 3.10, string-based patch resolves standalone_compile
    # to the wrapper function rather than the module. Use patch.object
    # with explicit module reference instead.
    import sys
    sc_module = sys.modules["torch._inductor.standalone_compile"]
    fake_mode_ctx: Any = patch.object(
        sc_module, "FakeTensorMode",
        lambda *a, **kw: input_fake_mode,
    )

at line 375. Expect this means that 0.18.0 is no longer compatible with python 3.10?

[miguelmanlyx]

Heads up — that looks like a ai api timeout failure. Might be worth trying npx ai-doctor. It usually points out what is breaking.

[wuzixia]

So what is causing this issue, SHM memory? It appears that vllm0.18.0 uses self._spin_condition.wait(timeout_ms=read_timeout.timeout_ms())
instead of
sched_yield()

My project is based on a modified version of vllm 0.11.0. Do I need to migrate this part of the code over?

[natech-persona]

TBH dude I'm really not sure - I expect that it's something to do with the new linear GDN layers that comes with the Q3.5 arch, hence why changing the backend for it fixes things. I tried installing the flashinfer JIT cache package that some similar issues recommend, but that didn't solve it etither. Sorry I can't be of more help, I'm a vllm noob.

[lianjiezh]

This kind of problem should indicate that a certain worker process has hung.

[ZJY0516]

Fixed now.

Updated to 0.18.0, enabled --gdn-prefill-backend triton, then had to patch compilation/compiler_interface.py with:

if input_fake_mode is not None:
    # In Python 3.10, string-based patch resolves standalone_compile
    # to the wrapper function rather than the module. Use patch.object
    # with explicit module reference instead.
    import sys
    sc_module = sys.modules["torch._inductor.standalone_compile"]
    fake_mode_ctx: Any = patch.object(
        sc_module, "FakeTensorMode",
        lambda *a, **kw: input_fake_mode,
    )

at line 375. Expect this means that 0.18.0 is no longer compatible with python 3.10?

We'll fix it in 0.18.1 soon