[Bug]: AWQ export of Qwen3.5-9B (DeltaNet Hybrid) crashes in vLLM due to non-divisible Output Channels (in_proj_ba)

[vishvaRam]

Problem Description

Description:
I successfully quantized the new Qwen3.5-9B hybrid architecture (which features Gated DeltaNets) using AutoRound. However, when exporting the quantized model to the AWQ format and serving it with vLLM, the inference engine immediately crashes upon receiving the first prompt.

The INC native export format works perfectly, but the AWQ export fails during inference.

Error Log:
When the forward pass hits the small internal projection layers of the DeltaNet block, vLLM's AWQ C++ kernel throws the following error:

  File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 169, in forward
    ba, _ = self.in_proj_ba(hidden_states)
...
  File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/awq.py", line 275, in apply
    out = ops.awq_gemm(reshaped_x, qweight, scales, qzeros, pack_factor)
...
ValueError: OC is not multiple of Group size

Root Cause Analysis:
The Qwen3.5-9B architecture contains small data-dependent projection layers (like in_proj_ba in the linear attention mechanism) where the Output Channels (OC) are very small (e.g., 32 or 64).

Because the AWQ kernel strictly requires the OC dimension to be perfectly divisible by the quantization group_size (usually 128), it triggers a fatal ValueError during the awq_gemm operation.

Expected Behavior:
AutoRound's AWQ export function should automatically detect linear layers where OC % group_size != 0 (or specifically target the known small projection layers in hybrid architectures like Qwen3.5's DeltaNets) and automatically fall back to bypassing/ignoring them, leaving them in standard fp16/bf16.

Steps to Reproduce:

1. Quantize Qwen3.5-9B using AutoRound with W4A16.
2. Export the result to AWQ format.
3. Load the AWQ model into vLLM using --quantization awq.
4. Send any generation request to the server.

Temporary Workaround:
Currently, users have to manually identify and pass ignore_modules=["in_proj_ba", ...] to the AutoRound API to prevent these layers from being quantized, or rely strictly on the INC format.

Thank you for looking into this!

Reproduction Steps

1. Run the AutoRound quantization script on the new Qwen3.5-9B (Hybrid Gated DeltaNet) architecture.
2. Set parameters to W4A16 (or bits=4, group_size=128) and export the quantized model to the AWQ format.
3. Serve the exported AWQ model using vLLM (vllm serve /path/to/model --quantization awq --enforce-eager).
4. Send a prompt to the vLLM server via the /v1/chat/completions endpoint.
5. See error: ValueError: OC is not multiple of Group size immediately during the forward pass at the in_proj_ba layer.

Environment Information

• OS: Windows 11 (running via WSL2 / Docker)
• Python version: 3.12.12 (inside vLLM Docker container)
• AutoRound version: [latest main branch]
• Hardware: NVIDIA RTX 3090 (24GB VRAM)
• vLLM version: (Nightly)
• Transformers version: 5.1.0 in vLLM docker image
• Transformers version: 5.2.0 in while Quantization

Error Logs

admin@WINRENRD28 MINGW64 /e/Vishva/vLLM-Inference/vLLM-Qwen-3.5-9B
$ docker compose up
[+] up 2/2
 ✔ Network vllm-qwen-35-9b_default Created                                                                            0.0s ✔ Container vllm-qwen3-5          Created                                                                            0.1sAttaching to vllm-qwen3-5
vllm-qwen3-5  | /usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/protocol.py:346: SyntaxWarning: invalid escape sequence '\e'
vllm-qwen3-5  |   "(e.g. 'abcdabcdabcd...' or '\emoji \emoji \emoji ...'). This feature "
vllm-qwen3-5  | /usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/completion/protocol.py:176: SyntaxWarning: invalid escape sequence '\e'
vllm-qwen3-5  |   "(e.g. 'abcdabcdabcd...' or '\emoji \emoji \emoji ...'). This feature "
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302] 
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302]        █     █     █▄   ▄█
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.16.1rc1.dev203+gedba15045
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302]   █▄█▀ █     █     █     █  model   /models/llm
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:302]
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:13 [utils.py:238] non-default args: {'model_tag': '/models/llm', 'host': '0.0.0.0', 'port': 8521, 'model': '/models/llm', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 6000, 'enforce_eager': True, 'gpu_memory_utilization': 0.7, 'enable_prefix_caching': True, 'max_num_seqs': 256}
vllm-qwen3-5  | (APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
vllm-qwen3-5  | (APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
vllm-qwen3-5  | (APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
vllm-qwen3-5  | (APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
vllm-qwen3-5  | (APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
vllm-qwen3-5  | (APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:26 [model.py:530] Resolved architecture: Qwen3_5ForConditionalGeneration
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:26 [model.py:1553] Using max model len 6000
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:27 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
vllm-qwen3-5  | (APIServer pid=1) WARNING 03-04 09:08:27 [config.py:381] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled                                                           
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:27 [config.py:401] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.             
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:28 [config.py:544] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:28 [config.py:575] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.                                                                    
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:28 [vllm.py:747] Asynchronous scheduling is enabled.
vllm-qwen3-5  | (APIServer pid=1) WARNING 03-04 09:08:28 [vllm.py:781] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none                                               
vllm-qwen3-5  | (APIServer pid=1) WARNING 03-04 09:08:28 [vllm.py:792] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.                                 
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:08:28 [vllm.py:957] Cudagraph is disabled under eager mode
vllm-qwen3-5  | (APIServer pid=1) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (APIServer pid=1) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (APIServer pid=1) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (APIServer pid=1) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (APIServer pid=1) WARNING 03-04 09:08:53 [interface.py:472] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:04 [core.py:101] Initializing a V1 LLM engine (v0.16.1rc1.dev203+gedba15045) with config: model='/models/llm', speculative_config=None, tokenizer='/models/llm', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=6000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/models/llm, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
vllm-qwen3-5  | (EngineCore_DP0 pid=94) WARNING 03-04 09:09:04 [interface.py:472] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:08 [parallel_state.py:1393] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.19.0.2:44069 backend=nccl
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:08 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
vllm-qwen3-5  | (EngineCore_DP0 pid=94) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:27 [base.py:106] Offloader set to NoopOffloader
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:27 [gpu_model_runner.py:4255] Starting to load model /models/llm...                                                                                                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:28 [cuda.py:453] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:28 [mm_encoder_attention.py:215] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:30 [cuda.py:405] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:30 [flash_attn.py:587] Using FlashAttention version 2
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:01<00:01,  1.54s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:03<00:00,  1.75s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:03<00:00,  1.72s/it]
vllm-qwen3-5  | (EngineCore_DP0 pid=94) 
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:34 [default_loader.py:293] Loading weights took 3.67 seconds
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:34 [gpu_model_runner.py:4338] Model loading took 8.13 GiB memory and 6.454295 seconds
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:09:35 [gpu_model_runner.py:5254] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:13 [gpu_worker.py:424] Available KV cache memory: 6.74 GiB
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:13 [kv_cache_utils.py:1314] GPU KV cache size: 54,912 tokens
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:13 [kv_cache_utils.py:1319] Maximum concurrency for 6,000 tokens per request: 23.22x                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:16 [core.py:282] init engine (profile, create kv cache, warmup model) took 41.40 seconds
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:16 [vllm.py:747] Asynchronous scheduling is enabled.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) WARNING 03-04 09:10:16 [vllm.py:781] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none                                         
vllm-qwen3-5  | (EngineCore_DP0 pid=94) WARNING 03-04 09:10:16 [vllm.py:792] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) INFO 03-04 09:10:16 [vllm.py:957] Cudagraph is disabled under eager mode
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:17 [api_server.py:495] Supported tasks: ['generate']
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:17 [serving.py:185] Warming up chat template processing...
vllm-qwen3-5  | (APIServer pid=1) You are using a model of type qwen3_5 to instantiate a model of type . This is not supported for all configurations of models and can yield errors.                                                               
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [hf.py:318] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [serving.py:210] Chat template warmup completed in 5124.7ms
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [api_server.py:500] Starting vLLM API server 0 on http://0.0.0.0:8521                                                                                                                         
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:38] Available routes are:
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /openapi.json, Methods: HEAD, GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /docs, Methods: HEAD, GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /docs/oauth2-redirect, Methods: HEAD, GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /redoc, Methods: HEAD, GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /tokenize, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /detokenize, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /load, Methods: GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /version, Methods: GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /health, Methods: GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /metrics, Methods: GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/models, Methods: GET
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /ping, Methods: GET                         
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /ping, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /invocations, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/chat/completions, Methods: POST         
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/chat/completions/render, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/responses, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/responses/{response_id}, Methods: GET   
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/responses/{response_id}/cancel, Methods: POST                                                                                                                     
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/completions, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/completions/render, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/messages, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /v1/messages/count_tokens, Methods: POST    
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /inference/v1/generate, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /scale_elastic_ep, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO 03-04 09:10:22 [launcher.py:47] Route: /is_scaling_elastic_ep, Methods: POST
vllm-qwen3-5  | (APIServer pid=1) INFO:     Started server process [1]                                                    
vllm-qwen3-5  | (APIServer pid=1) INFO:     Waiting for application startup.
vllm-qwen3-5  | (APIServer pid=1) INFO:     Application startup complete.
vllm-qwen3-5  | (APIServer pid=1) INFO:     172.19.0.1:35412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.16.1rc1.dev203+gedba15045) with config: model='/models/llm', speculative_config=None, tokenizer='/models/llm', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=6000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/models/llm, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []},
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-81918805a08c8f9d-a7190ac1,prompt_token_ids_len=30,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=2048, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1], [2], [3], [4]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={chatcmpl-81918805a08c8f9d-a7190ac1: 30}, total_num_scheduled_tokens=30, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 1], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.00959232613908878, encoder_cache_usage=0.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=30, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102] EngineCore encountered a fatal error.
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102] Traceback (most recent call last):            
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1093, in run_engine_core                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     engine_core.run_busy_loop()
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1128, in run_busy_loop                                                                    
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     self._process_engine_step()
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1165, in _process_engine_step                                                             
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     outputs, model_executed = self.step_fn()
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]                               ^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 501, in step_with_batch_queue                                                             
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     exec_model_fut.result()
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self.__get_result()
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result                                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     raise self._exception
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     result = run_method(self.driver_worker, method, args, kwargs)                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 459, in run_method                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 365, in execute_model                                                              
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self.worker.execute_model(scheduler_output)                                                                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context                                                              
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 720, in execute_model                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     output = self.model_runner.execute_model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context                                                              
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3613, in execute_model                                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     model_output = self._model_forward(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]                    ^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3126, in _model_forward                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self.model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                   
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 738, in forward                                                            
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     hidden_states = self.language_model.model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 389, in __call__                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self.forward(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1151, in forward                                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     hidden_states, residual = layer(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]                               ^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                   
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1045, in forward                                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     self.linear_attn(
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 169, in forward                                                            
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     ba, _ = self.in_proj_ba(hidden_states)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 576, in forward                                                             
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     output_parallel = self.quant_method.apply(self, input_, bias)                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/awq.py", line 275, in apply                                                     
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     out = ops.awq_gemm(reshaped_x, qweight, scales, qzeros, pack_factor)                                                                                                
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/vllm/_custom_ops.py", line 547, in awq_gemm                                                                             
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return torch.ops._C.awq_gemm(input, qweight, scales, qzeros, split_k_iters)                                                                                         
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]   File "/usr/local/lib/python3.12/site-packages/torch/_ops.py", line 1209, in __call__                                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]     return self._op(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102]            ^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ERROR 03-04 09:10:28 [core.py:1102] ValueError: OC is not multiple of Group size
vllm-qwen3-5  | (EngineCore_DP0 pid=94) Process EngineCore_DP0:                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94) Traceback (most recent call last):
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap                                                                                                                      
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     self.run()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     self._target(*self._args, **self._kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1104, in run_engine_core                                                                                                      
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     raise e
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1093, in run_engine_core
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     engine_core.run_busy_loop()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1128, in run_busy_loop                                                                                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     self._process_engine_step()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1165, in _process_engine_step                                                                                                 
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     outputs, model_executed = self.step_fn()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)                               ^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 501, in step_with_batch_queue                                                                                                 
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     exec_model_fut.result()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result                                                                                                                         
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self.__get_result()
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     raise self._exception
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc                                                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     result = run_method(self.driver_worker, method, args, kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 459, in run_method                                                                                                           
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 365, in execute_model                                                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self.worker.execute_model(scheduler_output)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 720, in execute_model                                                                                                   
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     output = self.model_runner.execute_model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context                                                                                                  
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return func(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3613, in execute_model                                                                                            
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     model_output = self._model_forward(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)                    ^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3126, in _model_forward
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self.model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 738, in forward                                                                                                
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     hidden_states = self.language_model.model(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 389, in __call__                                                                                                      
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self.forward(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1151, in forward                                                                                            
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     hidden_states, residual = layer(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)                               ^^^^^^                                              
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1045, in forward                                                                                            
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     self.linear_attn(
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 169, in forward                                                                                                
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     ba, _ = self.in_proj_ba(hidden_states)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                        
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl                                                                                               
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self._call_impl(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl                                                                                                       
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return forward_call(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 576, in forward                                                                                                 
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     output_parallel = self.quant_method.apply(self, input_, bias)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/awq.py", line 275, in apply                                                                                         
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     out = ops.awq_gemm(reshaped_x, qweight, scales, qzeros, pack_factor)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/vllm/_custom_ops.py", line 547, in awq_gemm                                                                                                                 
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return torch.ops._C.awq_gemm(input, qweight, scales, qzeros, split_k_iters)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94)   File "/usr/local/lib/python3.12/site-packages/torch/_ops.py", line 1209, in __call__                                                                                                                      
vllm-qwen3-5  | (EngineCore_DP0 pid=94)     return self._op(*args, **kwargs)
vllm-qwen3-5  | (EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (EngineCore_DP0 pid=94) ValueError: OC is not multiple of Group size
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708] AsyncLLM output_handler failed.                 
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708] Traceback (most recent call last):              
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 664, in output_handler                                                                 
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708]     outputs = await engine_core.get_output_async()                                                                                                                        
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                        
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1004, in get_output_async
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708]     raise self._format_exception(outputs) from None
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [async_llm.py:708] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390] Error in chat completion stream generator.
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390] Traceback (most recent call last):
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]   File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 714, in chat_completion_stream_generator
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]     async for res in result_generator:
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 583, in generate
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]     out = q.get_nowait() or await q.get()
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]                             ^^^^^^^^^^^^^
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/output_processor.py", line 85, in get
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]     raise output
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 664, in output_handler
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]     outputs = await engine_core.get_output_async()
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]   File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1004, in get_output_async
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390]     raise self._format_exception(outputs) from None
vllm-qwen3-5  | (APIServer pid=1) ERROR 03-04 09:10:28 [serving.py:1390] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
vllm-qwen3-5  | [rank0]:[W304 09:10:29.986027299 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
vllm-qwen3-5  | (APIServer pid=1) INFO:     Shutting down
vllm-qwen3-5  | (APIServer pid=1) INFO:     Waiting for application shutdown.
vllm-qwen3-5  | (APIServer pid=1) INFO:     Application shutdown complete.
vllm-qwen3-5  | (APIServer pid=1) INFO:     Finished server process [1]
vllm-qwen3-5 exited with code 0

admin@WINRENRD28 MINGW64 /e/Vishva/vLLM-Inference/vLLM-Qwen-3.5-9B
$ View in Docker Desktop   o View Config   w Enable Watch   d Detach

Additional Context

No response

[wenhuach21]

Thanks for reporting this issue. What behavior do you expect from us? If the kernel does not support the layer configuration, the only option we have is to add these layers to the predefined 'ignore layers' list, as we don’t own the kernel.

By the way, in vLLM, AutoRound leverages the GPTQ Marlin kernel. Could you clarify why you need to export to the AWQ format? Is AWQ Marlin faster than GPTQ Marlin?

[vishvaRam]

Thank you for the quick response and explanation!

Sorry to bother you with this, there was no specific performance reason for using AWQ. I was just exploring different export formats with the new Qwen3.5 architecture to push multiple format options to the Hugging Face hub for others to use.

Since the native AutoRound format works perfectly with the GPTQ Marlin kernel in vLLM without this dimension issue, I will just stick with that. I appreciate the clarification on how the kernels are handled. I'll go ahead and close this issue.