Slow inference on dual RTX 3090 with NVLINK

[duart]

Hi,
Can you please have a look into to my config and report results? Maybe you can help me to fill the gap between my inference performance and the one you are geting ...

Related to: Optimizing LLM inference performance with Dual RTX 3090 + NVLink on Proxmox

At the moment, after Proxmox VM configuration, I'm achieving around 41 tokens per second NARRATIVE and ~ 55 tokens pers second CODE. GPU power is capped to 260W (from 350W) but it makes near no difference in inference speed.

It is still much less than what docs mentions (69/89) but it is already usable as I started from ~ 20 tokens per second.

My hardware setup is
Dell T630 server
2X Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
256GB DDR4 RAM,
2x RTX3090 in PCIe 3.0 x 16
VLLM runs in Proxmox VM with GPU's being pass-through to VM

This is my report
~/club-3090$ bash scripts/report.sh --full

club-3090 rig report

Generated: 2026-05-18 14:48:51 UTC

Redacted output (paths, host, user, tokens). Re-run with --no-redact for full data.

System

• OS: Ubuntu 22.04.5 LTS
• Kernel: 5.15.0-177-generic
• Environment: kvm (virtualized)
• Locale: en_US.UTF-8
• Timezone: UTC
• Uptime: up 3 hours, 46 minutes

CPU + RAM

• CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (8 threads)
• RAM: 31Gi total, 21Gi available

Disk

• /var/lib/docker: 130G available, ext4 filesystem

GPU hardware

• GPU 0: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 590.48.01 | VBIOS 94.02.42.00.02 | persistence=Enabled
  • Power: limit=260.00 W (default=350.00 W, max=350.00 W) | current_draw=28.54 W ⚠ user-capped below default
  • PCIe: x16 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:02:00.0
• GPU 1: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 590.48.01 | VBIOS 94.02.42.40.34 | persistence=Enabled
  • Power: limit=260.00 W (default=350.00 W, max=350.00 W) | current_draw=25.00 W ⚠ user-capped below default
  • PCIe: x16 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:04:00.0
• CUDA Runtime (per driver): 13.1
• ECC mode: [N/A] (3090s don't have ECC; expect N/A)

NVLink

NVLink link status

GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-a47bc59a-32f9-7d09-cf3d-92c1757f7f28)
         Link 0: 14.062 GB/s
         Link 1: 14.062 GB/s
         Link 2: 14.062 GB/s
         Link 3: 14.062 GB/s
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-13c1bcfe-84cd-6937-6b8c-80b3e0296710)
         Link 0: 14.062 GB/s
         Link 1: 14.062 GB/s
         Link 2: 14.062 GB/s
         Link 3: 14.062 GB/s

Topology

PCIe / GPU topology matrix

        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV4     0-7     0               N/A
GPU1    NV4      X      0-7     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

PCIe / P2P detail (lspci)

lspci PCIe/P2P detail (LnkSta / ACS / topology)

# lspci -t  (PCIe topology tree)
-[0000:00]-+-00.0
           +-10.0-[01]----00.0
           +-1a.0
           +-1a.1
           +-1a.2
           +-1a.7
           +-1b.0
           +-1c.0-[02]----00.0
           +-1c.1-[03]----00.0
           +-1c.2-[04]----00.0
           +-1c.3-[05]----00.0
           +-1d.0
           +-1d.1
           +-1d.2
           +-1d.7
           +-1e.0-[06-0a]--+-01.0-[07]--+-08.0
           |               |            \-12.0
           |               +-02.0-[08]--
           |               +-03.0-[09]--
           |               \-04.0-[0a]--+-01.0
           |                            \-02.0
           +-1f.0
           +-1f.2
           \-1f.3

# lspci -vvv -s 0000:02:00.0  (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)

# lspci -vvv -s 0000:00:1c.0  (upstream bridge of 0000:02:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #1, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 2.5GT/s (downgraded), Width x16 (downgraded)
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

# lspci -vvv -s 0000:04:00.0  (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)

# lspci -vvv -s 0000:00:1c.2  (upstream bridge of 0000:04:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
                LnkCap: Port #3, Speed 16GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 2.5GT/s (downgraded), Width x16 (downgraded)
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

# lspci -nnk | grep -A3 -i nvidia  (driver binding + device IDs)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:1454]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nvidia_drm, nvidia
03:00.0 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1454]
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GA102 [GeForce RTX 3090] [1458:403b]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nvidia_drm, nvidia
05:00.0 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GA102 High Definition Audio Controller [1458:403b]
06:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
06:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]

Full nvidia-smi

Full nvidia-smi output

Mon May 18 14:48:52 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:02:00.0 Off |                  N/A |
| 49%   30C    P8             28W /  260W |   23848MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  |   00000000:04:00.0 Off |                  N/A |
| 31%   42C    P8             25W /  260W |   23848MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3905      C   VLLM::Worker_TP0                      23840MiB |
|    1   N/A  N/A            3906      C   VLLM::Worker_TP1                      23840MiB |
+-----------------------------------------------------------------------------------------+

Display / desktop state

• $DISPLAY: unset (headless)
• Display processes running: none detected
• GPU 0 idle VRAM: 23848 MiB ⚠ something is using this GPU (display, browser, container)
• GPU 1 idle VRAM: 23848 MiB ⚠ something is using this GPU (display, browser, container)

Container runtime

• Docker: 29.5.0
• docker compose (v2): 5.1.3
• NVIDIA Container Toolkit: 1.19.0

Stack version

• club-3090: v0.8.0-17-g135f2c4 (branch: master, SHA 135f2c4)
• GENESIS_PIN default: 7b9fd319 (per scripts/setup.sh)
• Cached vLLM images:
  • tag nightly-bf610c2f56764e1b30bc6065f4ceace3d6e59036 digest sha256:7f047b7e625283eee436cfc0c37784064f75422452ed4f9b6fa8c69eae6afe68 (3 days ago)

Profile state

• Profile schema version: 1
• Profile counts: 9 hardware, 4 models, 5 workloads, 7 engines, 6 drafters
• Compose registry: 42 entries
• Canonical scenarios: 9
• Calibration:
  • gemma-4-26b-a4b: 2 rows
  • gemma-4-31b: 7 rows
  • qwen3.6-27b: 11 rows
  • qwen3.6-35b-a3b: 2 rows
• Active estate: none (~/.club3090/estate.yml not found)

KV math calibration

• Overall: 22/22 (100%)
• No FAIL rows. kv-calc projections should agree with measured VRAM within the ±1.5 GB error band.

Full kv-calc --calibration output

========================================================================================
Calibration — predicted per-card VRAM vs measured BENCHMARKS rows
========================================================================================

  Predicted = weights + activation + overhead + drafter + (KV capped at available).
  Budget = mem_util × VRAM. Measured = nvidia-smi peak during bench (target ≈ budget).
  Verdict ✓ iff PASS/TIGHT and measured < VRAM (boot OK).

== qwen3.6-27b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  dual                          19.91 GB   22.80 GB    23.60 GB    PASS ✓
  dual-turbo                    18.84 GB   22.80 GB    19.80 GB    PASS ✓
  dual-dflash                   18.04 GB   22.80 GB    23.60 GB    PASS ✓
  dual-dflash-noviz             22.80 GB   22.80 GB    23.80 GB   TIGHT ✓
  dual4                         15.73 GB   22.80 GB    23.50 GB    PASS ✓
  dual4-dflash                  16.18 GB   22.80 GB    22.00 GB    PASS ✓
  dual-dflash-noviz@175K        22.80 GB   22.80 GB    21.80 GB   TIGHT ✓
  long-text                     22.32 GB   22.32 GB    22.30 GB   TIGHT ✓
  long-vision                   22.12 GB   22.80 GB    23.00 GB    PASS ✓
  bounded-thinking              22.80 GB   22.80 GB    21.70 GB    PASS ✓
  minimal@64K                   21.60 GB   21.60 GB    22.40 GB   TIGHT ✓

  Verdict accuracy: 11/11 (100%)

== qwen3.6-35b-a3b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  qwen-a3b-preview              22.03 GB   22.08 GB    21.94 GB    PASS ✓
  qwen-a3b-preview-mtp          22.08 GB   22.08 GB    22.72 GB    PASS ✓

  Verdict accuracy: 2/2 (100%)

== gemma-4-31b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  gemma-dual-int8               22.80 GB   22.80 GB    22.20 GB   TIGHT ✓
  gemma-dual-int8-262k          22.80 GB   22.80 GB    22.10 GB    PASS ✓
  gemma-dual-dflash             22.08 GB   22.08 GB    22.70 GB   TIGHT ✓
  gemma-dual-dflash             22.08 GB   22.08 GB    22.30 GB   TIGHT ✓
  gemma-dual-awq                20.40 GB   20.40 GB    19.80 GB   TIGHT ✓
  gemma-dual                    22.08 GB   22.08 GB    22.50 GB   TIGHT ✓
  gemma-dual-int8@256K          22.80 GB   22.80 GB    22.50 GB   TIGHT ✓

  Verdict accuracy: 7/7 (100%)

== gemma-4-26b-a4b ==
  compose                     predicted    budget   measured  verdict
  ─────────────────────────   ─────────  ────────  ─────────  ───────
  gemma-a4b-awq                 22.08 GB   22.08 GB    23.45 GB   TIGHT ✓
  gemma-a4b-awq-mtp             22.08 GB   22.08 GB    23.50 GB   TIGHT ✓

  Verdict accuracy: 2/2 (100%)

Overall: 22/22 (100%)

Notes:
  - This is a directional estimator (±1.5 GB error band on the breakdown).
  - vLLM's `gpu_worker.py` boot log is the authoritative source.
  - If predicted PASS but measured > budget, file an issue with `scripts/report.sh --bench`.

Active container

• Name: vllm-qwen36-27b-dual
• Engine: vllm
• Status: Up 4 hours
• Ports: 0.0.0.0:8010->8000/tcp
• Image: vllm/vllm-openai:nightly-bf610c2f56764e1b30bc6065f4ceace3d6e59036

Container Python / CUDA versions

• PyTorch: torch=2.11.0+cu130 torch_cuda_build=13.0 cudnn=91900
• vLLM: 0.20.2rc1.dev371+gbf610c2f5
• nvidia-smi inside container:

  0, NVIDIA GeForce RTX 3090, 590.48.01
  1, NVIDIA GeForce RTX 3090, 590.48.01
  

Boot log highlights

KV pool sizing:

(Worker_TP0 pid=99) INFO 05-18 11:07:12 [gpu_worker.py:462] Available KV cache memory: 10.92 GiB
(EngineCore pid=90) INFO 05-18 11:07:12 [kv_cache_utils.py:1730] GPU KV cache size: 616,550 tokens
(EngineCore pid=90) INFO 05-18 11:07:12 [kv_cache_utils.py:1731] Maximum concurrency for 262,144 tokens per request: 2.35x

Engine config (CLI flags + engine init):

(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:240] non-default args: {'model_tag': '~/.cache/huggingface/qwen3.6-27b-autoround-int4', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '~/.cache/huggingface/qwen3.6-27b-autoround-int4', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 262144, 'quantization': 'auto_round', 'served_model_name': ['qwen3.6-27b-autoround'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'disable_custom_all_reduce': True, 'kv_cache_dtype': 'fp8_e5m2', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(EngineCore pid=107) INFO 05-17 22:11:57 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev371+gbf610c2f5) with config: model='~/.cache/huggingface/qwen3.6-27b-autoround-int4', speculative_config=SpeculativeConfig(method='mtp', model='~/.cache/huggingface/qwen3.6-27b-autoround-int4', num_spec_tokens=3), tokenizer='~/.cache/huggingface/qwen3.6-27b-autoround-int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e5m2, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.6-27b-autoround, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')

Recent warnings/errors (last 5):

WARNING 05-17 22:09:59 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in a future version.
WARNING 05-18 09:21:25 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in a future version.
WARNING 05-18 11:05:21 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in a future version.

Full boot log (first 200 lines)

First 200 lines of docker logs

[club3090/pr35936] chat_completion/serving.py installed from /etc/club3090/pr35936-chat-completion-serving.py
[club3090/pr35936] engine/serving.py installed from /etc/club3090/pr35936-engine-serving.py
[club3090/pr41800] /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/utils.py: upstream get_max_tokens() already accepts truncate_prompt_tokens; no-op
[club3090/pr41800] install complete
[nvlink] detected NVLink (NV4) between GPU0-GPU1 — enabling NVLink mode
[nvlink] NVLink ENABLED — NCCL_P2P_LEVEL=NVL, custom all-reduce ON, expandable_segments OFF
WARNING 05-17 22:09:59 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in a future version.
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]        █     █     █▄   ▄█
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.2rc1.dev371+gbf610c2f5
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]   █▄█▀ █     █     █     █  model   ~/.cache/huggingface/qwen3.6-27b-autoround-int4
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:306]
(APIServer pid=1) INFO 05-17 22:09:59 [utils.py:240] non-default args: {'model_tag': '~/.cache/huggingface/qwen3.6-27b-autoround-int4', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '~/.cache/huggingface/qwen3.6-27b-autoround-int4', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 262144, 'quantization': 'auto_round', 'served_model_name': ['qwen3.6-27b-autoround'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'disable_custom_all_reduce': True, 'kv_cache_dtype': 'fp8_e5m2', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(APIServer pid=1) WARNING 05-17 22:09:59 [envs.py:1872] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer pid=1) WARNING 05-17 22:09:59 [envs.py:1872] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer pid=1) WARNING 05-17 22:09:59 [envs.py:1872] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer pid=1) WARNING 05-17 22:09:59 [envs.py:1872] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer pid=1) INFO 05-17 22:10:33 [model.py:569] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1) WARNING 05-17 22:10:33 [model.py:2038] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 05-17 22:10:33 [model.py:1700] Using max model len 262144
(APIServer pid=1) INFO 05-17 22:10:35 [cache.py:261] Using fp8_e5m2 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1) INFO 05-17 22:10:58 [model.py:569] Resolved architecture: Qwen3_5MTP
(APIServer pid=1) WARNING 05-17 22:10:58 [model.py:2038] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 05-17 22:10:58 [model.py:1700] Using max model len 262144
(APIServer pid=1) WARNING 05-17 22:10:58 [speculative.py:709] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=1) INFO 05-17 22:10:58 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) WARNING 05-17 22:10:58 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 05-17 22:10:58 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=1) INFO 05-17 22:10:58 [vllm.py:968] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 05-17 22:10:58 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=107) INFO 05-17 22:11:57 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev371+gbf610c2f5) with config: model='~/.cache/huggingface/qwen3.6-27b-autoround-int4', speculative_config=SpeculativeConfig(method='mtp', model='~/.cache/huggingface/qwen3.6-27b-autoround-int4', num_spec_tokens=3), tokenizer='~/.cache/huggingface/qwen3.6-27b-autoround-int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e5m2, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.6-27b-autoround, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=107) INFO 05-17 22:11:57 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.18.0.2 (local), world_size=2, local_world_size=2
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=116) INFO 05-17 22:12:26 [parallel_state.py:1410] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:50797 backend=nccl
(Worker pid=117) INFO 05-17 22:12:26 [parallel_state.py:1410] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:50797 backend=nccl
(Worker pid=116) INFO 05-17 22:12:28 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=116) WARNING 05-17 22:12:29 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=117) WARNING 05-17 22:12:29 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=116) INFO 05-17 22:12:29 [parallel_state.py:1723] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker pid=116) INFO 05-17 22:12:30 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(Worker pid=117) WARNING 05-17 22:12:30 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=116) WARNING 05-17 22:12:30 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=116) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker pid=117) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker_TP0 pid=116) INFO 05-17 22:12:54 [gpu_model_runner.py:4970] Starting to load model ~/.cache/huggingface/qwen3.6-27b-autoround-int4...
(Worker_TP0 pid=116) INFO 05-17 22:12:56 [cuda.py:427] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=116) INFO 05-17 22:12:56 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=116) INFO 05-17 22:12:56 [auto_gptq.py:353] Using MarlinLinearKernel for AutoGPTQLinearMethod
(Worker_TP0 pid=116) INFO 05-17 22:12:56 [gdn_linear_attn.py:169] Using Triton/FLA GDN prefill kernel
(Worker_TP1 pid=117) INFO 05-17 22:12:56 [auto_gptq.py:353] Using MarlinLinearKernel for AutoGPTQLinearMethod
(Worker_TP0 pid=116) INFO 05-17 22:12:56 [cuda.py:372] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(Worker_TP0 pid=116) INFO 05-17 22:12:58 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.69 GiB. Available RAM: 56.15 GiB.
(Worker_TP0 pid=116) INFO 05-17 22:12:58 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   9% Completed | 1/11 [00:00<00:08,  1.14it/s]
Loading safetensors checkpoint shards:  18% Completed | 2/11 [00:01<00:08,  1.12it/s]
Loading safetensors checkpoint shards:  27% Completed | 3/11 [00:02<00:07,  1.05it/s]
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:03<00:06,  1.04it/s]
Loading safetensors checkpoint shards:  45% Completed | 5/11 [00:04<00:05,  1.02it/s]
Loading safetensors checkpoint shards:  55% Completed | 6/11 [00:05<00:05,  1.00s/it]
Loading safetensors checkpoint shards:  64% Completed | 7/11 [00:06<00:03,  1.04it/s]
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:10<00:05,  1.85s/it]
Loading safetensors checkpoint shards:  91% Completed | 10/11 [00:14<00:01,  1.84s/it]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:14<00:00,  1.29s/it]
(Worker_TP0 pid=116)
(Worker_TP0 pid=116) INFO 05-17 22:13:12 [default_loader.py:397] Loading weights took 14.20 seconds
(Worker_TP1 pid=117) INFO 05-17 22:13:12 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=116) INFO 05-17 22:13:13 [gpu_model_runner.py:4994] Loading drafter model...
(Worker_TP0 pid=116) INFO 05-17 22:13:13 [vllm.py:968] Asynchronous scheduling is enabled.
(Worker_TP0 pid=116) INFO 05-17 22:13:13 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=116) INFO 05-17 22:13:13 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.69 GiB. Available RAM: 56.06 GiB.
Loading safetensors checkpoint shards:   0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  36% Completed | 4/11 [00:00<00:00, 30.55it/s]
Loading safetensors checkpoint shards:  73% Completed | 8/11 [00:03<00:01,  1.75it/s]
Loading safetensors checkpoint shards:  91% Completed | 10/11 [00:07<00:00,  1.05it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:07<00:00,  1.20it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:07<00:00,  1.39it/s]
(Worker_TP0 pid=116)
(Worker_TP0 pid=116) INFO 05-17 22:13:21 [default_loader.py:397] Loading weights took 7.93 seconds
(Worker_TP1 pid=117) INFO 05-17 22:13:28 [llm_base_proposer.py:1321] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP1 pid=117) INFO 05-17 22:13:28 [llm_base_proposer.py:1377] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP0 pid=116) INFO 05-17 22:13:28 [llm_base_proposer.py:1321] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP0 pid=116) INFO 05-17 22:13:28 [llm_base_proposer.py:1377] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP1 pid=117) INFO 05-17 22:13:29 [interface.py:649] Setting attention block size to 1600 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP1 pid=117) INFO 05-17 22:13:29 [interface.py:673] Padding mamba page size by 0.25% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=116) INFO 05-17 22:13:30 [gpu_model_runner.py:5085] Model loading took 8.93 GiB memory and 32.629689 seconds
(Worker_TP0 pid=116) INFO 05-17 22:13:30 [interface.py:649] Setting attention block size to 1600 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP0 pid=116) INFO 05-17 22:13:30 [interface.py:673] Padding mamba page size by 0.25% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=116) INFO 05-17 22:13:30 [gpu_model_runner.py:6054] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=116) INFO 05-17 22:14:26 [backends.py:1089] Using cache directory: ~/.cache/vllm/torch_compile_cache/6f497d7388/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=116) INFO 05-17 22:14:26 [backends.py:1148] Dynamo bytecode transform time: 49.15 s
(EngineCore pid=107) INFO 05-17 22:14:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=116) INFO 05-17 22:14:34 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(EngineCore pid=107) INFO 05-17 22:15:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=107) INFO 05-17 22:16:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=116) INFO 05-17 22:16:36 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 126.48 s
(EngineCore pid=107) INFO 05-17 22:17:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=116) INFO 05-17 22:18:05 [decorators.py:708] saved AOT compiled function to ~/.cache/vllm/torch_compile_cache/torch_aot_compile/11e113ea8a8071477ca5afbfaf3c588bd125dea86d4c31375581cd47c2877055/rank_0_0/model
(Worker_TP0 pid=116) INFO 05-17 22:18:05 [monitor.py:53] torch.compile took 268.05 s in total
(EngineCore pid=107) INFO 05-17 22:18:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=107) INFO 05-17 22:19:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=107) INFO 05-17 22:20:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=107) INFO 05-17 22:21:31 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=116) INFO 05-17 22:21:39 [monitor.py:81] Initial profiling/warmup run took 214.20 s
(Worker_TP0 pid=116) INFO 05-17 22:21:42 [backends.py:1089] Using cache directory: ~/.cache/vllm/torch_compile_cache/6f497d7388/rank_0_0/eagle_head for vLLM's torch.compile
(Worker_TP0 pid=116) INFO 05-17 22:21:42 [backends.py:1148] Dynamo bytecode transform time: 2.53 s
(Worker_TP0 pid=116) INFO 05-17 22:22:13 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 31.21 s
(Worker_TP0 pid=116) INFO 05-17 22:22:14 [decorators.py:708] saved AOT compiled function to ~/.cache/vllm/torch_compile_cache/torch_aot_compile/5c5fc15e1b0553edecf7965769dd7947d8695eca3d28d827165bd64abb729597/rank_0_0/model
(Worker_TP0 pid=116) INFO 05-17 22:22:14 [monitor.py:53] torch.compile took 35.08 s in total
(Worker_TP0 pid=116) INFO 05-17 22:22:17 [monitor.py:81] Initial profiling/warmup run took 2.45 s
(Worker_TP0 pid=116) WARNING 05-17 22:22:23 [kv_cache_utils.py:1154] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP0 pid=116) WARNING 05-17 22:22:23 [compilation.py:1407] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP1 pid=117) WARNING 05-17 22:22:23 [kv_cache_utils.py:1154] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP1 pid=117) WARNING 05-17 22:22:23 [compilation.py:1407] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP0 pid=116) INFO 05-17 22:22:23 [gpu_model_runner.py:6197] Profiling CUDA graph memory: PIECEWISE=5 (largest=16)
(Worker_TP1 pid=117) INFO 05-17 22:22:23 [gpu_model_runner.py:6197] Profiling CUDA graph memory: PIECEWISE=5 (largest=16)
(Worker_TP1 pid=117) INFO 05-17 22:22:28 [gpu_model_runner.py:6276] Estimated CUDA graph memory: 0.19 GiB total
(Worker_TP0 pid=116) INFO 05-17 22:22:28 [gpu_model_runner.py:6276] Estimated CUDA graph memory: 0.19 GiB total
(Worker_TP1 pid=117) INFO 05-17 22:22:29 [gpu_worker.py:477] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9118 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9282. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=116) INFO 05-17 22:22:29 [gpu_worker.py:462] Available KV cache memory: 10.92 GiB
(Worker_TP0 pid=116) INFO 05-17 22:22:29 [gpu_worker.py:477] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9118 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9282. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=107) WARNING 05-17 22:22:29 [kv_cache_utils.py:1154] Add 3 padding layers, may waste at most 6.25% KV cache memory
(EngineCore pid=107) INFO 05-17 22:22:29 [kv_cache_utils.py:1730] GPU KV cache size: 615,086 tokens
(EngineCore pid=107) INFO 05-17 22:22:29 [kv_cache_utils.py:1731] Maximum concurrency for 262,144 tokens per request: 2.35x
(Worker_TP0 pid=116) INFO 05-17 22:22:29 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP1 pid=117) INFO 05-17 22:22:29 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:01<00:00,  2.52it/s]
(Worker_TP0 pid=116) INFO 05-17 22:22:34 [gpu_model_runner.py:6367] Graph capturing finished in 5 secs, took 0.13 GiB
(Worker_TP0 pid=116) INFO 05-17 22:22:34 [gpu_worker.py:621] CUDA graph pool memory: 0.13 GiB (actual), 0.19 GiB (estimated), difference: 0.06 GiB (43.5%).
(Worker_TP1 pid=117) INFO 05-17 22:22:34 [gpu_worker.py:621] CUDA graph pool memory: 0.13 GiB (actual), 0.19 GiB (estimated), difference: 0.06 GiB (43.5%).
(Worker_TP0 pid=116) INFO 05-17 22:22:34 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=117) INFO 05-17 22:22:34 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=107) INFO 05-17 22:22:34 [core.py:299] init engine (profile, create kv cache, warmup model) took 544.47 s (compilation: 306.23 s)
(EngineCore pid=107) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=107) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=107) INFO 05-17 22:23:06 [vllm.py:968] Asynchronous scheduling is enabled.
(EngineCore pid=107) INFO 05-17 22:23:06 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) INFO 05-17 22:23:06 [api_server.py:599] Supported tasks: ['generate']
(APIServer pid=1) INFO 05-17 22:23:07 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) WARNING 05-17 22:23:07 [model.py:1457] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 05-17 22:23:07 [hf.py:488] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) INFO 05-17 22:23:56 [base.py:224] Multi-modal warmup completed in 49.046s
(APIServer pid=1) INFO 05-17 22:24:03 [base.py:224] Readonly multi-modal warmup completed in 6.599s
(APIServer pid=1) INFO 05-17 22:24:03 [api_server.py:603] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:37] Available routes are:
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 05-17 22:24:03 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.
(APIServer pid=1) INFO:     172.18.0.1:36980 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.18.0.1:36990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=116) WARNING 05-18 07:50:32 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:50:33 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:50:34 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _copy_page_indices_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=107) INFO 05-18 07:51:32 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=116) WARNING 05-18 07:52:26 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:28 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:29 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:29 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_step_slot_mapping_metadata_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:33 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:35 [jit_monitor.py:103] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO 05-18 07:52:35 [loggers.py:271] Engine 000: Avg prompt throughput: 2.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(Worker_TP0 pid=116) WARNING 05-18 07:52:36 [jit_monitor.py:103] Triton kernel JIT compilation during inference: expand_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0 pid=116) WARNING 05-18 07:52:37 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_inputs_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO 05-18 07:52:45 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-18 07:52:45 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.98, Accepted throughput: 0.00 tokens/s, Drafted throughput: 0.00 tokens/s, Accepted: 111 tokens, Drafted: 168 tokens, Per-position acceptance rate: 0.804, 0.661, 0.518, Avg Draft acceptance rate: 66.1%
(APIServer pid=1) INFO 05-18 07:52:55 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-18 07:52:55 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.66, Accepted throughput: 11.80 tokens/s, Drafted throughput: 21.30 tokens/s, Accepted: 118 tokens, Drafted: 213 tokens, Per-position acceptance rate: 0.789, 0.535, 0.338, Avg Draft acceptance rate: 55.4%
(APIServer pid=1) INFO 05-18 07:53:05 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%

Recent failed boot attempts

No recently-exited vLLM or llama.cpp containers found.

verify-full.sh output

verify-full output

[autodetect] using running container=vllm-qwen36-27b-dual url=http://localhost:8010  (skip: PREFLIGHT_NO_AUTODETECT=1)
Running FULL functional test against http://localhost:8010
  model=qwen3.6-27b-autoround  container=vllm-qwen36-27b-dual  engine=vllm

[1/8] Server reachable on /v1/models ...
  ✓ server is serving
[2/8] Genesis patches applied ...
  ⊘ no Genesis marker in logs (container restarted, or Genesis not loaded) (skipped)
[warmup] priming engine (cold cudagraph/JIT, up to 180s, not scored) ...
[warmup] engine warm
[3/8] Basic completion — capital of France ...
  ✓ reply contains 'Paris'
[4/8] Tool calling ...
  ✓ tool_calls[] populated with get_weather
[5/8] Streaming (SSE) ...
  ✓ streamed 32 chunks, 276 chars:  A three-sentence haiku is structurally impossible, as a haiku is a single poem c...
[6/8] Thinking / reasoning mode ...
  ✓ reasoning 503 chars, content 3 chars (finish=stop)
    reasoning: Here's a thinking process:  1.  **Analyze User Input:**    -...
    content:     4...
[7/8] Output quality / cascade detection (2K-token completion) ...
  ✓ output OK — 9782 chars, variety=0.630, max_line_repeat=0, finish=length
[8/8] MTP acceptance length threshold ...
  ✓ MTP acceptance length = 2.68 (>=2.0 — spec-decode contributing)

All checks passed. Stack is ready for full-functionality use.

verify-stress.sh output

verify-stress output (7 boundary checks incl. Cliff 2 needle recall)

[autodetect] using running container=vllm-qwen36-27b-dual url=http://localhost:8010  (skip: PREFLIGHT_NO_AUTODETECT=1)
Running STRESS / boundary test against http://localhost:8010
  model=qwen3.6-27b-autoround  container=vllm-qwen36-27b-dual  engine=vllm
  This script does the heavy stuff (longctx needle ladder + ~25K-token tool prefill).
  For the fast functional smoke (~2 min), use verify-full.sh instead.

[1/7] Long-context needle small rungs (10K / 30K) ...
    ✓   9818 tokens: recalled 'amber otter 48' (got: amber otter 48 )
    ✓  29319 tokens: recalled 'crimson falcon 12' (got: crimson falcon 12 )
  ✓ all long-ctx depths recalled secret correctly
[2/7] Tool response prefill OOM (~25K-token mock tool response) ...
  ✓ tool prefill OK — text response (814 chars, finish=stop)
[3/7] IDE-agent one-shot prompt (sys + tool schemas + user request) ...
  ✓ IDE-agent one-shot OK — 48 completion tokens (209 chars), finish=stop
[4/7] Multi-turn agent prompt (sys + tools + 4-turn history) ...
  ✓ multi-turn agent OK
[5/7] LCB-coding shape (LeetCode-style problem + structured plan) ...
  ✓ LCB-coding shape OK
[6/7] Reasoning-heavy (math problem + max_tokens=8192) ...
  ✓ reasoning-heavy OK — 4684 completion tokens
[7/7] Long-context needle large rungs (60K / 90K — Cliff 2 territory) ...
    ✓  58569 tokens: recalled 'silver chinchilla 63' (got: silver chinchilla 63 )
    ✓  91070 tokens: recalled 'turquoise capybara 35' (got: turquoise capybara 35 )
  ✓ all long-ctx depths recalled secret correctly

All stress / boundary checks passed. KV-cache and prefill paths are sound for the deployed config.

soak-test.sh (SOAK_MODE=continuous) output

soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)

[soak] running soak test against http://localhost:8010 (model=qwen3.6-27b-autoround, container=vllm-qwen36-27b-dual)
[soak] mode=continuous sessions=5 turns=5 max_growth=200MiB timeout=1800s
[soak] output=results/report-soak-20260518-145326
[soak] session 1/5
[soak]   turn 1/5: status=200 wall=839ms ttft=514ms decode_tps=89.034 vram=47696MiB
[soak]   turn 2/5: status=200 wall=3377ms ttft=3046ms decode_tps=84.751 vram=47696MiB
[soak]   turn 3/5: status=200 wall=5194ms ttft=4683ms decode_tps=76.317 vram=47696MiB
[soak]   turn 4/5: status=200 wall=6257ms ttft=5618ms decode_tps=68.884 vram=47696MiB
[soak]   turn 5/5: status=200 wall=21846ms ttft=5470ms decode_tps=46.104 vram=47696MiB
[soak] warm baseline after session 1: 47696 MiB
[soak] session 2/5
[soak]   turn 1/5: status=200 wall=798ms ttft=498ms decode_tps=96.666 vram=47696MiB
[soak]   turn 2/5: status=200 wall=1785ms ttft=1483ms decode_tps=92.829 vram=47696MiB
[soak]   turn 3/5: status=200 wall=1953ms ttft=1468ms decode_tps=80.458 vram=47696MiB
[soak]   turn 4/5: status=200 wall=1967ms ttft=1361ms decode_tps=72.721 vram=47696MiB
[soak]   turn 5/5: status=200 wall=15830ms ttft=1882ms decode_tps=50.975 vram=47696MiB
[soak] session 3/5
[soak]   turn 1/5: status=200 wall=827ms ttft=509ms decode_tps=91.219 vram=47696MiB
[soak]   turn 2/5: status=200 wall=1810ms ttft=1492ms decode_tps=88.153 vram=47696MiB
[soak]   turn 3/5: status=200 wall=1988ms ttft=1470ms decode_tps=75.375 vram=47696MiB
[soak]   turn 4/5: status=200 wall=2006ms ttft=1368ms decode_tps=68.987 vram=47696MiB
[soak]   turn 5/5: status=200 wall=17123ms ttft=5462ms decode_tps=48.451 vram=47696MiB
[soak] session 4/5
[soak]   turn 1/5: status=200 wall=823ms ttft=502ms decode_tps=90.432 vram=47696MiB
[soak]   turn 2/5: status=200 wall=1812ms ttft=1491ms decode_tps=87.366 vram=47696MiB
[soak]   turn 3/5: status=200 wall=1980ms ttft=1469ms decode_tps=76.313 vram=47696MiB
[soak]   turn 4/5: status=200 wall=2004ms ttft=1368ms decode_tps=69.095 vram=47696MiB
[soak]   turn 5/5: status=200 wall=2943ms ttft=1877ms decode_tps=55.326 vram=47696MiB
[soak] session 5/5
[soak]   turn 1/5: status=200 wall=802ms ttft=499ms decode_tps=95.711 vram=47696MiB
[soak]   turn 2/5: status=200 wall=1797ms ttft=1483ms decode_tps=89.209 vram=47696MiB
[soak]   turn 3/5: status=200 wall=1945ms ttft=1464ms decode_tps=81.108 vram=47696MiB
[soak]   turn 4/5: status=200 wall=1970ms ttft=1362ms decode_tps=72.413 vram=47696MiB
[soak]   turn 5/5: status=200 wall=13070ms ttft=1877ms decode_tps=52.534 vram=47696MiB

[soak] summary
[soak]   verdict              PASS
[soak]   boot_vram_mib        47696
[soak]   max_vram_mib         47696
[soak]   max_growth_mib       0 / 200
[soak]   errors               0
[soak]   silent_empty         0 / 25 (0.0%)
[soak]   p50_decode_tps       76.32
[soak]   p95_ttft_ms          5468
[soak]   tps_retention        100.0%
[soak]   note                 PASS = no failure signal on this sample;
[soak]                        not patch validation (topology alone can
[soak]                        sidestep what overlays target). See
[soak]                        scripts/soak-test.sh --help and docs/CLIFFS.md.
[soak] artifacts: results/report-soak-20260518-145326

Soak summary (results/report-soak-20260518-145326/summary.md):

Soak test summary

• Verdict: PASS
• Boot VRAM baseline: 47696 MiB
• Max VRAM observed: 47696 MiB
• Max growth observed: 0 MiB
• Sessions completed: 5
• Request errors: 0
• Silent-empty turns (HTTP 200 + 0 completion tokens): 0 / 25 (0.0%)

Metric	Value
p50 decode TPS	76.32
p95 decode TPS	95.13
first-5 median TPS	76.32
last-5 median TPS	76.32
TPS retention	100.0%
p50 TTFT	1470 ms
p95 TTFT	5468 ms
TTFT first/last ratio	1.00x
VRAM oscillation	0 MiB

Recommendation

• Runtime VRAM growth and throughput retention stayed within v1 soak thresholds.

bench.sh output

bench output (3 warmups + 5 measured per prompt)

[autodetect] using running container=vllm-qwen36-27b-dual url=http://localhost:8010  (skip: PREFLIGHT_NO_AUTODETECT=1)

========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
  warm-1     wall= 23.36s  ttft=   185ms  toks=1000  wall_TPS= 42.80  decode_TPS= 43.15
  warm-2     wall= 23.15s  ttft=   176ms  toks=1000  wall_TPS= 43.20  decode_TPS= 43.53
  warm-3     wall= 23.85s  ttft=   232ms  toks=1000  wall_TPS= 41.92  decode_TPS= 42.33

=== measured (5) ===
  run-1      wall= 22.04s  ttft=   230ms  toks=1000  wall_TPS= 45.37  decode_TPS= 45.85
  run-2      wall= 23.47s  ttft=   234ms  toks=1000  wall_TPS= 42.60  decode_TPS= 43.03
  run-3      wall= 23.86s  ttft=   229ms  toks=1000  wall_TPS= 41.92  decode_TPS= 42.33
  run-4      wall= 23.56s  ttft=   178ms  toks=1000  wall_TPS= 42.44  decode_TPS= 42.77
  run-5      wall= 24.41s  ttft=   235ms  toks=1000  wall_TPS= 40.97  decode_TPS= 41.36

=== summary [narrative] (n=5) ===
  wall_TPS       mean=  42.66   std=  1.65   CV= 3.9%   min=40.97   max=45.37
  decode_TPS     mean=  43.07   std=  1.68   CV= 3.9%   min=41.36   max=45.85
  TTFT          mean=   221ms  std=   24ms  min=178ms  max=235ms
  PP tok/s       mean=   1.00   std=  1.37   CV=136.9%   min=0.00   max=2.50

========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
  warm-1     wall= 15.73s  ttft=   202ms  toks= 800  wall_TPS= 50.86  decode_TPS= 51.52
  warm-2     wall= 14.83s  ttft=   233ms  toks= 759  wall_TPS= 51.17  decode_TPS= 51.99
  warm-3     wall= 13.95s  ttft=   235ms  toks= 750  wall_TPS= 53.77  decode_TPS= 54.69

=== measured (5) ===
  run-1      wall= 14.39s  ttft=   231ms  toks= 748  wall_TPS= 51.98  decode_TPS= 52.83
  run-2      wall=  9.18s  ttft=   231ms  toks= 490  wall_TPS= 53.40  decode_TPS= 54.78
  run-3      wall= 15.24s  ttft=   230ms  toks= 800  wall_TPS= 52.50  decode_TPS= 53.30
  run-4      wall= 11.30s  ttft=   177ms  toks= 606  wall_TPS= 53.64  decode_TPS= 54.50
  run-5      wall=  8.97s  ttft=   231ms  toks= 481  wall_TPS= 53.62  decode_TPS= 55.04

=== summary [code] (n=5) ===
  wall_TPS       mean=  53.03   std=  0.75   CV= 1.4%   min=51.98   max=53.64
  decode_TPS     mean=  54.09   std=  0.97   CV= 1.8%   min=52.83   max=55.04
  TTFT          mean=   220ms  std=   24ms  min=177ms  max=231ms
  PP tok/s       mean=   2.00   std=  1.12   CV=55.9%   min=0.00   max=2.50

=== GPU state ===
0, 81 %, 23848 MiB, 24576 MiB, 242.69 W, 66
1, 37 %, 23848 MiB, 24576 MiB, 206.87 W, 69

=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 05-18 15:00:01 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.35, Accepted throughput: 36.00 tokens/s, Drafted throughput: 45.90 tokens/s, Accepted: 360 tokens, Drafted: 459 tokens, Per-position acceptance rate: 0.915, 0.797, 0.641, Avg Draft acceptance rate: 78.4%
(APIServer pid=1) INFO 05-18 15:00:11 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.46, Accepted throughput: 37.90 tokens/s, Drafted throughput: 46.20 tokens/s, Accepted: 379 tokens, Drafted: 462 tokens, Per-position acceptance rate: 0.948, 0.838, 0.675, Avg Draft acceptance rate: 82.0%
(APIServer pid=1) INFO 05-18 15:00:21 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.46, Accepted throughput: 37.60 tokens/s, Drafted throughput: 45.90 tokens/s, Accepted: 376 tokens, Drafted: 459 tokens, Per-position acceptance rate: 0.928, 0.837, 0.693, Avg Draft acceptance rate: 81.9%

---

Generated by bash scripts/report.sh. Flags: --verify (verify-full), --stress (verify-stress 7/7 incl. Cliff 2 needles), --soak (SOAK_MODE=continuous, catches Cliff 2b), --bench (canonical TPS), --full (all four, ~35 min). Use --no-redact to disable redaction (internal sharing only).

[noonghunna]

Thanks for the full report.sh — this one has a clear root cause in it.

Your PCIe link is training at Gen1, not Gen3. From your own report (lspci -vvv for both GPUs):

0000:02:00.0  LnkCap: Speed 8GT/s, Width x16
              LnkSta: Speed 2.5GT/s (downgraded), Width x16
0000:04:00.0  LnkCap: Speed 8GT/s, Width x16
              LnkSta: Speed 2.5GT/s (downgraded), Width x16

2.5 GT/s is PCIe Gen1 — one quarter of the PCIe 3.0 (8 GT/s) bandwidth your cards and slots are capable of. dual.yml runs tensor-parallel (TP=2) with an all-reduce every layer, and on the PCIe-only path that all-reduce is bandwidth-bound. A Gen3→Gen1 link is the dominant reason you're at 41/55 instead of the ~69/89 dual.yml reference (which is measured on a PCIe-only Gen3 x16 rig). The 260 W power cap is not a factor — you correctly observed it makes no difference; 3090 decode isn't power-bound.

This Gen1 down-train is a known Proxmox VFIO-passthrough behavior. Things to check, in order:

1. VM machine type + PCIe modeling. Use q35 (not i440fx) and mark the passthrough as PCIe, e.g. in the VM conf: hostpci0: 0000:02:00,pcie=1 and hostpci1: 0000:04:00,pcie=1, machine: q35. Legacy PCI modeling caps the guest link at Gen1 regardless of the physical slot.
2. Dell T630 BIOS. Set the PCIe slot speed explicitly to Gen3 (often "Auto", which can mis-train under passthrough), and set the system profile to Performance / disable PCIe ASPM.
3. Host kernel cmdline: intel_iommu=on iommu=pt pcie_aspm=off (add pci=realloc if the link still won't train).
4. Re-verify before benching: inside the VM, nvidia-smi -q | grep -A3 "GPU Link Info" (or the same lspci LnkSta) must show 8 GT/s, not 2.5 GT/s. Then re-run the canonical bench.

Once the link is at Gen3, two secondary items for the remaining gap — both are the lesson from #161 (NUMA-alignment + vCPU pinning moved that rig 20→60 tok/s):

• Dual-socket NUMA. Your T630 is 2× E5-2695 v4 (two NUMA nodes), but the VM only exposes 8 vCPUs on node 0. Pin the VM's vCPUs and memory to the NUMA node the GPUs physically sit on, and give it more than 8 vCPUs.
• VM RAM. The guest sees only ~31 GiB of your 256 GB. That won't cap decode TPS but is tight for model load / page cache — worth raising.

On NVLink: your report shows it is bonded and engaged (NV4, ~14 GB/s/link), but dual.yml deliberately runs PCIe-only (custom all-reduce off). Per #161, NVLink under VM passthrough is fragile and frequently lands below a healthy PCIe-only Gen3 baseline — so please fix the Gen1 down-train first; that should get you to/near 69/89 on the PCIe-only path. Treat NVLink as a separate experiment afterward, not the fix.

(Minor: the report's "⚠ something is using this GPU — 23848 MiB" is a false positive here — that's just your vllm-qwen36-27b-dual workers resident, expected.)

docs/CONTAINER_RUNTIMES.md ("Proxmox passthrough performance") has the broader context. Once you've retrained the link to Gen3 and re-run the bench, post the numbers and we'll add a credited BENCHMARKS row.

[duart]

Thank you for your feedback :-)

I have checked your suggestions and I discovered that:

• LnkSta: Speed 2.5GT/s (downgraded), Width x16 was due to GPU (or whatever can do it) slowing PCI bandwidth down under no load
  When inference runs, it switches to 8.0GT/s.
• VM's vCPUs and memory are on the same NUMA node that the GPUs physically sit on
• 8 vCPUs are enough as only a few of them are really used (vLLM uses 3 cores — 2 for workers, 1 for admin tasks in dual configuration)

I made a number of experiments and configuration (simplified) that works best for me now is below.

Important changes:

• Image: v0.21.1rc1.dev132+g4f940896a (current vllm/vllm-openai:nightly)
• MTP FULL_AND_PIECEWISE works in this image (it gives +50% tokens/second vs. PIECEWISE only)
• I removed MTP= ... from config as it was slowing down in my case
• FLASH_ATTN is selected by the image (not FlashInfer)
• PR #41800 and PR #35936 (merged upstream into this image earlier)

Benchmark:
In short, I'm now at 78/78 narrative/code, still a bit lower in code but better at narrative vs original configuration

URL=http://localhost:8010 CONTAINER=vllm-qwen36-27b-dual ./scripts/bench.sh && ./scripts/verify-stress.sh
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 13.25s ttft= 210ms toks=1000 wall_TPS= 75.48 decode_TPS= 76.70
warm-2 wall= 12.86s ttft= 205ms toks=1000 wall_TPS= 77.79 decode_TPS= 79.05
warm-3 wall= 12.89s ttft= 209ms toks=1000 wall_TPS= 77.57 decode_TPS= 78.85

=== measured (5) ===
run-1 wall= 13.07s ttft= 206ms toks=1000 wall_TPS= 76.48 decode_TPS= 77.71
run-2 wall= 13.03s ttft= 230ms toks=1000 wall_TPS= 76.77 decode_TPS= 78.15
run-3 wall= 13.01s ttft= 244ms toks=1000 wall_TPS= 76.84 decode_TPS= 78.31
run-4 wall= 12.99s ttft= 235ms toks=1000 wall_TPS= 76.97 decode_TPS= 78.39
run-5 wall= 13.29s ttft= 258ms toks=1000 wall_TPS= 75.26 decode_TPS= 76.75

=== summary [narrative] (n=5) ===
wall_TPS mean= 76.46 std= 0.70 CV= 0.9% min=75.26 max=76.97
decode_TPS mean= 77.86 std= 0.67 CV= 0.9% min=76.75 max=78.39
TTFT mean= 235ms std= 19ms min=206ms max=258ms
PP tok/s mean= 1.50 std= 1.37 CV=91.3% min=0.00 max=2.50

========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 10.55s ttft= 236ms toks= 800 wall_TPS= 75.80 decode_TPS= 77.54
warm-2 wall= 9.15s ttft= 252ms toks= 678 wall_TPS= 74.12 decode_TPS= 76.22
warm-3 wall= 10.54s ttft= 245ms toks= 800 wall_TPS= 75.88 decode_TPS= 77.69

=== measured (5) ===
run-1 wall= 4.94s ttft= 250ms toks= 367 wall_TPS= 74.28 decode_TPS= 78.23
run-2 wall= 6.01s ttft= 269ms toks= 449 wall_TPS= 74.71 decode_TPS= 78.20
run-3 wall= 10.51s ttft= 267ms toks= 800 wall_TPS= 76.13 decode_TPS= 78.12
run-4 wall= 9.73s ttft= 243ms toks= 725 wall_TPS= 74.52 decode_TPS= 76.43
run-5 wall= 7.22s ttft= 243ms toks= 544 wall_TPS= 75.34 decode_TPS= 77.97

=== summary [code] (n=5) ===
wall_TPS mean= 75.00 std= 0.75 CV= 1.0% min=74.28 max=76.13
decode_TPS mean= 77.79 std= 0.77 CV= 1.0% min=76.43 max=78.23
TTFT mean= 254ms std= 13ms min=243ms max=269ms
PP tok/s mean= 3.00 std= 1.12 CV=37.3% min=2.50 max=5.00

=== GPU state ===
0, 37 %, 21478 MiB, 24576 MiB, 337.88 W, 75
1, 70 %, 21478 MiB, 24576 MiB, 343.64 W, 77

=== Last 3 SpecDecoding metrics ===
[autodetect] using running container=vllm-qwen36-27b-dual url=http://localhost:8010 (skip: PREFLIGHT_NO_AUTODETECT=1)
Running STRESS / boundary test against http://localhost:8010
model=qwen3.6-27b-autoround container=vllm-qwen36-27b-dual engine=vllm
This script does the heavy stuff (longctx needle ladder + ~25K-token tool prefill).
For the fast functional smoke (~2 min), use verify-full.sh instead.

[1/7] Long-context needle small rungs (10K / 30K) ...
✓ 9820 tokens: recalled 'crimson chinchilla 32' (got: crimson chinchilla 32 )
✓ 29318 tokens: recalled 'amber otter 53' (got: amber otter 53 )
✓ all long-ctx depths recalled secret correctly
[2/7] Tool response prefill OOM (~25K-token mock tool response) ...
✓ tool prefill OK — text response (636 chars, finish=stop)
[3/7] IDE-agent one-shot prompt (sys + tool schemas + user request) ...
✓ IDE-agent one-shot OK — 48 completion tokens (209 chars), finish=stop
[4/7] Multi-turn agent prompt (sys + tools + 4-turn history) ...
✓ multi-turn agent OK
[5/7] LCB-coding shape (LeetCode-style problem + structured plan) ...
✓ LCB-coding shape OK
[6/7] Reasoning-heavy (math problem + max_tokens=8192) ...
✓ reasoning-heavy OK — 4169 completion tokens
[7/7] Long-context needle large rungs (60K / 90K — Cliff 2 territory) ...
✓ 58568 tokens: recalled 'silver falcon 79' (got: silver falcon 79 )
✓ 91070 tokens: recalled 'crimson narwhal 81' (got: crimson narwhal 81 )
✓ all long-ctx depths recalled secret correctly

All stress / boundary checks passed. KV-cache and prefill paths are sound for the deployed config.
bash scripts/soak-test.sh --continuous
[soak] running soak test against http://localhost:8010 (model=qwen3.6-27b-autoround, container=vllm-qwen36-27b-dual)
[soak] mode=continuous sessions=5 turns=5 max_growth=200MiB timeout=1800s
[soak] output=results/soak-20260521-084728
[soak] session 1/5
[soak] turn 1/5: status=200 wall=749ms ttft=470ms decode_tps=104.221 vram=45948MiB
[soak] turn 2/5: status=200 wall=2809ms ttft=2540ms decode_tps=104.165 vram=45948MiB
[soak] turn 3/5: status=200 wall=3717ms ttft=3303ms decode_tps=94.182 vram=45948MiB
[soak] turn 4/5: status=200 wall=4620ms ttft=4148ms decode_tps=93.172 vram=45948MiB
[soak] turn 5/5: status=200 wall=12465ms ttft=4106ms decode_tps=75.485 vram=45948MiB
[soak] warm baseline after session 1: 45948 MiB
[soak] session 2/5
[soak] turn 1/5: status=200 wall=678ms ttft=399ms decode_tps=103.788 vram=45948MiB
[soak] turn 2/5: status=200 wall=1172ms ttft=870ms decode_tps=92.899 vram=45948MiB
[soak] turn 3/5: status=200 wall=1045ms ttft=629ms decode_tps=93.547 vram=45948MiB
[soak] turn 4/5: status=200 wall=1124ms ttft=588ms decode_tps=82.055 vram=45948MiB
[soak] turn 5/5: status=200 wall=18886ms ttft=4149ms decode_tps=75.119 vram=45948MiB
[soak] session 3/5
[soak] turn 1/5: status=200 wall=682ms ttft=402ms decode_tps=103.499 vram=45948MiB
[soak] turn 2/5: status=200 wall=898ms ttft=626ms decode_tps=103.205 vram=45948MiB
[soak] turn 3/5: status=200 wall=1055ms ttft=632ms decode_tps=92.292 vram=45948MiB
[soak] turn 4/5: status=200 wall=1300ms ttft=593ms decode_tps=62.22 vram=45948MiB
[soak] turn 5/5: status=200 wall=9441ms ttft=321ms decode_tps=73.58 vram=45948MiB
[soak] session 4/5
[soak] turn 1/5: status=200 wall=685ms ttft=404ms decode_tps=103.168 vram=45948MiB
[soak] turn 2/5: status=200 wall=912ms ttft=629ms decode_tps=98.688 vram=45948MiB
[soak] turn 3/5: status=200 wall=1058ms ttft=634ms decode_tps=91.854 vram=45948MiB
[soak] turn 4/5: status=200 wall=1221ms ttft=592ms decode_tps=69.934 vram=45948MiB
[soak] turn 5/5: status=200 wall=6262ms ttft=321ms decode_tps=66.664 vram=45948MiB
[soak] session 5/5
[soak] turn 1/5: status=200 wall=684ms ttft=403ms decode_tps=103.114 vram=45948MiB
[soak] turn 2/5: status=200 wall=899ms ttft=628ms decode_tps=103.189 vram=45948MiB
[soak] turn 3/5: status=200 wall=1056ms ttft=635ms decode_tps=92.64 vram=45948MiB
[soak] turn 4/5: status=200 wall=1402ms ttft=608ms decode_tps=55.461 vram=45948MiB
[soak] turn 5/5: status=200 wall=14959ms ttft=4201ms decode_tps=74.922 vram=45948MiB

[soak] summary
[soak] verdict PASS
[soak] boot_vram_mib 45948
[soak] max_vram_mib 45948
[soak] max_growth_mib 0 / 200
[soak] errors 0
[soak] silent_empty 0 / 25 (0.0%)
[soak] p50_decode_tps 92.90
[soak] p95_ttft_ms 4149
[soak] tps_retention 100.0%
[soak] note PASS = no failure signal on this sample;
[soak] not patch validation (topology alone can
[soak] sidestep what overlays target). See
[soak] scripts/soak-test.sh --help and docs/CLIFFS.md.
[soak] artifacts: results/soak-20260521-084728

docker-compose.yml
services:
vllm-qwen36-27b-dual:
image: vllm/vllm-openai:nightly
container_name: vllm-qwen36-27b-dual
restart: "no"
ports:
- "0.0.0.0:8010:8000"
volumes:
- /mnt/ramdisk/club3090-models:/root/.cache/huggingface
- /home/ubuntu/club-3090/models/qwen3.6-27b/vllm/cache/torch_compile:/root/.cache/vllm/torch_compile_cache
- /home/ubuntu/club-3090/models/qwen3.6-27b/vllm/cache/triton:/root/.triton/cache
- /home/ubuntu/club-3090/models/qwen3.6-27b/vllm/patches/froggeric-chat-template/chat_template.jinja:/etc/qwen-froggeric-chat-template.jinja:ro
environment:
- NVIDIA_VISIBLE_DEVICES=all
- VLLM_WORKER_MULTIPROC_METHOD=spawn
- NCCL_P2P_LEVEL=NVL
- NCCL_CUMEM_ENABLE=0
- VLLM_NO_USAGE_STATS=1
- VLLM_USE_FLASHINFER_SAMPLER=1
- OMP_NUM_THREADS=1
- PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
shm_size: "16gb"
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command:
- /root/.cache/huggingface/qwen3.6-27b-autoround-int4
- --served-model-name
- qwen3.6-27b-autoround
- --quantization
- auto_round
- --dtype
- float16
- --tensor-parallel-size
- "2"
- --pipeline-parallel-size
- "1"
- --max-model-len
- "262144"
- --gpu-memory-utilization
- "0.92"
- --max-num-seqs
- "2"
- --max-num-batched-tokens
- "8192"
- --kv-cache-dtype
- fp8_e5m2
- --trust-remote-code
- --chat-template
- /etc/qwen-froggeric-chat-template.jinja
- --reasoning-parser
- qwen3
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
- --enable-auto-tool-choice
- --tool-call-parser
- qwen3_coder
- --enable-prefix-caching
- --enable-chunked-prefill
- --host
- 0.0.0.0
- --port
- "8000"

[noonghunna]

This is a great result, @duart — 41/55 → 76.46/75.00 wall (77.86/77.79 decode), verify-stress 7/7 incl. the 91K Cliff 2 needle, soak PASS clean (0/25 silent-empty, 0 MiB growth, p50 92.9). A healthy dual-3090 on a virtualized enterprise box — exactly what we hoped for.

And thank you for the correction — you're right, and it's a lesson for us:

• The 2.5 GT/s was an idle-ASPM reading, not a real Gen1 down-train. report.sh samples LnkSta while the GPUs are idle, where PCIe ASPM downclocks the link to save power; under inference load it trains back to 8 GT/s. So our root-cause call was an artifact of when the report sampled the link, not a real bottleneck. We'll fix report.sh to sample under load or label it as an idle reading (same class as the "idle VRAM ⚠" false-positive). Good detective work.
• The real lever was MTP FULL_AND_PIECEWISE (+50% over PIECEWISE-only) — which is the cudagraph mode our shipped dual config runs too, so you've effectively matched our MTP path on the current nightly.

Where 76/78 sits + the last ~30% if you want it: our NVLink dual rows land ~100–110 narr / ~130–138 code (@JusefPol, @danbedford) — that delta is the Genesis patches (the v7.72.2 uplift), which your stock upstream nightly doesn't carry. bash scripts/switch.sh vllm/dual is the Genesis-patched path (on a pinned nightly that also has FULL_AND_PIECEWISE), and with NVLink you can enable custom all-reduce (+~15% per @danbedford's controlled A/B) where our PCIe default leaves it off. But 76/78 clean + virtualized is already a strong place to land.

Added a credited row to BENCHMARKS.md (bc6e20b) — it's the first Proxmox VFIO-passthrough + enterprise-server (Dell T630 / Xeon E5-2695 v4) datapoint, and a useful "stock upstream, no Genesis" anchor showing what the v7.72.2 dividend buys. Thanks for the thorough write-up + the models-on-ramdisk trick. 🙏