Skip to content

1Cat-vLLM 1.5.0

Choose a tag to compare

@yangzhuxinyzx yangzhuxinyzx released this 02 Sep 09:47
d8f42b3

1Cat-vLLM 1.5.0

1Cat-vLLM 1.5.0 is the largest SM70/V100 release since the project moved to
the current vLLM base. It turns the Qwen3.8 DFlash2 work into a practical
serving path, substantially improves long-context decode and prefill, adds
modern hybrid-model and NVFP4 support, and closes the wheel/API reliability
gaps required for normal deployment.

This release supersedes 1.3.0. The audited release range contains 126 merged
pull requests and 400 commits after v1.3.0.

The performance figures below are measured results under the recorded model,
GPU count, topology, context, sampling, KV dtype, and speculative-decoding
contract. They are not interchangeable universal throughput claims.

Highlights

Production DFlash2 on V100

1.5.0 backports the official MRV2 DFlash dependency closure and DFlash2 model
while preserving the existing Eagle, MTP, and DDTree implementations
(#252,
#253). The SM70 path adds:

  • independent target and draft KV dtypes;
  • checkpoint-owned seven-token drafts and selector top-K 16;
  • Qwen3.8 hidden-state capture at the correct five layer boundaries;
  • non-causal sliding-window draft attention through Flash-V100;
  • target and draft CUDA Graphs;
  • exact probabilistic rejection sampling;
  • TP4-sharded context projection and compact LM-head selection;
  • exact grouped q8 verification over FP8 E5M2 KV;
  • prefix-cache and Mamba-align state recovery;
  • automatic, capability-based fast-path selection instead of one global
    all-or-nothing gate.

An omitted per-layer RMSNorm dependency initially reduced acceptance. After
repair, the fixed 64-prompt GSM8K contract reaches request-mean acceptance
length 5.3888 and pooled acceptance length 4.5644, close to the public
official request mean of 5.46. This confirms that target FP8 quantization was
not the cause of the earlier acceptance gap.

The practical release path now measures approximately 17.38 ms per complete
DFlash2 round
without requiring the whole server to be configured as
max_num_seqs=1. Each verifier operator checks its own dtype, TP shape, KV
format, and live batch shape, then falls back independently when its local
contract is not met
(#420,
#436).

Representative full-model results on four V100 GPUs are:

Workload Result Contract / status
Historical web prompt, 512 output tokens 206.06 tok/s 17.463 ms/round, 3.599 emitted/round
High-acceptance MBPP request 251.60 tok/s AL 4.686, natural EOS, EvalPlus Base/Plus 1/1
Repeated-context adaptive lookup q16 316.27 tok/s Opt-in lookup-hit workload, not an ordinary-request default
32K cold prefill 4,039-4,069 tok/s QPN2-packed prefill, E5M2 target KV
64K pure prefill 3,567 tok/s Same DFlash2 serving stack

The lookup-augmented route keeps the checkpoint-native seven DFlash2 drafts
for ordinary requests and expands target verification only after sustained
context-copy hits. Normal, low-confidence, and structured-output requests stay
on q8. The q16 result is therefore a specialized opt-in capability rather than
the expected speed of every request
(#355,
#366).

The recommended fully quantized target checkpoint for 1.5.0 is
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4. Its fresh-wheel TP4 release gate reaches
17.645 ms per warmed complete round, 4,038.5 tok/s at 32K prefill, and
3,596.5 tok/s at 64K prefill. This checkpoint has a distinct all-NVFP4
layout, so these results are reported separately from the earlier mixed-NVFP4
206 tok/s workload above
(#445).

Qwen3.8 target-only, MTP, and long-context performance

The target model was optimized independently of speculative decoding. This is
important because a faster drafter cannot compensate for an inefficient
target verifier.

Model / route Workload Measured result Evidence
Qwen3.8-27B-NVFP4, no spec exact 128K decode 61.834 tok/s #285
Qwen3.8-27B-NVFP4, no spec exact 256K decode 50.376 tok/s #285
Qwen3.8 Flash-Next-NVFP4, no MTP 8K/512 pure decode 80.732 tok/s #415
Qwen3.8 Flash-Next-NVFP4 + MTP4 final cold-JIT gate 138.26 tok/s #389
Qwen3.6-35B-A3B NVFP4, no MTP 4096/1024 decode 116.99 tok/s #270
Qwen3.6-35B-A3B NVFP4 + MTP4 matched MTP4 decode 174.76 tok/s #270
DeepSeek-V4-Flash PP2 x TP4 strict quality control 73.539 tok/s #344

For Qwen3.8 Flash-Next, grouped Page4 QSA, indexed NVFP4 MoE input, fused
SwiGLU, and grouped-GEMM prefill raise pure-prefill throughput to:

Prompt length Pure prefill
8K 7,056.26 tok/s
32K 6,603.30 tok/s
64K 6,327.43 tok/s
131K 5,927.33 tok/s
256K 5,248.63 tok/s

The key steps are documented in
#378,
#387,
#390, and
#393. These are Flash-Next
target-prefill results; they must not be relabeled as DFlash2 prefill results.

Flash-V100 long-context improvements

1.5.0 continues the Volta-native attention work from 1.3.0:

  • batched FP8 E5M2 XQA reuses page IDs and widens paired cache loads to 128
    bits, improving B16/16K full-model pure decode by 7.92% while preserving
    bitwise operator output
    (#268);
  • long E4M3 XQA routing improves Qwen3.8 NVFP4 decode by 52.45% at 128K
    and 83.48% at 256K
    (#285);
  • the default long-prefill GQA architecture packs six heads into wider
    Tensor-Core work and reaches approximately 60.8 useful causal TFLOP/s,
    versus 46.6-47.1 TFLOP/s in the 1.3.0 path
    (#286);
  • DFlash2 draft sliding-window attention skips unused prefix tiles, removing
    its post-32K context slope
    (#328);
  • the grouped q8 verifier reduces operator latency by about 20-21% from 32K
    through 256K
    (#284).

Modern model and quantization support

  • ModelOpt NVFP4 on exact SM70. ModelOpt W4A16/W4A4 weights now use proven
    TurboMind, Marlin, or explicit-emulation backends and fail closed otherwise.
    Scale provenance is preserved without magnitude heuristics
    (#228).
  • QUASAR Qwen3.8-27B full NVFP4. Adds the fully quantized
    QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 target to the DFlash2 production path.
    The SM70 implementation separates the checkpoint's logical TP-local GDN
    output width of 4,120 from its zero-padded physical execution width of 4,128,
    enables quality-audited QPN2 execution for compatible projections and packed
    prefill, and gives MRV2 its own K+1 draft buffers and slot mappings
    (#445).
  • Qwen3.6-35B-A3B mixed FP8/NVFP4 MoE. Adds graph-safe grouped NVFP4 MoE,
    compact active-expert paths, duplicate-slot correctness, and cold-start MTP
    warmup. The retained GSM8K result is 122/128 with zero invalid or repetitive
    outputs (#270).
  • Qwen3.8 Flash-Next-NVFP4. Adds PLE offload, QSA sparse attention,
    NVFP4 MoE, MTP4, and quality-audited target-only decode
    (#338,
    #345,
    #389,
    #415).
  • GLM-5.3-Flash-NVFP4. Introduces the TP4/PP2 SM70 model path with packed
    E4M3 KV, NVFP4 MoE, KPool/indexer, sparse MLA, mHC, and KDA support. Treat
    this as a newer support lane and retain the documented topology/quality
    gates (#341).
  • DeepSeek-V4-Flash. Corrects YaRN, indexer layout, sparse/hybrid cache
    behavior, PP2 x TP4 memory lifetime, and quality-gates aggressive FP8/FP13
    routes instead of enabling them only from operator speed
    (#250,
    #251,
    #283,
    #344).
  • Multimodal capacity and safety. Adds explicit DeepSeek-OCR resolution
    modes, generic UVA multimodal-tower offload, and bounded Gemma4 tower
    attention memory
    (#293,
    #297,
    #310).
  • Quark W4A16. Repairs INT4/UINT4 checkpoint support
    (#401).

Quality and serving validation

Coding and distribution gates

The practical Qwen3.8-27B-NVFP4 + DFlash2 quality campaign uses official
temperature=1.0, top_p=0.95, top_k=20, xhigh reasoning, natural EOS,
and a 16K output cap:

Dataset Requests / scored Primary score Additional score Note
MBPP / EvalPlus 96 / 93 Base 89/93 Plus 80/93 95/96 natural stops
HumanEval / EvalPlus 96 / 96 Base 94/96 Plus 92/96 91/96 natural stops
LiveCodeBench 48 / 48 33/48 - Three seeds, exactly 11/16 each

On the first matched seed, target-only and DFlash2 both score Base 62/63 and
Plus 59/63 across MBPP and HumanEval. Eight fixed WikiText segments produce:

Target-only PPL : 5.4993116
DFlash2 PPL     : 5.4993622
Absolute delta  : 0.0000506

The quality audit and exact sampling contract are retained in
#346.

The recommended all-NVFP4 QUASAR checkpoint has an additional matched gate.
At seed 20260925, target-only and DFlash2 both score MBPP 32/32, EvalPlus
Base 31/31, and Plus 28/31. Across two predeclared seeds there is only one
paired discordance per score class, while the maximum matched WikiText PPL
difference is 0.00851. Structured-output B1/B2/B4, a 96-request mixed B4
stress, tool-chain, nested/escaped JSON, stream parity, and repeated-prefix
state tests also pass
(#445).

Tool calling, structured output, and prefix state

The DFlash2 serving path was tested beyond plain chat:

Gate Result
Structured API B1/B4 24/24
Long alternating-prefix state 5/5
BFCL 29/32
ToolACE 12/12
NexusRaven 13/16
Strict JSON Schema 7/8

The accelerated results match their retained target-only/q7 references.
Parser and state fixes cover reasoning, streaming tool calls, JSON Schema,
prefix hits, and lookup q15 burst boundaries. The release also normalizes
malformed historical tool-call arguments instead of crashing a conversation
(#346,
#366,
#432).

Reliability and correctness fixes

Hybrid KV, prefix cache, and CPU offload

  • Native CPU KV offload now supports Mamba/GDN aligned external hits, preserves
    MTP accepted-token ownership, and can opt into VMM-safe transfers with
    expandable segments
    (#247,
    #248,
    #249).
  • MRV2 zeroes newly reused KV blocks before attention, including interleaved
    multi-pool layouts
    (#276,
    #280).
  • Prefix-cache eviction tracks in-flight boundaries instead of invalidating
    active work (#296).
  • Compressed attention storage and QSA block tables consistently use physical
    page units (#373,
    #381).

Runtime and API stability

  • Multimodal cache eviction no longer hangs on stale LRU-order keys
    (#261).
  • Custom all-reduce CUDA Graph capture temporarily disables incompatible VMM
    allocations and restores the complete allocator configuration afterward
    (#262).
  • Safetensors index loading honors tensor-to-file assignments instead of
    accepting unrelated shards
    (#407).
  • KV zeroing avoids H2D staging races on first use
    (#406).
  • DFlash q16 checks the native grouped-verifier capability before dispatch;
    older q8-only extensions fall back safely instead of terminating EngineCore
    (#422).
  • API failures drain sibling multimodal work, and in-process shutdown releases
    target/draft models, KV/state tensors, CUDA Graphs, and SM70 global
    workspaces (#432).

Packaging and installation

The recommended 1.5.0 build target is:

Python 3.12
CUDA 12.8
PyTorch 2.10
NVIDIA SM70 / Tesla V100

The SM70 wheel contains the vLLM CUDA extensions, FlashAttention-V100,
paged-KV utilities, TurboMind SM70 kernels, the compact sampler, and the
precompiled FlashQLA extension. A supported wheel install does not require the
project source tree or runtime NVCC compilation.

Wheel assembly now:

  • fails closed when patchelf cannot remove build-host RPATHs;
  • emits the compact sampler with the CPython stable ABI;
  • resets only the pinned vendored FlashAttention checkout before applying
    patches, making incremental builds repeatable;
  • pins a compatible FastAPI metrics stack;
  • validates all native-library imports and grouped-verifier capabilities in an
    isolated environment.

The final 1.5.0 release-candidate audit contained ten native libraries with no
RPATH/RUNPATH and passed /v1/models, /metrics, plain chat, streaming and
non-streaming tool calls, JSON Schema, and repeated-prefix requests. A repeated
10,017-token prefix moved from 2.642 s cold to 0.164 s cached
(#319,
#320,
#421,
#427).

Install the published wheel with:

pip install ./1cat_vllm-1.5.0-cp312-cp312-linux_x86_64.whl

Recommended QUASAR NVFP4 + DFlash2 serving example

Download the recommended fully quantized target checkpoint from ModelScope:

modelscope download \
  --model QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
  --local_dir /data/models/Qwen3.8-27B-QUASAR-NVFP4

The following is the tested four-V100 production profile:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve /data/models/Qwen3.8-27B-QUASAR-NVFP4 \
  --served-model-name qwen3.8-27b-dflash2 \
  --trust-remote-code \
  --dtype half \
  --tensor-parallel-size 4 \
  --attention-backend FLASH_ATTN_V100 \
  --kv-cache-dtype fp8_e5m2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.80 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 4 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","revision":"dedf8df68adfb1afeaf7b7480c0a0243108177b4","kv_cache_dtype":"auto","draft_sample_method":"probabilistic"}' \
  --host 0.0.0.0 \
  --port 8000

The official draft checkpoint declares block size 8 and selector top-K 16, so
the runtime resolves seven speculative tokens automatically when the width is
omitted. TP4, E5M2 target KV, a 4,096-token scheduler budget, and four resident
sequences are the release-validation profile, not global DFlash2 requirements.
TP degree, KV dtype, scheduler capacity, and prefill chunk size do not globally
disable DFlash2; individual optimized operators still retain their own exact
local admission contracts.

Upgrade notes and known limitations

  • DFlash2 performance depends strongly on acceptance length, context,
    sampling, prefix reuse, and q8/q16 verification width. The 206, 251, and 316
    tok/s results above are different workloads.
  • Adaptive lookup q16 and drafter-free chaining remain specialized opt-in
    paths. Ordinary traffic remains on the checkpoint-native q7/q8 contract.
  • SM70 Flash-V100 disables saved AOT graph-cache reload by default because a
    cached artifact can cause deterministic token drift. This protects output
    quality but increases cold-start compilation time; warm the service before
    production traffic.
  • Reasoning tokens and final content share the request output budget. Short
    JSON/tool tasks should either allocate enough tokens or disable thinking for
    that request.
  • Several aggressive QPN8, FP13, and prescaled routes remain default-off after
    model-level quality gates rejected small speed wins. Do not enable research
    toggles solely from an operator microbenchmark.
  • GLM-5.3-Flash and other newer hybrid-model lanes have narrower validated
    topology and checkpoint contracts than the mature Qwen3.6/Qwen3.8 paths.
  • Source/editable builds may invoke explicit FlashQLA JIT fallback and require
    a compiler/toolkit matching the PyTorch CUDA ABI. The published SM70 wheel
    uses the precompiled extension.

Selected pull requests

DFlash and speculative decoding

#252,
#253,
#256,
#257,
#284,
#288,
#328,
#346,
#355,
#366,
#417,
#420,
#422,
#426,
#436, and
#445.

Attention, quantization, and model performance

#228,
#268,
#270,
#275,
#281,
#285,
#286,
#315,
#338,
#341,
#344,
#378,
#387,
#389,
#390,
#393, and
#415.

Serving, correctness, and packaging

#247,
#248,
#249,
#261,
#262,
#276,
#280,
#296,
#297,
#319,
#320,
#373,
#381,
#406,
#407,
#421,
#427, and
#432.

Full comparison:
v1.3.0...v1.5.0.

Contributors

Thank you to everyone who contributed code, testing, hardware time, bug
reports, and review during this release cycle, including
@yangzhuxinyzx,
@Leonccaa,
@andrewleech,
@dg1kjd,
@lwh9346,
@113636xfh,
@kkobold,
@rivetphilbot, and
@wfhe, as well as upstream vLLM and the broader
open-source inference community.