Skip to content

v2.2.0

Choose a tag to compare

@halt95 halt95 released this 25 Sep 12:21
· 25 commits to main since this release

Flash-Next v2.2.0: rebased on vLLM 0.30.0 — requires the new runtime. Same checkpoint, same serving shape
(TP2 × PP2 + EP, MTP K=3) and the same 806,792-token KV pool as v2.0.1 (three full 262,144-token sessions measured resident
at once on v2; re-tested at 3 × 256K on v2.2), on the public vLLM v0.30.0 tag plus 58 commits. A reliability release at parity with v2, not a speed-up:
single-stream decode lands at 0.973–1.040 of the v2 gate medians (v2.2.0: one release-candidate boot; v2: five-boot
gate medians; maintainer-measured).

  • Greedy output repeats within one compile cache. Four determinism switches (MERLIN_FULL_K, MERLIN_QSA_SORT,
    MERLIN_TIE_DET, MERLIN_TIE_RECENT) plus in-kernel sorted emission. With prefix caching and thinking off, 8
    identical T=0 requests gave 1 distinct output in every sampled cell from 3,960 to 261,802 prompt tokens on release
    candidates; sampled, not a batch-invariance guarantee, and only within one compile cache.
  • A fail-closed fault on any GPU rank now stops the engine instead of hanging it (workers gone in 3.0–3.5 s in the
    injection tests); drain-order shutdown fixes (orderly shutdown ~9.0 s to 2.65–3.55 s, measured on release candidates
    before the final tree).
  • The rare illegal-address engine death is contained, not proven absent. Ring rows are tagged and validated
    before pooling; with the shipped GUARD=warn a mismatched row stops the engine with a named cause instead of an
    illegal-address fault. No rate is claimed.
  • First-serve toolchain. FlashInfer and Triton compile kernels on the first request: the build now installs the
    CUDA 13.0 nvcc wheels and the links FlashInfer needs, and serve-v2.2.sh checks compiler, ninja, the Python headers
    and nvcc against the driver before starting. The container recipe carries all of them (image build and entrypoint checks tested without GPUs; GPU serving
    verified on the bare-metal routes). Thanks to the user who reported the
    runtime-JIT failures on the v2.0.1 image.
  • Opt-in topology selector (AUTO_TOPO=1). For NVLink-bridged pairs, hosts with more than four GPUs and
    multi-socket hosts: it orders the lane's four GPUs so each tensor-parallel pair gets the best link (the NVLink rule is
    tested on synthetic topologies, not on bridged hardware), and turns on
    vLLM's P2P check when the driver reports P2P on a layout other than the reference one. Off by default, where the launch is exactly as before. See docs/topology-selector.md.

Build: scripts/build-v2.2.sh (verifies commit and tree of the source against upstream/PIN-v2.2). Serve:
scripts/serve-v2.2.sh. Use a new compile cache; v2.0.x artefacts do not carry over. Details, the benchmark card and
known behaviours: README ("What changed in v2.2.0") and benchmarks/2026-09-24/BENCH-CARD.md.

Requirements: Linux x86-64 with glibc 2.34 or newer (the two extensions are built on Ubuntu 22.04), Python 3.13 with
headers, an NVIDIA driver with CUDA 13.0 or newer, four 24 GB Ampere cards (qualified with peer-to-peer; also runs
without, with lower prefill), headless with no other CUDA process on them, 96 GB host RAM (qualified; ~69 GiB measured
resident floor).

Assets:

file sha256
v2.2.0-from-upstream-v0.30.0.bundle.gz (the source; upstream/PIN-v2.2 pins the uncompressed bundle) 4de4ae5e175f58cdde9bfbfa570e813ff9d089b3e6627f4cf22c933a9c9cb206
build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz (the two v2.2 extensions for sm_86 + the stock 0.30.0 build products) cdca5ffcb3003d15e00896f0969134e16d7bfbe0a3c6dfe10698bec32573f592
build-artifacts.list (the 23 build-product paths; also in the repository) 2befac68d349aea26e0b35f59f4d04eda6c652ae7037f48fab8920c33ab5e432