Merlin 3.0.0

2026-10-22. Built on vLLM 0.32.0. One engine, one install and one command for all seven checkpoints, in one repository.

In brief

• Seven checkpoints, one engine. Flash-Next, Swift1.5, Swift1.5-Huihui, Qwen3.8-27B, ThinkingCap and two new ones,
  Nemotron 3.5 Super VL and Ornith 1.5 (ornith-ai's Qwen3.5-122B-A10B fine-tune), each with a recipe under models/.
  Every one runs at its native context (262,144 tokens; Nemotron 3.5 Super VL 1,048,576) with an FP8 KV cache, MTP
  speculative decoding and vision.
• One command. ./setup.sh checks the host, recommends a checkpoint and layout for your GPUs, shows every download
  with its size, installs the prebuilt engine (about 1.5 minutes, no compiling, nothing system-wide) and starts the
  server. Answers can be changed before anything downloads. Walkthrough: How to serve.
• One launcher. merlin serve identifies the checkpoint and picks the largest tested layout for the cards it finds
  (--tp, --profile, --instances N for several servers on one host); merlin doctor checks the host first.
• Prebuilt engine for Ampere and Ada. One wheel for sm_80, sm_86 and sm_89, checked against its published sha256.
• More hardware support. Profiles for RTX 3090 without P2P, eight RTX 3090s, RTX 4090 (24 and 48 GB), RTX A6000 / A40,
  A100, A30 and CMP 170HX; see Which checkpoint runs where.
• Faster prefill. Flash-Next cold prefill 5,020 → 5,660 tok/s at 10K and 5,417 → 6,051 tok/s at 100K against
  v2.5.3 (+12–13 %); Qwen3.8-27B on four cards 2,483 → 3,009 tok/s at 10K (+21 %) against v1.3.0.
• RecoverSSM on every checkpoint, including 27B on four cards (pool 1,974,072 → 2,012,105 tokens) and both new
  ones, and 27B on one RTX 3090: a 271,506-token pool, one 262K or two 131K sessions, images up to 4 MP.

Measured on Merlin 3.0.0

4× RTX 3090 without NVLink, P2P on, 220 W, 2026-10-20 bench cards (27B one- and two-card agents/GSM8K, ThinkingCap two-card and Nemotron agents/GSM8K are from a
2026-10-17 pre-release build). Decode: one stream, thinking on, median of 3, at a
4K / 131K prompt and the full 262,144-token context (261,888 prompt + 128 out).

Checkpoint	GPUs	KV pool, tokens	Decode, tok/s, 4K / 131K / 262K	262K prefill	8 agents	GSM8K-200
Flash-Next	4×	924,993	168 / 170 / 184	5,838 tok/s	638 tok/s	199
Swift1.5	4×	924,993	177 / 180 / 177	5,834 tok/s	643 tok/s	197
Swift1.5-Huihui	4×	924,993	187 / 188 / 189	5,880 tok/s	660	198
Qwen3.8-27B	1×	271,506	124 / 95 / 73	456 tok/s	n/a (two at most)	195
Qwen3.8-27B	2×	826,689	207 / 173 / 142	858 tok/s	517 tok/s	194
Qwen3.8-27B	4×	2,012,105	250 / 235 / 215	1,596 tok/s	752 tok/s	197
ThinkingCap	2×	826,689	198 / 182 / 165	860 tok/s	502 tok/s	193
ThinkingCap	4×	2,012,105	253 / 255 / 248	1,594 tok/s	733 tok/s	195
Nemotron 3.5 Super VL	4×	1,392,275 at 262K (1,856,367 at the shipped 1M context)	97 / 101 / 91	2,951 tok/s	429 tok/s	195
Ornith 1.5	4×	894,618	144 / 106 / 100	1,704 tok/s	468	190

Ornith 1.5 was measured on 2026-10-21 with its engine changes (uniform MoE verify on) before they merged into 3.0.0;
its bench card is models/ornith1.5/benchmarks/2026-10-21.

New checkpoints

• Nemotron 3.5 Super VL (NVIDIA, 120B-A12B MoE): W4A16 routed experts with an INT4 MTP head, tensor parallel 2 ×
  pipeline parallel 2 with expert parallel on four cards, RecoverSSM on its Mamba-2 path. It serves its native
  1,048,576-token context by default (pool 1,856,367 tokens): a 1,048,052-token needle came back exact and, re-sent, was
  served from the resident cache (1,047,936 tokens cached, 3 s to first token). At a 262K context the pool held 5 × 262K
  and 9 × 131K sessions at once with 0 preemptions; needles 23/23 at 262K and 10/10 at 131K (2026-10-17 pre-release build). One image per prompt,
  downscaled to the model's native ~3.4 MP.
• Ornith 1.5 (ornith-ai's Qwen3.5-122B-A10B fine-tune, 122B MoE): INT4 routed experts, INT4 MTP experts, INT8
  GDN projections and calibrated FP8 KV scales; tensor parallel 2 × pipeline parallel 2 with expert parallel,
  RecoverSSM, MTP K=3 with uniform MoE verify (the verify batch through one MoE graph). The INT4 MTP experts accept as the BF16 ones do (mean acceptance length 2.943 against 2.947). Pool
  894,618 tokens (3 × 262K); three ~262K prompts sent at once all came back with their needles; a recorded tool-use
  session replayed with 52/52 tool calls valid. Tested with P2P on the reference host and without P2P on a second
  4× RTX 3090 host (pool 977,838 tokens, GSM8K-200 193/200). Checkpoint about 66.4 GiB.
• Qwen3.8-27B on one card. The single-card layout returns, tested on an RTX 3090 with Qwen3.8-27B: pool 271,506
  tokens, decode 124 / 95 / 73 tok/s at 4K / 131K / 262K, 262K prefill 456 tok/s and, on the 2026-10-17 pre-release build, GSM8K-200 195/200 and needles 5/5 at
  131K and 12/12 at 262K. ThinkingCap serves on the same profile but was not run on one card.

One install, one command

• ./setup.sh fetches uv and a Python 3.13 into the checkout when the host has none, then checks the host and
  names the exact fix for each failure: driver R580 or later, GPUs and P2P, GCC/G++ 13 or newer, disk against the
  plan, host RAM for the model. It asks one question at a time, with Enter taking the answer it recommends for the
  hardware it found: model, layout, models folder, local or network access (with a generated API key) and port.
• Every download shown before it starts: the engine wheel, the 222 locked packages and the checkpoint, each with
  its size, then one confirmation. The checkpoint download resumes, is pinned to one published revision and is checked
  against the release checksums; a marker skips the full re-hash while the files are unchanged (--rehash forces it).
  Setup ends with merlin doctor, the endpoint, a curl test and the server in the foreground.
• Re-runs and updates. ./setup.sh again starts with the saved answers and asks nothing; git pull && ./setup.sh
  reinstalls only when the wheel, lock or commit moved and never downloads the model again; a failed reinstall
  restores the previous environment. --reconfigure, --yes --model M and --dry-run (fetches and writes nothing)
  are there for scripted installs, and merlin pull <model> downloads and verifies a checkpoint without a GPU.
• Careful with your host. Answers are kept per model, and a folder that holds another checkpoint is refused. The
  API key is stored owner-only, shown once and never printed in commands. uv's archive is checked against its
  published sha256. Disk is budgeted per filesystem.
• merlin serve <checkpoint> identifies the checkpoint, picks the largest tested layout that fits the idle cards
  and starts the server through one launcher. Every layout is a profile; every knob (PORT, HOST, KV_BYTES,
  MAX_PIXELS, CHUNK, BLOCK_SIZE, LPT, ...) is an environment variable. --dry-run prints the resolved command.
• Card selection follows the board. Cards are ordered by tested model, NUMA node, host bridge and PCIe switch, read
  from nvidia-smi topo -m. --instances N cuts separate card sets over that order, each with its own ports and
  compile cache, and fills mixed hosts class by class. A layout qualified with P2P is never picked without it; the
  one-card layout and --tp 1 --instances N are named instead. On 4× CMP 170HX, eight agents at 16K measured 873 tok/s
  on --tp 1 --instances 4, 523 on two TP=2 servers and 371 on one TP=4 server.
• merlin doctor checks GPUs, driver, host RAM and shared memory, honours GPUS / CUDA_VISIBLE_DEVICES, and
  reads available memory within a container's limit. merlin list shows every profile with its hardware.
• Hardware profiles beyond the reference host: RTX 3090 without P2P, eight RTX 3090s, RTX 4090 (24 and 48 GB),
  RTX A6000 / A40, A100, A30 and CMP 170HX. 4× CMP 170HX (64 GB) is tested; the others ship untested and expected to
  work. A card below its VRAM class gets a smaller pool instead of a refusal.
• Prebuilt wheel. One wheel for sm_80, sm_86 and sm_89 installs without compiling. The installer accepts only the
  wheel named in the published sha256, built from the checkout's native sources; a source install builds the same three
  architectures (about 1.5 hours on 8 cores). Requirements are locked to 222 exact pins with SHA-256 hashes, with the
  CUDA 13.0 toolkit wheels. A fresh install (2026-10-21) and a from-source build (2026-10-22) passed on the reference
  host.

Engine

• Rebased onto vLLM 0.32.0 (an upstream release candidate, the five upstream commits of the next release candidate, then the final release's one commit,
  a macOS-only socket-family fix). Eleven earlier backports now come
  from upstream, including prefix-cache and MTP fixes, GDN metadata reuse and the 1× M-RoPE cache. The rebased engine
  was temperature-0 token- and logprob-identical to the reference build on four configurations.
• FP8 KV cache on Ampere. Merlin's sm_86 FP8 attention reader for Flash-Next is kept, and 27B gets the FlashInfer
  0.7.0.post1 sm_86 FP8 small-query verify tile, which restores four-card step-time parity with vLLM 0.30: inter-token
  latency at 4K / 32K / 131K 19.99 / 21.41 / 25.27 ms on two cards and 15.83 / 16.62 / 18.37 ms on four.
• Tables in host RAM. Flash-Next's n-gram embedding table (about 47.7 GiB) lives in system memory, one copy shared
  by every GPU. The token embedding tables of the main mo...

v2.5.3: Swift1.5-Huihui checkpoint

A second checkpoint on the same engine: Swift1.5-Huihui (abliterated). The engine is unchanged (Merlin engine v2.5.2): the Quick start still downloads the v2.5.2 bundle and the v2.2.0 build artifacts; this release only moves the clone to v2.5.3.

• Swift 1.5 W4A16 with huihui-ai's abliteration transplanted onto its 96 dense output projections; routed experts unchanged. Treat it as a model that does not refuse.
• Against Swift 1.5 in the same run: GSM8K 96.29 % vs 95.98 %, agent-turn tokens 0.86× (95 % CI includes parity); 167–179 tok/s decode on v2.5.2 with MTP.
• New: scripts/serve-huihui.sh, its KV-scale sidecar, release/checkpoint-huihui.sha256, licenses/Huihui-* and the bench card. Commands: Second checkpoint.

Full notes: CHANGELOG.

v2.5.2

Flash-Next v2.5.2: idle cards drop to P8 with the bounds guards on. Point release on Merlin engine v2.5.2 = the v2.5.1
engine plus one commit, the bounds-guard reader fix. Same checkpoint, serving shape (TP2 × PP2 + EP, MTP K=3), KV pool
and compiled ops as v2.5.1.

• Idle power. With GUARD=warn (the default) the guard reader used to copy its counters to the host every 50 ms
  forever, which holds an idle RTX 3090 in P2. It now parks once the engine drains, after a final read of every
  counter: four idle cards draw about 110 W in P8 instead of 446 W in P2 on the reference host.
• Every bounds check stays on. A hit from the last step is read and reported before the reader parks; while
  serving, detection is unchanged. VLLM_E2_GUARD_IDLE=0 brings back the always-polling reader.
• Speed unchanged. Single-stream decode and cold prefill read as v2.5.1's within run-to-run noise (cell medians
  within ±1.8 %).
• Shared with the Qwen3.8-27B releases: the same engine commit, applied from the same patch file.

Thanks to CoolShades for the idle-power report in issue #2 and for PR #3, which led to this fix.

Build: scripts/build-v2.5.sh with this release's bundle and the v2.2.0 release's build-artifacts tarball (v2.5.2 changes
no compiled source; the tarball is reused, pinned by sha256). Serve: scripts/serve-v2.5.sh. Patch notes:
CHANGELOG.md; measurements: benchmarks/2026-10-12/BENCH-CARD.md; reference: docs/reference.md.

Requirements: as v2.5.1 (Linux x86-64, glibc 2.34+, Python 3.13 with headers, an NVIDIA driver with CUDA 13.0 or newer,
four 24 GB Ampere cards, 96 GB host RAM qualified).

Assets:

file	sha256
v2.5.2-from-upstream-v0.30.0.bundle.gz (the source; upstream/PIN-v2.5 pins the uncompressed bundle, ab70e1a39768d4ae016f7f9f0ae99165c20c0e0136579f557084218cef8bd685)	b154901627a985ce5469d4aebc2a15dbd07bb2c3e5c38f9517bdde9ae55ee0e2
build-artifacts.list (the 23 build-product paths; also in the repository)	2befac68d349aea26e0b35f59f4d04eda6c652ae7037f48fab8920c33ab5e432
build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz (reused from the v2.2.0 release)	cdca5ffcb3003d15e00896f0969134e16d7bfbe0a3c6dfe10698bec32573f592

v2.5.1

Flash-Next v2.5.1: agent follow-up turns resume from cache. Same checkpoint and serving shape (TP2 × PP2 + EP,
MTP K=3) as v2.2.0, on the public vLLM v0.30.0 tag plus 97 commits, with a 924,993-token KV pool (806,792 on v2.2.0).
v2.5.0 was not published: these notes cover everything since v2.2.0.

• Follow-up turns resume on a 64-token grid (--prefix-match-unit 64) instead of a 4,096-token block, and a
  conversation's cached prefix is kept until its next turn. One conversation: a follow-up starts in 0.96 s (v2.2.0:
  1.56 s). Eight deep conversations at once: 92.5 % of follow-up tokens from cache, median follow-up 1.61 s.
• A 15 % larger KV pool on the same weights: three full 262K sessions, or six 131K sessions, resident at once.
• Cold prefill 1.0–2.8 % faster than v2.2.0; single-stream decode about 3 % slower (v2.5.0's 4-boot release
  candidate: −2.81 %, significant; v2.5.1's 2 boots: −2.87 %).
• Tokenization: transformers 5.18.0 reads tokenizer.json, so combining marks (Hindi, Thai, Arabic) tokenize as
  upstream intends; the Hugging Face repository carries the upstream tokenizer.json.
• Images: up to 42 per request, each capped at 4 MP.

Build: scripts/build-v2.5.sh with this release's bundle and the v2.2.0 release's build-artifacts tarball (v2.5.1 changes
no compiled source; the tarball is reused, pinned by sha256). Serve: scripts/serve-v2.5.sh. Use a new compile cache.
Patch notes: CHANGELOG.md; measurements: benchmarks/2026-10-06/BENCH-CARD.md; reference: docs/reference.md.

Requirements: as v2.2.0 (Linux x86-64, glibc 2.34+, Python 3.13 with headers, an NVIDIA driver with CUDA 13.0 or newer,
four 24 GB Ampere cards, 96 GB host RAM qualified).

Assets:

file	sha256
v2.5.1-from-upstream-v0.30.0.bundle.gz (the source; upstream/PIN-v2.5 pins the uncompressed bundle, d904e4685877e56b57c9a23a621b537239ba6316ad238c3848c5f03437d03d7c)	660b20b07f6f190890e4c1dbfccd4f4282ec336cd9bb0562f8adf01fd81192a3
build-artifacts.list (the 23 build-product paths; also in the repository)	2befac68d349aea26e0b35f59f4d04eda6c652ae7037f48fab8920c33ab5e432
build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz (reused from the v2.2.0 release)	cdca5ffcb3003d15e00896f0969134e16d7bfbe0a3c6dfe10698bec32573f592

v2.2.0

Flash-Next v2.2.0: rebased on vLLM 0.30.0 — requires the new runtime. Same checkpoint, same serving shape
(TP2 × PP2 + EP, MTP K=3) and the same 806,792-token KV pool as v2.0.1 (three full 262,144-token sessions measured resident
at once on v2; re-tested at 3 × 256K on v2.2), on the public vLLM v0.30.0 tag plus 58 commits. A reliability release at parity with v2, not a speed-up:
single-stream decode lands at 0.973–1.040 of the v2 gate medians (v2.2.0: one release-candidate boot; v2: five-boot
gate medians; maintainer-measured).

• Greedy output repeats within one compile cache. Four determinism switches (MERLIN_FULL_K, MERLIN_QSA_SORT,
  MERLIN_TIE_DET, MERLIN_TIE_RECENT) plus in-kernel sorted emission. With prefix caching and thinking off, 8
  identical T=0 requests gave 1 distinct output in every sampled cell from 3,960 to 261,802 prompt tokens on release
  candidates; sampled, not a batch-invariance guarantee, and only within one compile cache.
• A fail-closed fault on any GPU rank now stops the engine instead of hanging it (workers gone in 3.0–3.5 s in the
  injection tests); drain-order shutdown fixes (orderly shutdown ~9.0 s to 2.65–3.55 s, measured on release candidates
  before the final tree).
• The rare illegal-address engine death is contained, not proven absent. Ring rows are tagged and validated
  before pooling; with the shipped GUARD=warn a mismatched row stops the engine with a named cause instead of an
  illegal-address fault. No rate is claimed.
• First-serve toolchain. FlashInfer and Triton compile kernels on the first request: the build now installs the
  CUDA 13.0 nvcc wheels and the links FlashInfer needs, and serve-v2.2.sh checks compiler, ninja, the Python headers
  and nvcc against the driver before starting. The container recipe carries all of them (image build and entrypoint checks tested without GPUs; GPU serving
  verified on the bare-metal routes). Thanks to the user who reported the
  runtime-JIT failures on the v2.0.1 image.
• Opt-in topology selector (AUTO_TOPO=1). For NVLink-bridged pairs, hosts with more than four GPUs and
  multi-socket hosts: it orders the lane's four GPUs so each tensor-parallel pair gets the best link (the NVLink rule is
  tested on synthetic topologies, not on bridged hardware), and turns on
  vLLM's P2P check when the driver reports P2P on a layout other than the reference one. Off by default, where the launch is exactly as before. See docs/topology-selector.md.

Build: scripts/build-v2.2.sh (verifies commit and tree of the source against upstream/PIN-v2.2). Serve:
scripts/serve-v2.2.sh. Use a new compile cache; v2.0.x artefacts do not carry over. Details, the benchmark card and
known behaviours: README ("What changed in v2.2.0") and benchmarks/2026-09-24/BENCH-CARD.md.

Requirements: Linux x86-64 with glibc 2.34 or newer (the two extensions are built on Ubuntu 22.04), Python 3.13 with
headers, an NVIDIA driver with CUDA 13.0 or newer, four 24 GB Ampere cards (qualified with peer-to-peer; also runs
without, with lower prefill), headless with no other CUDA process on them, 96 GB host RAM (qualified; ~69 GiB measured
resident floor).

Assets:

file	sha256
v2.2.0-from-upstream-v0.30.0.bundle.gz (the source; upstream/PIN-v2.2 pins the uncompressed bundle)	4de4ae5e175f58cdde9bfbfa570e813ff9d089b3e6627f4cf22c933a9c9cb206
build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz (the two v2.2 extensions for sm_86 + the stock 0.30.0 build products)	cdca5ffcb3003d15e00896f0969134e16d7bfbe0a3c6dfe10698bec32573f592
build-artifacts.list (the 23 build-product paths; also in the repository)	2befac68d349aea26e0b35f59f4d04eda6c652ae7037f48fab8920c33ab5e432

Flash-Next v2.0.1

Flash-Next v2.0.1: Qwen3.8-Flash-Next at an 806,792-token KV pool on 4x RTX 3090 (TP2 x PP2 + EP, MTP K=3, host-mapped PLE, host-resident embeddings).

v2.0.1 is v2 plus a port of upstream vLLM PRs #54442 and #56802 (issue #54437), which fix response_format / guided JSON failing under concurrency on the pipeline-parallel shape. Maintainer-reported, measured on the reference host with the shipped serve command (the requalification record, raw requests and logs are not published; the two upstream PRs are public): 80 structured requests across four load cells pass with every body schema-valid and zero grammar rejections, where v2 passed 8/10, 1/10, 2/10 and 6/10 under load. No throughput regression was found on a two-boot ladder: 0.978-1.028 of the v2 median at 4K/32K/131K/261K in both thinking modes. Same checkpoint, same serve command, same KV budget.

Source identity (see upstream/PIN-v2): vLLM fork tag v2.0.1 = commit ad5c3c223999de577b04cdb9caeab2dcb76b61b9, tree 5426cf19586a66946a1d8f7e68f48bdc7f52d62d. scripts/build-v2.sh asserts both.

Assets (four project files; the two "Source code" archives below them are generated by GitHub):

• v2.0.1-from-upstream-e962733e08.bundle.gz - the delta bundle over the public upstream commit. The checksum file hashes the uncompressed bundle: gunzip first, then sha256sum should give eb2b17399794512d57156fff1e9b2c72af18c8b4dd6c55abd1dd0ed39097e9e3. The gzipped asset as downloaded: sha256 7145fd659caea704d2b80c0a40a9bf3c7e5c5b5132de219fc21eea5048f42980.
• e2-base-src.tar.gz - snapshot of the base tree (e2-base) the 76 patches and the combined diff apply to
• build-artifacts-sm86-py313-cu130.tar.gz - the 22 compiled build products from the reference host (optional; the pinned upstream wheel gives the same files)
• build-artifacts.list - the list of those 22 files

Hashes of every asset and of the patch series: release/v2/SHA256SUMS.v2.0.1 in the tree (run cd release/v2 && sha256sum -c --ignore-missing SHA256SUMS.v2.0.1 from a checkout; place the downloaded assets there to verify them too).

Read the README section "Known behaviours of the Qwen3.8-Flash-Next architecture in vLLM" before putting this in front of clients: two behaviours remain open and documented (an occasional empty completion on a warm repeat of a long cached prefix, and prefix-cache loss for a session that finishes while other long sessions are decoding).

Also in this release: container and install fixes (privileged LXC recipe with host GPU preparation documented, a health check that performs a real generation, a build preflight that detects a python without venv, a guard against an existing vLLM checkout shadowing the built tree).