Skip to content

Releases

QEngine 1.3.6

Choose a tag to compare

@bpx03 bpx03 released this 15 Nov 18:31

QEngine 1.3.6

This is the new stable release: latest, vision and the main branch move here from 1.3.5.9. It collects three weeks of work on both models: a 4-bit KV cache for small cards (opt-in), faster single-stream decode on both models, faster Flash-Next prefill, and a long list of fixes from code reviews. The model weights are the same as in 1.3.5.9.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.6, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.6-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.6, then rebuild.

Measured on our machine (3x RTX 3090, PCIe 4.0, P2P), server at its defaults, November 15, 2026:

output tok/s Flash-Next, 3 cards Flash-Next, 2 cards 27B, 3 cards
single-stream decode, narrative / code 311 / 454 201 / 226 279 / 516
prefill, 10K / 90K-token prompt 8,057 / 8,372 4,265 / 7,134 6,309 / 5,113
8 agents at once (total output) 760 198 1,129

Flash-Next quality on the public BF16 set (jyohukuchan/qwen38-flash-next-bf16-logits, its own compare.py): mean KL 0.114, top-1 89.6 %, WikiText-2 0.125, the same as 1.3.5.

A 4-bit KV cache, opt-in

People on 12 GB and 8 GB cards asked for more context. QE_KV_BITS=4 (27B) and FLASH_KV_BITS=4 (Flash-Next) store the KV cache in 4 bits with a rotation and per-group scales, instead of FP8. The 27B on 3 x 12 GB goes from 255,649 to the full 262,144 tokens, 2 x 12 GB from 30,720 to 46,080, and 3 x 8 GB from 44,675 to 67,584. Flash-Next with FLASH_KV=ram keeps about 1.8x the tokens in the same host RAM (7,488 instead of 13,520 bytes per token).

It is not free: 4 bits are not lossless. Next-token KL against the BF16 model is 4.4e-3 on the 27B (FP8: 1.5e-3) and 4.95e-2 on Flash-Next (FP8: 2.52e-2). Needle tests pass up to 512K (27B, YaRN) and 1M (Flash-Next). FP8 stays the default.

Faster single-stream decode

  • Flash-Next drafts as far as its draft head is sure. When one conversation decodes alone, each step now verifies 3 to 5 MTP drafts, picked from the draft head's confidence, instead of a fixed 3. On a 30-prompt set that is +4.7 % tok/s at c=1. Greedy text can differ from 1.3.5.9 where two tokens are nearly tied, because the number of rows a step verifies changes; FLASH_MTP_CONF=0 goes back.
  • 27B on one card: an INT4 output head whose top candidates are rescored exactly from the BF16 rows (greedy output still takes the BF16 argmax, and every card keeps 0.64 GB more VRAM), a bigger DFlash2 draft vocabulary, padded verify kernels, and second-choice drafts when the drafter is unsure (QE_DF_SIB). Two conversations on one card now both draft with DFlash2 (QE_DF_BATCH).
  • Sampling with a presence penalty (the non-thinking default is presence 1.5) used to walk a step's verify rows one by one on one thread. It now does them in parallel, with the same output: Flash-Next with 8 agents went from 697 to 808 tok/s on random prompts and from 882 to 991 on real agent text.

Against 1.3.5 (the last release with a full set of runs), same machine:

point 1.3.5 1.3.6
Flash-Next 3 cards, code (club-3090 bench) 404.8 449.3
Flash-Next 3 cards, prefill 10K / 90K (club-3090 bench) 6,128 / 7,515 8,207 / 8,394
Flash-Next 3 cards, soak median decode 372.4 411.4
27B 3 cards, code / narrative (club-3090 bench.sh) 418.9 / 276.1 507.0 / 281.2
27B 3 cards, short replies code / prose / math (385 tokens, greedy) 440 / 247 / 346 (1.3.5.8) 474 / 264 / 401
27B 1 card, the same (--max-ctx 150K, DFlash2) 206 / 119 / 158 (1.3.5.8) 212 / 150 / 196
Flash-Next 8 agents at once, 3 cards 867 760 (852 without the presence penalty)

The 8-agent drop is the non-thinking default presence penalty 1.5 that came with 1.3.5.3 (taken from the model cards); clients that send their own sampling values aren't affected, and QE_PRESENCE_PENALTY=0 restores the old default.

Faster Flash-Next prefill

A series of smaller changes, each with the same output: prompt-row attention at two blocks per SM, a faster indexer scan, the next batch's n-gram rows built during the current one, no slow path for a 1-32-token tail chunk (a 4,097-token prompt: 707 to 457 ms), MTP work only where a draft needs it, and appends of 1,024 to 2,047 tokens on 3 cards with peer access staging their host-held experts by the copy engine (FLASH_PF_STAGE_MIN).

Behaviour changes

  • No conversation gets priority any more, the main agent thread included. VRAM, the host-RAM archive, pauses and the prefill queue treat every conversation alike: least recently used goes first. FLASH_MAIN_STALE_S and FLASH_EC_RECLAIM_MAIN_S are gone, --main-bonus is accepted and ignored.
  • usage.completion_tokens counts the end token the model generated, as vLLM does. A reply that ended on its own reports one token more than before, so tok/s from usage-based benchmarks is a little higher than in older releases for the same speed.
  • A tool call that does not parse reaches the client as text instead of an empty reply.
  • The thinking budget closes at exactly its count, whatever the drafts.

Fixes

  • tool_choice: none now bans the tool-call token inside the normal decode path, so such a reply no longer slows down itself or the replies beside it; a client stop string can no longer cut a forced call; none without the grammar tier follows a re-opened <think>.
  • Wrong-typed request fields (messages, system, tools, chat_template_kwargs) get HTTP 400 instead of 500, on both APIs.
  • The loop guard: sliding-window clients, reminders inside user turns, a note that would overflow the context, --loop-guard-turns range checks.
  • The 27B disk cache syncs and checksums its files (old files still load) and checks for cancellation per piece.
  • Downloads at start-up no longer race the environment setup.
  • Flash-Next: a reply near the context limit no longer stops a few tokens early; small graph budgets (FLASH_GRAPH_MB) no longer recapture every decode step; the FLASH_KV=ram decode calibration measures with the real cache; a batched step with a slot listed twice is folded into one; the start-up FAULT line lists the settings chosen at start; the archive retries sooner with --disk-cache.

Known limits and how to work around them

Flash-Next on four cards is no longer on this list: the long-context recall fix of 1.3.5.5 was confirmed on four real cards by a community run (4x RTX 5080, 1.3.5.7 / 1.3.5.9, see #24): needles found up to 240,664 tokens at 262K and up to 843,147 at 1M, the same level as 3 cards.

what you may see why what to do
temperature-0 Flash-Next text differs from 1.3.5.9 in places the confidence-based draft count changes how many rows a step verifies, and a near-tie can flip FLASH_MTP_CONF=0 gives the old drafting; for reproducible greedy runs keep using FLASH_ATTN_CHUNK=1
a single 27B card stays on MTP and prints For DFlash2: --max-ctx <= 165K ... the drafter doesn't fit next to the full context on one card ask for that --max-ctx, or set QE_DRAFTER=dflash2
an idle main agent conversation can lose its cache like any other no conversation gets priority now give the archive more RAM (FLASH_KV_ARCHIVE_GB, QE_KV_ARCHIVE_GB) if your main thread often comes back to a re-prefill

What I tested

On 3x RTX 3090, from a clean build of this release:

check result
greedy output against the references (Flash-Next 2 and 3 cards, batched, first start, 27B identity set) identical where the code paths are the same; the new single-stream drafting has its own references
262K-token stress with images, 3 and 2 cards pass, no errors, no graph fallbacks
VRAM taken by another process mid-run (both models), host RAM taken mid-run (16 / 32 / 48 GB) the engine shrinks and keeps serving; no failed request, no OOM kill
decay over a long agent soak (club-3090 #1928 shape) 0.96 / 0.99 of the fresh speed
conversation cases (side tasks, forks, two sessions, a new request that needs room) pass under the plain LRU rule
KL against BF16 on the public set 0.114 / 89.6 % / 0.125, same as 1.3.5
club-3090 8-pack, 3 full runs thinking off 125 / 121 / 125 (mean 123.7), thinking on 125 / 130 / 124 (mean 126.3)
club-3090 rebench-full and the 27B bench see the table above; verify-stress 8/8, soak pass
27B on 3 cards with an empty cache, on one card drafter downloaded and checked, loaded; correct answers
Flash-Next 2 cards, sub-agents next to a 115K main returning sub-agent turns start in 0.87 s (1.3.5.8: 1.21 s)
the llama-swap style 4M / 6M host-RAM KV commands ready in 63 / 67 s
Docker images: 27B on 3 cards with an empty qengine-cache volume, Flash-Next, 27B with --vision 1 drafter downloaded into the volume and loaded; all three answer correctly, return HTTP 400 for a too-long prompt, show 1.3.6 in /api/stats, no error lines; the image is described correctly

QEngine 1.3.5.9

Choose a tag to compare

@bpx03 bpx03 released this 12 Nov 08:59

QEngine 1.3.5.9

A one-fix release on top of 1.3.5.8: latest, vision and the main branch move here. It fixes slow or stuck Flash-Next starts on a host whose RAM is fragmented, which happened most with a large FLASH_KV=ram pool. The model weights and everything else are the same as in 1.3.5.8.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.9, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.9-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.9, then rebuild.

The fix: host-backed experts are pinned on 4 KB pages

Flash-Next keeps the experts that don't fit in VRAM in pinned host memory. Up to 1.3.5.8 it asked the kernel for 2 MB pages for that buffer. The model files it copies into pinned memory already used 4 KB pages. With the usual defrag=madvise setting, a 2 MB page fault on a host whose RAM is fragmented runs direct compaction before it falls back to 4 KB pages. A large page cache plus a big pinned FLASH_KV=ram pool is enough to cause that. On my machine a start with a 4M-token pool spent 182-197 s loading the experts instead of the usual 27-39 s. A 6M-token pool start ran past a 5-minute health check and got killed.

Now both buffers use 4 KB pages by default, which never need compaction. On this build a 4M-token start reached ready in 66 s, with 38 s of expert loading (that run did not have the fragmented RAM of the slow starts). FLASH_PIN_THP=1 brings back the 2 MB pages, but only to compare with an older build (docs/advanced.md).

What I tested

On 3× RTX 3090, from a clean build of this release: Flash-Next on 3 cards with FLASH_KV=ram FLASH_KV_RAM_TOKENS=4M, --max-ctx 1M and --vision 1 started without error lines and answered a chat correctly. No other checks ran on this build; everything else is unchanged from 1.3.5.8, whose notes list what was tested there.

The known limits of 1.3.5.8 still apply.

QEngine 1.3.5.8

Choose a tag to compare

@bpx03 bpx03 released this 12 Nov 07:34

QEngine 1.3.5.8

This is the new stable release: latest, vision and the main branch move here from 1.3.5.7. The main change: the 27B drafts with a DFlash2 block drafter when one conversation decodes alone, which makes short single-stream replies faster with unchanged output. Flash-Next on 2 cards no longer slows down when sub-agents run next to a long main conversation. tool_choice now works, NVLink bridges are picked up, and the licence is now MIT. The model weights are the same; the 27B downloads the drafter (1.28 GB) on its first start.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.8, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.8-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.8, then rebuild.

DFlash2 block drafter (27B)

Until now the 27B drafted with its MTP head (1 to 7 tokens per step). DFlash2 is a small block drafter that reads the hidden states of five layers of the model and proposes 7 tokens at once. We use the W4A16 build syvai/Qwen3.8-27B-DFlash2-W4A16 (Apache-2.0) of incoai/Qwen3.8-27B-DFlash2. The model verifies every draft as before, so greedy output is identical to MTP and sampled output follows the same distribution.

It is used when one conversation decodes alone. When two or more decode in the same step, or a reply is grammar-constrained (tool_choice, response_format), the step drafts with MTP as before, so batched throughput doesn't change.

Measured on this hardware, server, one conversation, greedy, 385 tokens, three short prompts (a Python function, a short story, a proof):

setup code prose math
3 cards, 1.3.5.8 (DFlash2) 440 tok/s 247 tok/s 346 tok/s
3 cards, MTP (QE_DRAFTER=mtp, measured at the drafter merge) 321 230 297
1 card, 1.3.5.8 (DFlash2, --max-ctx 150K, 1 slot) 206 119 158
1 card, 1.3.5.7 (MTP, 1 slot, QE_RESERVE_GB=0.5) 147 100 131

The gain depends on the text. Code and other predictable output gain the most. Long reasoning that MTP and the lookup drafts already predict well gains little. Agent turns that edit a file already in the context come out about 3 % slower: the model copies text from further back than the drafter's last ~2K positions. At 131K tokens of context the gain was +10 % on one card.

What it costs, and why one card stays on MTP by default. The drafter needs VRAM on every card: 0.70 GB per card on 3 cards, 0.82 GB on 2, 1.16 GB on one. On 2 and 3 cards the default 262K context still fits, so it is on. On a single card it costs about 38K tokens of context (one slot: 191K → 153K), so QE_DRAFTER=auto (the default) keeps a single card on MTP with its full context and prints what would fit:

DFlash2 drafter not loaded (QE_DRAFTER=auto): GPU0 has 8.52 GB free after the weights; ... For DFlash2: --max-ctx <= 150K or QE_DRAFTER=dflash2

To use it on one card, ask for that context, or force it and let the context shrink:

QE_DEVICES=0 QE_MAX_SEQS=1 server/run.sh 8080 --model models/Qwen3.8-27B-INT4 --max-ctx 150K
QE_DEVICES=0 QE_MAX_SEQS=1 QE_DRAFTER=dflash2 server/run.sh 8080 --model models/Qwen3.8-27B-INT4

With the default 16 slots one card holds only about 43K tokens per conversation, and the line says "fewer slots (QE_MAX_SEQS)". On one card DFlash2 is meant for 1 to 4 slots. QE_DRAFTER=mtp never loads it and downloads nothing.

The file. On the first start the server downloads the drafter from a pinned revision, checks its sha256 and caches it in ~/.cache/qengine/dflash2 (Docker: the qengine-cache volume). Later starts use it offline. Without network access and without the file the server starts on MTP and says so in one line; it never stops for it. QE_FETCH_ASSETS=0 turns the download off, QE_DFLASH_URL points it at a mirror, and QE_DFLASH=<dir or file> (or a dflash2/ directory inside the model directory) uses a local copy. Details: README, DFlash2.

Flash-Next on 2 cards: sub-agents no longer slow the main conversation

On 2 cards Flash-Next keeps its KV cache in VRAM by default, next to the expert cache. When a main conversation grew while sub-agents ran beside it, the main had to move above the sub-agents' regions to find room. That pushed the top of the KV pool up, and the expert cache lost the VRAM above it. In a Claude-Code-style run (a main conversation growing to 115K tokens, two sub-agents before each turn), decode fell to 146 tok/s against 181 with FLASH_KV=ram (thanks for the report in club-3090 #2002).

Now idle conversations move out of a growing conversation's way, and a finished sub-agent that sits above every generating conversation leaves VRAM for the host-RAM archive after 0.5 s under expert-cache pressure (FLASH_EC_RECLAIM_ABOVE_S; a negative value turns it off). Same run: 183 tok/s at 52-111K and 165 tok/s at 112-115K, within 1 % of FLASH_KV=ram per step, and each main turn is 7-8 % shorter than with FLASH_KV=ram, because prefill stays faster with the KV cache in VRAM. The 2-card default stays FLASH_KV=vram. Details: docs/advanced.md.

I also checked sub-agents that come back after a short tool call, which is what the 0.5 s rule could hurt. In this run two sub-agents worked side by side, 4 turns each, with 0.3 s or 1.5 s tool gaps, before every main turn, while the main grew to 115K tokens. With the KV cache in VRAM, 17 of 96 returning sub-agent turns were restored from the host-RAM archive (about 10 ms each). Returning turns started in 1.21 s on average against 1.18 s with FLASH_KV=ram. Main turns started in 1.5-1.7 s against 1.9 s.

NVLink

Cards joined by an NVLink bridge use it through CUDA peer access for peer copies and collectives, and the start-up link measurement picks the faster transport per operation. No flag is needed. The measurement is cached per link setup. Until now the cache key didn't include NVLink, so a bridge added after the first start kept the PCIe-era measurement for up to 30 days. The key now includes each card's active NVLink links and the card at the other end. The first start of this release measures again on every machine. Thanks to @Berberian007 for tracking it down (club-3090 #2000).

Fixes and smaller changes

  • tool_choice is honoured on both APIs. The server used to ignore it: none still returned a tool call, and required or a named function forced nothing. Now none (Anthropic {"type": "none"}) keeps the tools in the prompt but masks the tool-call token, so the model answers in text. required (Anthropic any) and a named function ({"type": "function", "function": {"name": ...}}, Anthropic {"type": "tool", "name": ...}) make the answer start with that call after any thinking. An unknown function name, or a forcing choice without tools, is a 400 invalid_request_error. A none reply decodes on the slower constrained path; a forced call does so only until the call header.
  • Prompts are no longer cut at a checkpoint's tokenizer limit. A tokenizer.json that ships a truncation setting (some finetunes set max_length: 4096) made the server cut every longer prompt silently. The server now ignores the file's truncation and padding settings.
  • W4A16 exports convert for Flash-Next. tools/convert_flash_next.py also takes auto_gptq dense INT4 exports with fused MTP experts. The new tools/convert_ngram_fp8.py turns an FP8 n-gram table into the INT8 one; run it first, then the converter with QE_NGRAM_EXTERNAL=1. Act-order exports are refused. Contributed by @Berberian007 in #23, with review changes.

Licence

QEngine is now under the MIT License. Versions up to 1.3.5.7 were released under the PolyForm Noncommercial License 1.0.0. The model weights and the downloaded drafter keep their own licences.

Known limits and how to work around them

what you may see why what to do
a single card stays on MTP and prints For DFlash2: --max-ctx <= 150K ... with the default context the drafter doesn't fit next to the KV cache on one card ask for that --max-ctx (the number comes from your free VRAM), or set QE_DRAFTER=dflash2 and accept the smaller context
agent turns that rewrite a file already in the context are about 3 % slower on the 27B the drafter sees only the last ~2K positions, while MTP reads the whole context nothing to do for most loads; QE_DRAFTER=mtp brings back the 1.3.5.7 drafting
the first turn of a 27B conversation restored from the archive drafts a little worse the drafter's recent-context ring isn't archived, so it starts empty it refills over the next ~2K tokens
temperature-0 27B replies with a thinking budget can end at a different token than with MTP the forced </think> lands at a step boundary, and DFlash2 takes longer steps the text before it is identical; set QE_DRAFTER=mtp if you compare against old outputs
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window, or keep 659K; the context: start-up line shows what you get
temperature-0 Flash-Next replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards for long context, and please report it with t...
Read more

QEngine 1.3.5.7

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 22:10

QEngine 1.3.5.7

This is the new stable release: latest, vision and the main branch move here from 1.3.5.6. The main change: --lsa, the sparse attention for long 27B prompts, now works with no manual steps. It also drops the model files from the page cache while the model loads, and fixes a few small things found in code reviews. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.7, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.7-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.7, then rebuild.

--lsa without manual steps (27B)

At 512K or 1M tokens most of the 27B prompt time goes to attention, and that share grows with depth. --lsa makes that attention sparse, but until now you had to download the index map and convert it with torch yourself, so few people tried it. Now this is all it takes:

server/run.sh 8080 --model models/Qwen3.8-27B-INT4 --max-ctx 1M --lsa
# Docker: -e QE_EXTRA_ARGS="--max-ctx 1M --lsa" (or -e QE_LSA=auto), with the qengine-cache volume mounted
  • On the first start the server downloads the released index map (6.3 MB, from the authors' GitHub release d-lsa-weights-v1 of Prefrontal AI's latent-sparse-attention repository) from a pinned URL, checks its sha256 and converts it to safetensors. The conversion runs in plain Python: the file's pickle goes through a restricted reader, so nothing in the file runs, and torch isn't needed. The result is checked against a pinned sha256 too and cached in ~/.cache/qengine/lsa (in Docker, the qengine-cache volume). Later starts use the cached file without network.
  • QEngine doesn't ship the map; it is fetched from upstream. The start-up log names the map, its source and the cache path.
  • Offline, or with QE_FETCH_ASSETS=0 and no cached map, the server starts with full attention and logs a curl line to fetch the file by hand. It never refuses to start because of the map.
  • --lsa MAP (or QE_LSA=MAP) still takes a map file, and a released .pt is converted into the cache. The documented forms are now just --lsa / --lsa MAP and QE_LSA=auto / QE_LSA=MAP; older spellings (--lsa auto, QE_LSA=1) still work.

Measured on 3× RTX 3090, --max-ctx 1M, one needle prompt, engine-level prompt time without / with --lsa:

prompt without --lsa with --lsa needle
92,160 tokens 20.9 s 20.6 s found / found
262,144 tokens 78.6 s 57.4 s found / found
524,288 tokens 232 s 117 s found / found

With --lsa the prompt speed stays about flat with depth (about 4,480 tok/s at 90K, 262K and 512K), while with full attention it falls from 4,400 to 2,260 tok/s. It is lossy, and only one long conversation at a time uses it; the trade-off and its limits are in docs/advanced.md.

The model files leave the page cache during the load

While the model loads, the engine reads tens of GB of model files. Their pages stayed in the page cache next to the memory the engine pins at the same time, so on a host or in a container short of RAM the kernel had to reclaim hard, and it could page other things out. Now each layer's file pages are dropped once every card has read that layer (QE_LOAD_DROP).

  • Flash-Next: on by default. The n-gram table keeps its pages. In a 110 GB container the load went from 28 million directly reclaimed pages to none, with the same load time and the same output. QE_LOAD_DROP=0 (or FLASH_WEIGHTS_COLD=0) turns it off.
  • 27B: automatic. It is on only when free RAM is short of the files plus the default archive and pinned prompts; on a large host the files stay cached, so a restart stays quick. QE_LOAD_DROP=1 forces it on. The start-up line load: model file pages ... says which it picked.

Small fixes

  • Grammar-constrained image prompts with the opt-in grammar lookup (QE_LK_GRAMMAR=1) no longer fail. A lookup draft that ran into an image row indexed past the grammar's masks and the request failed. Image rows now enter that lookup as <|image_pad|>, as the engines' own lookups do since 1.3.5.6.
  • 27B: two image prompts prefilling at once each keep their own image rows. Up to two prompts can be in the prefill at the same time, and one image table held only the newest prompt's rows, so the first could embed the second one's image. In a test that overlaps two image prompts, the first one's logits now match the run alone (before: up to 0.44 apart).
  • 27B host-RAM plan counts only the snapshots it pins at start. The plan counted the whole snapshot store, while only the first 32 entries are pinned at start (the rest only when the store fills and the host has room). The 1.3.5.6 known limit about larger engine buffers in the host RAM plan line is gone (QE_SNAPSHOTS=128: 23.9 against 7.9 GB), so a small host gives that room to the KV archive.
  • OpenRouter reasoning.enabled and reasoning.exclude are read. reasoning: {"enabled": false} turns thinking off when the request sends no other thinking setting; "enabled": true turns it on over the server default at the effort the request names. reasoning: {"exclude": true} on /v1/chat/completions: the model reasons as usual (the reasoning tokens are counted in usage), but the reply carries no reasoning_content, streamed or not.
  • Flash-Next with FLASH_KV_RAM_LAZY=1 reports the pool it can really use. The context: start-up line and the no-room error now show what is pinned plus what the host can still pin, not the full reserved pool. One full conversation window is pinned from the start, so a conversation of the full --max-ctx always fits.

Known limits and how to work around them

what you may see why what to do
a 27B turn after an image, or after a prompt under 4,096 tokens, prefills a few thousand tokens again the state at the very end of such a prompt isn't stored, so the turn resumes from the nearest checkpoint before it nothing to do; this is the same as in earlier releases
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
--lsa logs LSA: off: ... and the server runs with full attention no network and no cached map, a checkpoint without a released map, or a setup the sparse path doesn't support (no peer access between the cards, an uneven card split, a KV pool too small for one conversation above 262K) the log line says which; for the map, fetch it with the curl line from the log into the cache directory and start again
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it beyond one conversation lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, more conversations can't get room keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
decode is slower than expected on a board with peer access the start-up check picked P2P or the host path from a few timed steps, and your real load may favour the other one look at the [comm] decode collectives start-up line (both timings and the pick); compare QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and tell me which is faster, with that line

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards, also Flash-Next with --vision 1 ready, no error lines
a chat on each, a too-long prompt correct answer; HTTP 400 for the too-long prompt
greedy output on 2 and 3 cards (Flash-Next), also with QE_DECODE_COMM=host, and the 27B identity set, against 1.3.5.6 identical
Flash-Next decode step, 3 cards, one stream P2P 8.04 ms, host path 8.33 ms (1.3.5.6: 7.99 / 8.36)
27B single-stream decode, engine level 123.8 tok/s (1.3.5.6: 124.3)
--max-ctx 1M --lsa on the 27B with an empty cache map downloaded and checked, sparse attention on; needle found at 80K tokens (15.2 s) and at 266K tokens (56.3 s, whole request)
load-time page-cache drop Flash-Next dropped 67 GB of file ranges during the load; 27B kept the files cached (auto, large host)
reasoning_effort: "none" in its three forms, effort low no reasoning / short reasoning on both models
context: and host RAM plan start-up lines present on both models
an image question with --vision 1 correct description
the 1.3.5.6 image crash repro (27B, 2 cards, 32 slots, vision on, 4 parallel requests, 10 rounds) 40 of 40 correct, lookup drafts active

What I didn't test: a real 4-card machine, a host or container short of RAM on this build (the load-time drop was measured on its own branch), the Docker images beyond a start and a chat per image, and the long release runs (262K stress, KL, the 8-pack). Those ran on 1.3.5, and the README numbers are from 1.3.5.

What I'd like to hear

If you r...

Read more

QEngine 1.3.5.6

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 18:39

QEngine 1.3.5.6

This is the new stable release: latest, vision and the main branch move here from 1.3.5. It fixes slow decode through host memory on boards where P2P stores are expensive, a 27B crash with images while the n-gram lookup was drafting, and 27B agent conversations under load losing their prompt cache between turns. It also contains everything from the experimental 1.3.5.1 to 1.3.5.5 builds. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.6, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.6-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.6, then rebuild.

Fixes

  • Decode through host memory is fast again (#21, thanks @jorgermp; #20, thanks @cristianflorett). Since 1.3.5 both engines time decode on two paths at start-up and keep the faster one. The host candidate in 1.3.5 to 1.3.5.5 was not the path 1.3.4.3 used without peer access: it ran the P2P kernels over per-receiver host buffers, and every attention block read its exchange rows from host memory. On 3 cards it took 12.1 ms per step against 8.3 ms for the old path. So on boards where P2P stores are expensive (some Gen3 and multi-socket machines), the start-up check could only choose between two slow options, and one stream decoded far slower than on 1.3.4.3 (210 against about 130 tok/s on one 4-card box). Now the host path is the old no-peer one: each card writes its words once into pinned host memory, the other cards read them, and Flash-Next copies its attention exchanges into device memory in one step. Peer access stays on, so prefill keeps its P2P or copy-engine path. On our 3-card box P2P still wins (8.0 against 8.3 ms per step), and the output is identical on both paths.
  • 27B crash with images (illegal memory access, Xid 13). The n-gram lookup drafts the next tokens from the conversation's own history. For image rows that history held the ids the server derives from the image content, not the image placeholder token, so a draft that copied an image could hand the embedding an index far past its table, and every card faulted at once. It showed up as cudaErrorIllegalAddress in a batched step, often while the KV pool was shrinking under VRAM pressure, but the pool was not the cause. Image rows now enter the lookup history as <|image_pad|>, as Flash-Next already did. Flash-Next was not affected. Text-only conversations give the same output as before.
  • 27B agent conversations kept their prompt cache. The store of recurrent-state snapshots, which lets a new turn resume where the previous prompt ended, had 32 entries for every slot count. With 8 agent conversations on 32 slots they pushed each other's states out, and turns resumed from the system prompt: in a test with a VRAM hog, about 308K tokens were prefilled again, against 49K now, and the agent throughput went from 94.8 to 148.8 tok/s. The store now holds 4 snapshots per sequence slot (at least 32; QE_SNAPSHOTS overrides it). The first 32 are pinned at start as before; the rest are pinned only as the store fills, and only while the host has room. When host RAM runs short, unused snapshot buffers are freed and the store stays at its size.

QE_DECODE_COMM: how decode picks its path

Decode sends many small messages per step (all-reduces, the attention exchanges). They go either as P2P stores into the other cards' memory, or through the host path described above.

  • auto (the default): at the end of the start-up the engine runs a few seconds of dummy decode on both paths, with one sequence and with four batched sequences, and keeps the faster path for each. The host path has to win clearly (by 5 %, in at least 3 of 4 rounds). The [comm] decode collectives line shows both timings and the pick. The result is cached in ~/.cache/qengine (the Docker volume) and measured again when the cards, the driver or the build change; QE_DECODE_COMM=remeasure measures again once.
  • QE_DECODE_COMM=host forces the host path for decode. Prefill keeps peer access.
  • QE_DECODE_COMM=p2p forces P2P stores for decode.
  • QE_P2P=0 still turns peer access off for everything, prefill included, which is usually slower for long prompts.

The old QE_DECODE_HOST_FENCE setting is gone with the old host path; if you set it, it is ignored.

Known limits and how to work around them

what you may see why what to do
the 27B host RAM plan start-up line shows larger engine buffers than 1.3.5.5 (13.3 against 7.9 GB on 3 cards with the default 16 slots) the plan counts the whole snapshot store (4 per slot, about 80 MB each), although entries past the first 32 are pinned only when needed; on a host short of RAM the KV archive starts smaller than it has to nothing on a large host; on a small one set QE_SNAPSHOTS=32 (the old size) or fewer slots (QE_MAX_SEQS)
a 27B turn after an image, or after a prompt under 4,096 tokens, prefills a few thousand tokens again the state at the very end of such a prompt isn't stored, so the turn resumes from the nearest checkpoint before it nothing to do; this is the same as in earlier releases
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, the pool can't reach the full window keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
Flash-Next on a host with little RAM pages to swap (versions up to 1.3.5.4) the KV pool was counted twice (fixed in 1.3.5.5) upgrade; on older versions lower FLASH_KV_ARCHIVE_GB (for example by the size of the KV pool) or turn swap off
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
an OpenRouter client sends reasoning: {"enabled": false} and still gets thinking reasoning.enabled and reasoning.exclude aren't read yet send reasoning: {"effort": "none"}, reasoning_effort: "none" or chat_template_kwargs: {"enable_thinking": false}
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
a small host still pages out a little while the model loads the model files fill the page cache next to the pinned memory, and the kernel reclaims hard when both reach the limit raise QE_MEM_MARGIN_GB by a few GB, or lower FLASH_KV_RAM_TOKENS; turning swap off avoids paging the server out
decode is slower than expected on a board with peer access the start-up check picked P2P or the host path from a few timed steps, and your real load may favour the other one look at the [comm] decode collectives start-up line (both timings and the pick); compare QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and tell me which is faster, with that line

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards ready, no error lines
a chat on each correct answer
greedy output on 2 and 3 cards (Flash-Next) and the 27B identity set, against 1.3.5.5 identical
the same greedy runs with QE_DECODE_COMM=host identical to P2P
Flash-Next decode step, 3 cards, one stream P2P 7.99 ms, host path 8.36 ms (the 1.3.5.5 host path: 12.1 ms)
auto on this box picks P2P for one and for four sequences, on both models
reasoning_effort: "none", reasoning: {"effort": "none"}, and none with enable_thinking: true no reasoning on both models
context: and host RAM plan start-up lines present on both models
start with --vision 1 and an image question correct description
the image crash: 27B on 2 cards, 32 slots, vision on, 4 parallel requests that make the lookup copy an image, 10 rounds 40 of 40 correct with lookup drafts active, no error lines (1.3.5.5 crashed on the first request)
27B single-stream decode, 1.3.5.5 / 1.3.5.6, 3 and 2 cards, alternating runs (club-3090 bench) unchanged within run-to-run noise (3 cards: narrative 261 / 266, code 409 / 402 tok/s; 2 cards: narrative 204 / 205, code 315 / 318 tok/s; engine-level decode 124.0 / 124.1 tok/s, 124.3 after the decode path change)

What I didn't test: a real 4-card machine or a Gen3 board where the host path should win (the reporters' boxes), a host short of RAM on this build (the snapshot store's host-short path was checked on its own branch), the images themselves on a GPU, and the long release runs (262K stress, KL, the 8-pack): they ran on 1.3.5, and the numbers in the README are from 1.3.5.

What I'd like to hear

If your board picks the host path, or you compare QE_DECODE_COMM=host and p2p, the [comm] decode collectives line and your tok/s help a lot. If you run the 27B with agents or images, a start-up log and a note on whether later turns start fast helps. The issue tracker is the place.

QEngine 1.3.5.5 (experimental)

Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 13:24

QEngine 1.3.5.5 (experimental)

This build fixes four things reported in club-3090#1966: Flash-Next on 4 cards losing details in long prompts, reasoning_effort: "none" still thinking, a served context larger than the KV pool, and Flash-Next pushing a host short of RAM into swap with FLASH_KV=ram. It also contains everything from 1.3.5.4. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:1.3.5.5-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.5          # text only

From source: git fetch --tags && git checkout v1.3.5.5, then rebuild.

Fixes

  • Flash-Next on 4 cards lost details in long prompts. With four GPUs, one small DeltaNet projection wrote its output rows wider than they were spaced, so part of the gate values of the next token were overwritten, in a different order on every run. Long-context recall suffered first: the right phrase came back without its trailing number. Fixed. 2 and 3 cards were never affected and give the same output as before.
  • reasoning_effort: "none" now turns thinking off. Before, the server read it as an unknown effort and rendered medium with thinking on. none (any case) now works like enable_thinking: false in every place an effort can come from: chat_template_kwargs.reasoning_effort, the top-level reasoning_effort, the Responses / OpenRouter-style reasoning: {"effort": ...}, Anthropic output_config.effort and the server default (--reasoning-effort none / QE_REASONING_EFFORT=none). When a request carries both none and enable_thinking: true, thinking is off.
  • Served context never exceeds the KV pool. When the pool is smaller than --max-ctx, the server still starts and serves at most the pool's length. The start-up log prints context: requested ..., KV pool ..., served ... for both models, and /v1/models reports the served length.
  • Flash-Next no longer overcommits host RAM with FLASH_KV=ram. The KV pool in host RAM was left out of the host-RAM plan, so the KV archive was given the same memory a second time (on a 134.5 GB host the plan added up to 148 GB). Now the experts, the pinned n-gram table and the pool must fit MemAvailable minus the margin at start. When they don't, the archive goes first, then the pool shrinks to seats × window, then to one window, then the n-gram table moves to the page cache. Experts are never reduced. The host RAM plan start-up line shows the pool and what stays free. The runtime guard also respects a cgroup limit (docker --memory, systemd MemoryMax), for both models.

The default KV pool in host RAM (Flash-Next, FLASH_KV=ram)

A start-up line such as KV pool 1,000,448 tokens is the pool shared by all conversations together, not the longest prompt you can send. Its default is FLASH_KV_RAM_TOKENS=1M (rounded to whole 1,024-token blocks), the same on 2, 3 or 4 cards. One conversation is still limited by the window (--max-ctx, 262,144 by default, up to 1M), and the new context: requested ..., KV pool ..., served ... line shows both numbers side by side.

What the pool costs and how to size it:

FLASH_KV_RAM_TOKENS pinned host RAM (all cards together) VRAM per card for indexer keys, 3 / 2 cards
1M (default) 13.5 GB 277 / 416 MB
4M 54 GB 1.1 / 1.7 GB
10M 135 GB 2.8 / 4.2 GB
  • More long conversations at once: raise it (for example 4M). Each conversation keeps its own tokens in the shared pool, so 4M holds four 1M conversations or many shorter ones.
  • A host with little RAM: lower it to what you use, for example FLASH_KV_RAM_TOKENS=300K for one 262K conversation plus a little room. Since this release the engine also fits the pool, the KV archive and the pinned n-gram table under MemAvailable at start, so it no longer pushes the host into swap. On 1.3.5.4 and older, set it by hand and lower FLASH_KV_ARCHIVE_GB as well.
  • Pin only what you use: FLASH_KV_RAM_LAZY=1 pins the pool in slabs as conversations grow (on GPUs that can use registered host memory directly). See the limit in the table below.
  • Without FLASH_KV=ram the KV pool lives in VRAM (262,144 tokens shared) and none of this applies.

Known limits and how to work around them

what you may see why what to do
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, the pool can't reach the full window keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
Flash-Next on a host with little RAM pages to swap (versions up to 1.3.5.4) the KV pool was counted twice (see Fixes) upgrade; on older versions lower FLASH_KV_ARCHIVE_GB (for example by the size of the KV pool) or turn swap off
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
an OpenRouter client sends reasoning: {"enabled": false} and still gets thinking reasoning.enabled and reasoning.exclude aren't read yet send reasoning: {"effort": "none"}, reasoning_effort: "none" or chat_template_kwargs: {"enable_thinking": false}
Flash-Next on 4 cards still misses a needle after this release the fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
a small host still pages out a little while the model loads the model files fill the page cache next to the pinned memory, and the kernel reclaims hard when both reach the limit raise QE_MEM_MARGIN_GB by a few GB, or lower FLASH_KV_RAM_TOKENS; turning swap off avoids paging the server out
on some multi-socket / Gen3 boards one stream decodes slower than with QE_P2P=0 on 1.3.4.3 the host-memory decode path adds a memory fence per thread (QE_DECODE_HOST_FENCE=1) for PCIe ordering; under investigation (#20) try QE_DECODE_COMM=host QE_DECODE_HOST_FENCE=0 and check that the output looks right; QE_P2P=0 turns peer access off for everything (prefill gets slower)

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards ready, no error lines
a chat on each correct answer
greedy output on 2 and 3 cards (Flash-Next) and the 27B identity set, against 1.3.5.4 identical, so speed is unchanged too; one decode run per model matched 1.3.5.4
reasoning_effort: "none", reasoning: {"effort": "none"}, and none with enable_thinking: true no reasoning on both models
context: and host RAM plan start-up lines present on both models
start with --vision 1 and an image question correct description
the 4-card fix four ranks on three cards: the needle is recalled, 2- and 3-card output unchanged

What I didn't test: a real 4-card machine, a host short of RAM on this build (the plan was checked on its own branch), the images themselves on a GPU, speed suites, KL or the 8-pack on this build.

Thanks

Thanks to @Hazelhof for the detailed report in club-3090#1966, which led to all four fixes, and to @cristianflorett for the decode report in #20.

What I'd like to hear

If you have 4 cards, please try a long-context recall on Flash-Next and send the start-up log. If you run with FLASH_KV=ram on a smaller host, the host RAM plan line helps. The issue tracker is the place.

QEngine 1.3.5.4 (experimental)

Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 07 Nov 09:45

QEngine 1.3.5.4 (experimental)

This build closes most of the open issues: logprobs (#10), the 4-card mode pick (#12), a possible crash during the load (#11), a clearer decode-path log line (#19), and NUMA placement on boards that don't report a card's node (the idea from #16). It also contains everything from 1.3.5.3. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:1.3.5.4-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.4          # text only

From source: git fetch --tags && git checkout v1.3.5.4, then rebuild.

logprobs and top_logprobs (#10)

Both models accept logprobs: true and top_logprobs (0-20) on /v1/chat/completions, also when streaming. The numbers come from the target head's raw logits, before temperature, penalties and the sampling filters, as in the OpenAI API.

  • Only the reply's content tokens get entries; the reasoning part doesn't.
  • A request with logprobs runs about 3× slower than without, because it steps one token at a time with no drafts. Requests running next to it slow down too, to a bit under half their speed alone in my runs. Requests without the field are unchanged.
  • In tp (Flash-Next) and on the 27B, the values matched a no-draft reference exactly on my 3 cards. In pp the chosen token always is the top entry, but the low-probability entries of the top 20 carry pp's own rounding, the same difference pp and tp already have without logprobs. Use tp if you compare distributions.
  • No prompt logprobs. One request per position with max_tokens: 1 gives you the same.

4 cards: auto picks tp when it can (#12)

On exactly 4 equal cards, FLASH_PAR=auto now picks tp when peer access between all of them passes the data check at start and tp fits the free VRAM. Otherwise it picks pp, as before. With 5 or more cards it stays pp. If tp can't start (out of device or host memory during the load, a CUDA error, or a stall in the check), the server restarts itself in pp and says so in the log. A mode you set yourself is never changed.

I don't have 4 cards. The rule passed its unit tests, and 2- and 3-card starts give the same output and speed as before, but no 4-card machine has run it. Reports from 4-card machines are the most useful thing you can send me for this one.

Smaller changes

  • Load (#11): a setting was written while the load threads were already reading settings, which can crash the load. It's written before any thread starts now, and not at all when CUDA is already up (as with --vision 1).
  • Log (#19): the [qflash] parallel mode line names the prefill path and, when it's known, decode's path. With auto it says the decode path is timed at the end of the start-up, and a closing line gives the result.
  • NUMA: on a machine with several NUMA nodes where the system doesn't report a card's node (GPU NUMA ID: N/A in nvidia-smi topo -m, seen on Threadripper Pro boards), pinned host memory is now spread over all nodes instead of wherever the first touch puts it. QE_NUMA=0,1,2,3 sets the nodes by hand. Machines with one node, or with the nodes reported, are unchanged. My machine has one node, so this path hasn't run here.
  • Logs: FLASH_SLOT_LOG is documented, the idle reclaim line shows the longer wait of an active main thread, and flash_run prints peer access instead of a path it hadn't picked yet.

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
start with --vision 1 ready, no error lines
an image question correct description
logprobs with top_logprobs: 3 entries for every token
a non-thinking request on the default sampling normal reply
greedy output, same request twice identical
2 and 3 cards, tp: greedy output and decode speed against 1.3.5.3 same output, speed within run-to-run noise

What I didn't test: 4 cards, the NUMA fallback, the images themselves on a GPU, speed beyond the pairs above, KL or the 8-pack on this build.

What I'd like to hear

If you have 4 cards, please send the start-up log: the [qflash] parallel mode line and anything about a restart in pp. The issue tracker is the place.

QEngine 1.3.5.3 (experimental)

Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 07 Nov 00:49

QEngine 1.3.5.3 (experimental)

This build gives Qwen3.8-Flash-Next the disk cache the 27B already has, which people kept asking me for (#15). With --disk-cache DIR, Flash-Next conversations survive a restart. It's off by default, and without the flag nothing changes. It also contains everything from 1.3.5.2. The stable version is still 1.3.5: latest, vision and the main branch stay there.

It also changes the sampling defaults, including for people who send nothing. That part is new and unmeasured; see below.

docker pull ghcr.io/bpx03/qengine:1.3.5.3-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.3          # text only

From source: git fetch --tags && git checkout v1.3.5.3, then rebuild.

Using it

server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4 --disk-cache /path/to/a/roomy/disk

In Docker, mount a volume for that directory and give docker stop more time (-t 120), or the save at shutdown gets cut off. The files from the last idle save are still there in that case.

The flag and its defaults are the 27B's: 200 GB at most (--disk-cache-gb), and a save after 120 idle seconds (--persist-idle). Until now the flag stopped a Flash-Next start with an error.

What you'd see

A follow-up to a 95K-token conversation after a restart, on my 3× RTX 3090:

time to the reply
recomputed from nothing, first turn 13.3 s
read back from the file (2.3 GB; the read took 0.5 s of that) 0.7 s

That's one short run on a fast NVMe drive, not a benchmark. The file is also the cost: about 2.3 GB for that conversation, because it holds the KV and the recurrent states. Conversations under 2048 tokens aren't written.

A new conversation with the same system prompt reads only the part it needs: the 95K system prompt came back in 0.3 s, from 1.5 GB of a 2.3 GB file.

How it behaves

  • Writes run in the background when a conversation leaves VRAM (at most once a minute per conversation), after the idle time, and at shutdown. Requests don't wait for them.
  • Reads run in the background too. Other conversations keep generating, and after 30 s the prompt gives up on the file and prefills.
  • A file is only used by the same setup: same build, same weights (path, size and date of each file), same number of cards, same layout (tp and pp don't share files), same YaRN factor. An upgrade starts with an empty cache. Old files stay until room is needed.
  • A truncated or damaged file is refused and removed, and the prompt prefills. Every read is checked with a checksum.
  • A disk that fails costs the cache, not the server. A directory it can't write, a full disk or a write error gives a line in the log, and it keeps serving from memory. It also leaves 1 GB free on the disk.

Steered conversations are written too, unlike the 27B: a file records its vector by name and contents, plus mode, layers and scale, so it serves only the same steering. Details and the extra variables are in docs/advanced.md.

What's different from the 27B

The flag is the same, but the code isn't shared, because the two engines archive conversations differently. Flash-Next checks more: the 27B compares file sizes and its own model identity, while Flash-Next also compares the build and uses checksums. Its idle save doesn't block requests, and its reads don't stop the other conversations. The 27B path is untouched, and its greedy output matches the reference.

Sampling defaults from the model cards

When a request doesn't send a value, the server now uses what the Qwen3.8 model cards recommend for that reply's mode:

mode temperature top_p top_k min_p presence penalty
thinking 1.0 0.95 20 0 0
non-thinking 0.7 0.80 20 0 1.5

What changes if you send nothing:

  • Non-thinking replies, both models: temperature 0.7, top_p 0.80 and presence 1.5. Before, both models used top_p 0.95 and no penalty, with temperature 1.0 on Flash-Next and 0.6 on the 27B.
  • 27B thinking replies: temperature 1.0 instead of 0.6.
  • Flash-Next thinking replies: the same as before.

Values you send still win, one by one. So a client that sends only a temperature still gets presence 1.5 on a non-thinking reply. Greedy requests (temperature 0) stay without a penalty. --temperature, --top-p, --top-k, --min-p and --presence-penalty set one value for both modes.

The cards warn that a higher presence penalty may occasionally mix languages and slightly lower quality. To turn it off, start with --presence-penalty 0 or send presence_penalty: 0 in a request.

I haven't measured these defaults. No GPU run on this build used them, and I didn't start the Docker images, so I don't know the speed cost of presence 1.5, how tool calls behave under it, or what it does to quality. The code itself got only the mock server test and the parser test.

What I tested

These runs were on the disk cache branch just before the final build, on 3× RTX 3090. The final build and the images themselves weren't started.

A light pass on 3× RTX 3090, with a needle question in each conversation:

check result
flag off: Flash-Next greedy c=1, 27B greedy all items same as the references
tp: write, clean restart, follow-up turn 95,011 of 95,046 tokens from the file, needle correct
tp: new conversation with the same system prompt 94,978 of 95,007 from the file, needle correct
tp: follow-up after kill -9, from the idle save 91,109 of 91,144 from the file, needle correct
tp: follow-up with an image in the conversation 92,547 of 92,582 from the file, needle correct
tp: follow-up read while another reply is streaming both complete and correct
pp: write, restart, follow-up and new conversation 95,011 of 95,046 and 94,978 of 95,007, needles correct
steered (strata_l8_33): restart, follow-up 95,011 of 95,046 from the file, needle correct; steering off, another scale, another default vector or a changed sidecar don't load it
tp files on a pp server refused
truncated file refused and removed at start, correct reply after a prefill
one flipped byte in a file refused at the read and removed, correct reply after a prefill
another model directory, or one weight file replaced refused, correct reply after a prefill
unwritable directory warning, serves
1 GB disk, no file fits log line, serves, nothing left behind
a write failing partway (file size limit) log line, serves, nothing left behind

The format and the writer also have a CPU test: truncation at many points, flipped bytes, a full device, replaced and cancelled writes.

What I didn't run: speed with the flag on, the 8-pack and KL, 2 or 4 cards, FLASH_KV=ram, contexts above 262K, a slow or hanging disk, the budget's removal of old files, long runs, images with steering, per-request steering mode or layers in a file, and the Docker images with the flag. Nobody but me has reviewed this code yet. I only had one set of converted weights here, so the "another checkpoint" check used the same bytes under another path and a recopied file. The refusal depends only on path, size and date, not on the bytes.

Known limits: a conversation that doesn't fit the host archive isn't written (older conversations are evicted first, so this needs one larger than the archive, or an archive held by pinned prefixes and writes in progress), each save rewrites the whole file, and one directory serves one server (a second one runs without the cache and says so). A write in progress holds its memory until the file is done, so with a small archive and a slow disk a conversation leaving VRAM can be refused a place it would get moments later. It then resumes from an earlier state or prefills again, and the log says so. A conversation evicted from RAM before its save is queued may have no file or an older one. I read these two from the code and didn't test them.

A log line on the 27B

The 27B start-up line now says P2P: peer access between all N GPUs ... instead of naming a decode path before it's chosen, as reported on issue #19. Only the log text changed.

What I'd like to hear

If you run it, please send your start-up log, your disk type, your card count and anything that looks wrong on the issue tracker. This one depends on your testing: I'd especially like to hear about slow disks and long-running agents.

QEngine 1.3.5

Choose a tag to compare

@bpx03 bpx03 released this 06 Nov 16:41

QEngine 1.3.5

This release is mostly about fixing things. I went through every open issue and every finding from my own code reviews,
and the list got long: a use-after-free in the archive, a disk cache that lost checkpoints or forgot itself, tool calls that
could end up hidden in the reasoning, stop strings leaking, one bad request failing everybody else's. There are also four
new things: long context on the 27B, sparse attention for long prompts, faster Flash-Next prefill, and a decode transport
that picks itself per machine.

The Flash-Next and 27B weights are the same as in 1.3.4.3, so there is nothing new to download. Please read
What behaves differently before you upgrade.

Closeness to BF16 (KL)

Flash-Next's next-token KL against the official BF16 model, on jyohukuchan's public reference set (21 cases, 32,359
positions), scored with the set's own compare.py, 3 cards, default settings. Lower is closer.

Flash-Next, default weights mean KL top-1 agreement WikiText-2 KL KL, context by batched prefill (last 512 positions)
1.3.5 0.114 89.6 % 0.125 0.126

The weights are the same as in 1.3.4.3, where the mean was 0.119, so I expected the number to stay put. It moved a little, mostly on WikiText-2, and the long code case got further away. I haven't traced this to one change, so please don't read it as a quality gain.

The last column comes from my own harness; the others you can reproduce with tools/flash_kl_ref.py or the Docker kl
command (README, "Quality").

case positions mean KL top-1 agreement KL, last 512 positions after batched prefill
en_prose_129 129 0.090 89.1 % 0.090
ja_prose_129 129 0.070 90.7 % 0.070
zh_tech_257 257 0.051 87.5 % 0.051
rust_code_257 257 0.088 90.3 % 0.088
python_code_513 513 0.138 89.5 % 0.133
sql_code_1025 1,025 0.189 87.6 % 0.168
arithmetic_65 55 0.033 83.6 % 0.033
structured_257 257 0.063 94.2 % 0.063
agent_code_513 513 0.084 91.8 % 0.081
natural_4097 4,097 0.051 93.5 % 0.077
code_4097 4,097 0.291 88.1 % 0.219
wiki2_r0 2,103 0.162 87.9 % 0.170
wiki2_r1 2,103 0.143 91.6 % 0.179
wiki2_r2 2,103 0.191 88.4 % 0.344
wiki2_r3 2,103 0.169 85.4 % 0.163
wiki2_r4 2,103 0.064 91.9 % 0.098
wiki2_r5 2,103 0.093 91.7 % 0.067
wiki2_r6 2,103 0.092 92.5 % 0.133
wiki2_r7 2,103 0.156 85.5 % 0.257
wiki2_r8 2,103 0.110 87.9 % 0.115
wiki2_r9 2,103 0.068 89.3 % 0.057

club-3090 bench

8-pack. Three full runs on the same build, each on a fresh server, Flash-Next on 3 cards, vision on, the club's launch
settings. Runs vary a lot from one to the next, so please read the means and the spread together.

pack N thinking off: runs mean pass@3 thinking on: runs mean pass@3
toolcall 15 13 / 12 / 13 12.7 14.0 13 / 13 / 13 13.0 14.0
instructfollow 15 14 / 12 / 14 13.3 14.0 14 / 15 / 15 14.7 15.0
structoutput 15 13 / 12 / 12 12.3 14.0 13 / 13 / 14 13.3 14.0
dataextract 15 12 / 12 / 10 11.3 12.7 13 / 13 / 13 13.0 13.7
reasonmath 15 13 / 14 / 15 14.0 15.0 13 / 15 / 13 13.7 15.0
bugfind 15 15 / 15 / 15 15.0 15.0 14 / 14 / 14 14.0 15.0
hermesagent 20 12 / 13 / 14 13.0 15.7 15 / 13 / 15 14.3 17.0
cli 40 28 / 31 / 28 29.0 32.3 32 / 29 / 31 30.7 33.7
total 150 120 / 121 / 121 120.7 132.7 127 / 125 / 128 126.7 137.3

The same weights scored 117.0 and 124.7 in the 1.3.4.1 runs (119 / 117 / 115, 128 / 122 / 124). The thinking-on ranges overlap and the thinking-off ones just don't, and three runs per version are too few for me to call that a gain. There were no engine errors in the three runs. One reply came back as a tool call that didn't parse and recovered on its retry, and the loop guard fired once.

rebench-full. rebench-full.sh plus report.sh --full at club commit 4862eb3b, stock club slug: 8 slots, thinking on,
effort low, vision on. 3× RTX 3090, P2P on, November 6, 2026.

Flash-Next 1.3.5
decode, narrative (tok/s) 313.6
decode, code (tok/s) 404.8
prefill, 10K / 90K tokens (tok/s) 6,128 / 7,515
decode at 10K / 90K depth (tok/s) 350.1 / 290.4
1 / 2 / 4 requests at once, total (tok/s) 293.4 / 429.6 / 651.5
soak, median decode (tok/s) 372.4
verify-stress 8/8
soak PASS, 0 of 50 silent empties

Decode at a given depth is one reply per run, so those two are the noisiest numbers here.

27B, club bench.sh, 3 cards, P2P 1.3.5
decode, narrative / code (tok/s) 276.1 / 418.9
prefill, 10K / 90K tokens (tok/s) 6,437 / 5,159

The 27B prefill row was measured on its own, on cooled cards, with 32 slots and vision on.

Release checks

Every check of this release, once. The memory, load and restart checks and the 27B rows ran on the build just before the final one, with FLASH_ATTN_CHUNK=1 on: the final build has the same engine source with that default off and two small server changes. The rows marked * ran on the final engine.

check what it checks result key number
Test suites unit tests, parser, mock server, loop guard, agents, thinking budget pass 0 failed
Same text, one request* Flash-Next greedy output is the expected one, with FLASH_ATTN_CHUNK=1 pass same as reference
Same text, 8 at once* the same, 8 short requests pass same as reference
First start* empty start-up cache, re-measured, cached: same text in all three pass 3 of 3
27B output 27B output matches the expected one over its modes pass same as reference
27B single stream decode speed with no drafts pass above the gate floor
Disk cache and archive 5 tests: restart, races, a full archive, eviction pass restart recomputes 18 of 7,113 tokens
Long prompt, idle server finds a needle in a long prompt, version line, CPU when idle pass 196,924 tokens, 0.025 cores
Stress, 3 cards* 262K-token prompts plus image traffic pass 0 errors
Stress, 2 cards the same pass 0 errors
Mode A on 2 cards, no P2P starts, finds a needle, same text with weights paged out or not pass 71.3 GB paged out in 1.6 s
GPU memory taken, Flash-Next another process takes 1, 2, 4 GB, then 2 GB on the other card, under load pass 86 requests, 0 failed
GPU memory taken, 27B another process takes 16 GB under load pass 0 failed
Host RAM taken another process takes 16, 32, 48 GB under load pass 214 requests, 0 failed, 0 OOM kills
Decode after a soak* speed stays after a long multi-turn run (2 cards, no P2P) pass 97.7 % and 99.0 % of fresh
A side task on a main thread main turn resumes after a side task pass 60,347 of 60,766 tokens reused
8 side turns on a main thread main turn resumes after 8 of them pass 60,360 of 60,779 tokens reused
Two long sessions both come back from the archive pass about 150,000 tokens reused each
A new request needs the room of an idle one 3 variants of it, also with a 1 GB archive pass 80,366 of 80,799 reused with a normal archive; 0 with 1 GB (lost, logged)
8 subagents with a 1 GB archive all served pass 8 of 8
Steering vector refusals on a safety probe, with the vector pass 0.8 % on safe prompts, 3.5 % on unsafe
Server start-up* both models, with and without vision: answer, too-long prompt, image pass 0 errors

New

27B: context up to 1M tokens (--max-ctx 512K or 1M, opt-in). Above 262K the 27B needs YaRN, the same way Flash-Next
does, and --max-ctx now turns it on with the factor context / 262,144. It's static: it applies to every request of that
server, short ones too, so I'd run a separate server for long documents. The default is unchanged, bit for bit. On 3 cards
on November 3 the needle was found in a 524K prompt and in a 1.04M prompt. That check is a needle and a few spot checks at 16K, not a quality benchmark.

27B: sparse attention for long prompts (--lsa MAP, opt-in). At these lengths most of the time goes to attention, so
--lsa makes each token read 2,048 to 4,096 of its keys instead of all of them. On the same November 3 runs 512K and 1M prompts
got faster, and so did decode at 1M. The price is exactness: those keys carry about 60-80 % of the full attention weight in my measurement. I checked
two retrieval tasks at 128K and 512K and one needle at 1M, and the answers matched full attention, but I did not run RULER or
measure reasoning quality over long documents. Only one conversation at a time benefits, it needs P2P, and the KV pool
gets about 0.5 GB per card smaller. Below a prompt position of about 64K and a context of about 192K nothing changes. You
have to fetch and convert the map yourself: see docs/advanced.md.

Flash-Next: faster prefill. The block scores of the attention index now run on tensor cores, the DeltaNet projection
kernel picks its width per shape, and short prompts take the faster path from 80 tokens per batch when peer access is
available. In quick flash_run checks during the work, against the master of that day and not against 1.3.4.3, prefill got faster at
90K and 240K on 3 and on 2 cards. There is an opt-in FLASH_GRAPH_PRIO that makes long prefills faster still, but it stays
off: in the server, decode stayed slower after a long prefill with it, and I haven't found out why.

Decode transport picks itself (QE_DECODE_COMM, both models). On some boards the small messages of a decode step are
much faster through host memory than as stores into the other card. A ...

Read more

QEngine 1.3.5.2 (experimental)

Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 06 Nov 21:38

QEngine 1.3.5.2 (experimental)

This build adds INT8 layer weights for Qwen3.8-27B, as an option. INT4 stays the default, and INT8 runs only if you point the server at an 8-bit checkpoint. It also contains everything from 1.3.5.1. The stable version is still 1.3.5: latest, vision and the main branch stay there.

The INT8 weights are at bpx03/Qwen3.8-27B-INT8-QEngine, and INT4 users need nothing new.

docker pull ghcr.io/bpx03/qengine:1.3.5.2-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.2          # text only

From source: git fetch --tags && git checkout v1.3.5.2, then rebuild.

Running the 27B on INT8

hf download bpx03/Qwen3.8-27B-INT8-QEngine --local-dir models/Qwen3.8-27B-INT8
server/run.sh 8080 --model models/Qwen3.8-27B-INT8     # Docker: -e QE_MODEL=/app/models/Qwen3.8-27B-INT8

Or build the same kind of checkpoint yourself from the official BF16 weights. It takes no GPU and no calibration data, and it needs the Python packages in requirements-ref.txt:

hf download Qwen/Qwen3.8-27B --local-dir models/Qwen3.8-27B-BF16
python3 tools/quant_int8.py --src models/Qwen3.8-27B-BF16 --out models/Qwen3.8-27B-INT8 \
    --kv-from models/Qwen3.8-27B-INT4     # copies the FP8 KV scales, which keep the full KV pool

Three cards are what I'd use. On 2 cards it needed 16 slots or fewer on November 1, and one card can't hold the weights.

What it costs and what it gives

It's closer to the BF16 model and it's slower, because every decode step reads twice the weight bytes. The INT8 KL and top-1 are from this build, on 3 cards. Everything else in the table is from November 1, on older code, and I didn't rerun it, including the INT4 KL and top-1 (the INT4 greedy output on this build matches the 1.3.5 reference). The November 1 speed and pool numbers are from one day and one build.

metric INT8 INT4 INT8 / INT4
KL vs BF16, mean (320 positions) 6.66e-4 1.47e-2 22× closer
top-1 match with BF16 99.7 % 95.3 %
decode narrative, tok/s 184.6 266.6 0.69
decode code, tok/s 276.7 387.7 0.71
prefill 10K, tok/s 4,816 6,077 0.79
prefill 90K, tok/s 4,098 4,866 0.84
agent c=8, output tok/s 810 1,215 0.67
KV pool, 32 slots, 262K context, tokens 566,272 932,864 0.61

I don't know yet whether the lower KL gives better answers in real tasks, because the 8-pack has never run on INT8. If you compare the two on your own work, I'd like to hear what you find.

Using your own INT8 checkpoint

Pass its directory with --model. The start-up check reads the file headers, and a format the engine would misread stops the start with the reason (exit code 3) instead of giving wrong output. What the engine accepts:

  • one model.safetensors, not shards
  • compressed-tensors pack-quantized, symmetric, group 128 along the columns
  • weight_packed as int32 [N][K/4], bytes LSB-first, stored value w + 128
  • weight_scale as bf16 [N][K/128]
  • the same tensor names as the 27B INT4 checkpoint

The check can't prove everything: it looks at tensor types, shapes and a sample of the bytes.

  • No FP8 KV scales: the server still starts, on a bf16 KV cache with half the pool and no position split of the KV heads, and it says so. QE_KV_SCALES=<a 27B INT4 model.safetensors> borrows the scales. I checked this logic with a CPU test, but I haven't run this fallback on cards.
  • MTP head: the engine looks for model_mtp.safetensors next to the weights. An INT8 checkpoint uses that head and skips the fine-tuned one, which was trained on INT4 outputs (QE_INT8_MTP_FT=1 uses it anyway). --mtp-file always wins.
  • lued/Qwen3.8-27B-INT8-W8A16-MTP: I haven't tried it. It's sharded, so the start refuses it as it is. As far as I know, its tensor layout matches once merged into one file, and it has no KV scales, so the point above would apply.

What I tested on this build

A light pass, on 3× RTX 3090:

check result
Flash-Next greedy, c=1 same as the 1.3.5 reference
27B INT4 greedy, all items same as the 1.3.5 reference
INT8 start on 3 cards, 16 slots starts, 0 error lines
INT8 replies through the server a short sum, a 5.4K-token prompt with a needle and a code question, all correct
the Docker images: 27B INT4, Flash-Next, 27B with an image, 27B INT8 start and correct replies, 0 error lines
INT8 KL vs BF16, 320 positions mean 6.66e-4, p50 1.40e-5, p99 7.68e-3, top-1 99.7 %
INT8 under pool pressure, default path, 4 sequences batched == alone for 1200/1200 tokens each
INT8 resume after eviction from the archive 6,557 of 6,560 tokens reused, logits match the live continuation (cos 1.000000)
INT8 follow-up turn through the server found the needle, 7,757 of 7,783 tokens reused
INT8, 2 bursts of 8 parallel requests with known answers 16/16 correct, 0 error lines
rejected formats (shards, group 64, zero point, signed bytes, wrong packed type) clear refusal, CPU test on made-up headers

What I didn't run on this build: speed, the 8-pack (never run on INT8), 2 cards, images, steering, the disk cache, long runs on INT8, and the lookup-draft path under QE_GEMV_H16=0.

The INT8 kernels also fix one INT4 detail with QE_GEMV_H16=0, the off-switch path: a sequence verifying lookup drafts alone could differ from the same sequence in a batch. The default path isn't affected. That fix was checked on November 1 on the older code; the pool test above ran the default path only.

Thanks

The idea of INT8 layer weights came from lh2892 in PR #1. This is a separate implementation with a different weight layout.

What I'd like to hear

If you run it, please send your start-up log, your card count and anything that looks wrong on the issue tracker.