Repository navigation
QEngine 1.3.6
This is the new stable release: latest, vision and the main branch move here from 1.3.5.9. It collects three weeks of work on both models: a 4-bit KV cache for small cards (opt-in), faster single-stream decode on both models, faster Flash-Next prefill, and a long list of fixes from code reviews. The model weights are the same as in 1.3.5.9.
docker pull ghcr.io/bpx03/qengine:latest # = 1.3.6, text only
docker pull ghcr.io/bpx03/qengine:vision # = 1.3.6-vision, with the image encoderFrom source: git fetch --tags && git checkout v1.3.6, then rebuild.
Measured on our machine (3x RTX 3090, PCIe 4.0, P2P), server at its defaults, November 15, 2026:
| output tok/s | Flash-Next, 3 cards | Flash-Next, 2 cards | 27B, 3 cards |
|---|---|---|---|
| single-stream decode, narrative / code | 311 / 454 | 201 / 226 | 279 / 516 |
| prefill, 10K / 90K-token prompt | 8,057 / 8,372 | 4,265 / 7,134 | 6,309 / 5,113 |
| 8 agents at once (total output) | 760 | 198 | 1,129 |
Flash-Next quality on the public BF16 set (jyohukuchan/qwen38-flash-next-bf16-logits, its own compare.py): mean KL 0.114, top-1 89.6 %, WikiText-2 0.125, the same as 1.3.5.
A 4-bit KV cache, opt-in
People on 12 GB and 8 GB cards asked for more context. QE_KV_BITS=4 (27B) and FLASH_KV_BITS=4 (Flash-Next) store the KV cache in 4 bits with a rotation and per-group scales, instead of FP8. The 27B on 3 x 12 GB goes from 255,649 to the full 262,144 tokens, 2 x 12 GB from 30,720 to 46,080, and 3 x 8 GB from 44,675 to 67,584. Flash-Next with FLASH_KV=ram keeps about 1.8x the tokens in the same host RAM (7,488 instead of 13,520 bytes per token).
It is not free: 4 bits are not lossless. Next-token KL against the BF16 model is 4.4e-3 on the 27B (FP8: 1.5e-3) and 4.95e-2 on Flash-Next (FP8: 2.52e-2). Needle tests pass up to 512K (27B, YaRN) and 1M (Flash-Next). FP8 stays the default.
Faster single-stream decode
- Flash-Next drafts as far as its draft head is sure. When one conversation decodes alone, each step now verifies 3 to 5 MTP drafts, picked from the draft head's confidence, instead of a fixed 3. On a 30-prompt set that is +4.7 % tok/s at c=1. Greedy text can differ from 1.3.5.9 where two tokens are nearly tied, because the number of rows a step verifies changes;
FLASH_MTP_CONF=0goes back. - 27B on one card: an INT4 output head whose top candidates are rescored exactly from the BF16 rows (greedy output still takes the BF16 argmax, and every card keeps 0.64 GB more VRAM), a bigger DFlash2 draft vocabulary, padded verify kernels, and second-choice drafts when the drafter is unsure (
QE_DF_SIB). Two conversations on one card now both draft with DFlash2 (QE_DF_BATCH). - Sampling with a presence penalty (the non-thinking default is presence 1.5) used to walk a step's verify rows one by one on one thread. It now does them in parallel, with the same output: Flash-Next with 8 agents went from 697 to 808 tok/s on random prompts and from 882 to 991 on real agent text.
Against 1.3.5 (the last release with a full set of runs), same machine:
| point | 1.3.5 | 1.3.6 |
|---|---|---|
| Flash-Next 3 cards, code (club-3090 bench) | 404.8 | 449.3 |
| Flash-Next 3 cards, prefill 10K / 90K (club-3090 bench) | 6,128 / 7,515 | 8,207 / 8,394 |
| Flash-Next 3 cards, soak median decode | 372.4 | 411.4 |
| 27B 3 cards, code / narrative (club-3090 bench.sh) | 418.9 / 276.1 | 507.0 / 281.2 |
| 27B 3 cards, short replies code / prose / math (385 tokens, greedy) | 440 / 247 / 346 (1.3.5.8) | 474 / 264 / 401 |
27B 1 card, the same (--max-ctx 150K, DFlash2) |
206 / 119 / 158 (1.3.5.8) | 212 / 150 / 196 |
| Flash-Next 8 agents at once, 3 cards | 867 | 760 (852 without the presence penalty) |
The 8-agent drop is the non-thinking default presence penalty 1.5 that came with 1.3.5.3 (taken from the model cards); clients that send their own sampling values aren't affected, and QE_PRESENCE_PENALTY=0 restores the old default.
Faster Flash-Next prefill
A series of smaller changes, each with the same output: prompt-row attention at two blocks per SM, a faster indexer scan, the next batch's n-gram rows built during the current one, no slow path for a 1-32-token tail chunk (a 4,097-token prompt: 707 to 457 ms), MTP work only where a draft needs it, and appends of 1,024 to 2,047 tokens on 3 cards with peer access staging their host-held experts by the copy engine (FLASH_PF_STAGE_MIN).
Behaviour changes
- No conversation gets priority any more, the main agent thread included. VRAM, the host-RAM archive, pauses and the prefill queue treat every conversation alike: least recently used goes first.
FLASH_MAIN_STALE_SandFLASH_EC_RECLAIM_MAIN_Sare gone,--main-bonusis accepted and ignored. usage.completion_tokenscounts the end token the model generated, as vLLM does. A reply that ended on its own reports one token more than before, so tok/s from usage-based benchmarks is a little higher than in older releases for the same speed.- A tool call that does not parse reaches the client as text instead of an empty reply.
- The thinking budget closes at exactly its count, whatever the drafts.
Fixes
tool_choice:nonenow bans the tool-call token inside the normal decode path, so such a reply no longer slows down itself or the replies beside it; a client stop string can no longer cut a forced call;nonewithout the grammar tier follows a re-opened<think>.- Wrong-typed request fields (
messages,system,tools,chat_template_kwargs) get HTTP 400 instead of 500, on both APIs. - The loop guard: sliding-window clients, reminders inside user turns, a note that would overflow the context,
--loop-guard-turnsrange checks. - The 27B disk cache syncs and checksums its files (old files still load) and checks for cancellation per piece.
- Downloads at start-up no longer race the environment setup.
- Flash-Next: a reply near the context limit no longer stops a few tokens early; small graph budgets (
FLASH_GRAPH_MB) no longer recapture every decode step; theFLASH_KV=ramdecode calibration measures with the real cache; a batched step with a slot listed twice is folded into one; the start-up FAULT line lists the settings chosen at start; the archive retries sooner with--disk-cache.
Known limits and how to work around them
Flash-Next on four cards is no longer on this list: the long-context recall fix of 1.3.5.5 was confirmed on four real cards by a community run (4x RTX 5080, 1.3.5.7 / 1.3.5.9, see #24): needles found up to 240,664 tokens at 262K and up to 843,147 at 1M, the same level as 3 cards.
| what you may see | why | what to do |
|---|---|---|
| temperature-0 Flash-Next text differs from 1.3.5.9 in places | the confidence-based draft count changes how many rows a step verifies, and a near-tie can flip | FLASH_MTP_CONF=0 gives the old drafting; for reproducible greedy runs keep using FLASH_ATTN_CHUNK=1 |
a single 27B card stays on MTP and prints For DFlash2: --max-ctx <= 165K ... |
the drafter doesn't fit next to the full context on one card | ask for that --max-ctx, or set QE_DRAFTER=dflash2 |
| an idle main agent conversation can lose its cache like any other | no conversation gets priority now | give the archive more RAM (FLASH_KV_ARCHIVE_GB, QE_KV_ARCHIVE_GB) if your main thread often comes back to a re-prefill |
What I tested
On 3x RTX 3090, from a clean build of this release:
| check | result |
|---|---|
| greedy output against the references (Flash-Next 2 and 3 cards, batched, first start, 27B identity set) | identical where the code paths are the same; the new single-stream drafting has its own references |
| 262K-token stress with images, 3 and 2 cards | pass, no errors, no graph fallbacks |
| VRAM taken by another process mid-run (both models), host RAM taken mid-run (16 / 32 / 48 GB) | the engine shrinks and keeps serving; no failed request, no OOM kill |
| decay over a long agent soak (club-3090 #1928 shape) | 0.96 / 0.99 of the fresh speed |
| conversation cases (side tasks, forks, two sessions, a new request that needs room) | pass under the plain LRU rule |
| KL against BF16 on the public set | 0.114 / 89.6 % / 0.125, same as 1.3.5 |
| club-3090 8-pack, 3 full runs | thinking off 125 / 121 / 125 (mean 123.7), thinking on 125 / 130 / 124 (mean 126.3) |
| club-3090 rebench-full and the 27B bench | see the table above; verify-stress 8/8, soak pass |
| 27B on 3 cards with an empty cache, on one card | drafter downloaded and checked, loaded; correct answers |
| Flash-Next 2 cards, sub-agents next to a 115K main | returning sub-agent turns start in 0.87 s (1.3.5.8: 1.21 s) |
| the llama-swap style 4M / 6M host-RAM KV commands | ready in 63 / 67 s |
Docker images: 27B on 3 cards with an empty qengine-cache volume, Flash-Next, 27B with --vision 1 |
drafter downloaded into the volume and loaded; all three answer correctly, return HTTP 400 for a too-long prompt, show 1.3.6 in /api/stats, no error lines; the image is described correctly |