Skip to content

QEngine 1.3.6

Latest

Choose a tag to compare

@bpx03 bpx03 released this 15 Nov 18:31
· 1 commit to main since this release

QEngine 1.3.6

This is the new stable release: latest, vision and the main branch move here from 1.3.5.9. It collects three weeks of work on both models: a 4-bit KV cache for small cards (opt-in), faster single-stream decode on both models, faster Flash-Next prefill, and a long list of fixes from code reviews. The model weights are the same as in 1.3.5.9.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.6, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.6-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.6, then rebuild.

Measured on our machine (3x RTX 3090, PCIe 4.0, P2P), server at its defaults, November 15, 2026:

output tok/s Flash-Next, 3 cards Flash-Next, 2 cards 27B, 3 cards
single-stream decode, narrative / code 311 / 454 201 / 226 279 / 516
prefill, 10K / 90K-token prompt 8,057 / 8,372 4,265 / 7,134 6,309 / 5,113
8 agents at once (total output) 760 198 1,129

Flash-Next quality on the public BF16 set (jyohukuchan/qwen38-flash-next-bf16-logits, its own compare.py): mean KL 0.114, top-1 89.6 %, WikiText-2 0.125, the same as 1.3.5.

A 4-bit KV cache, opt-in

People on 12 GB and 8 GB cards asked for more context. QE_KV_BITS=4 (27B) and FLASH_KV_BITS=4 (Flash-Next) store the KV cache in 4 bits with a rotation and per-group scales, instead of FP8. The 27B on 3 x 12 GB goes from 255,649 to the full 262,144 tokens, 2 x 12 GB from 30,720 to 46,080, and 3 x 8 GB from 44,675 to 67,584. Flash-Next with FLASH_KV=ram keeps about 1.8x the tokens in the same host RAM (7,488 instead of 13,520 bytes per token).

It is not free: 4 bits are not lossless. Next-token KL against the BF16 model is 4.4e-3 on the 27B (FP8: 1.5e-3) and 4.95e-2 on Flash-Next (FP8: 2.52e-2). Needle tests pass up to 512K (27B, YaRN) and 1M (Flash-Next). FP8 stays the default.

Faster single-stream decode

  • Flash-Next drafts as far as its draft head is sure. When one conversation decodes alone, each step now verifies 3 to 5 MTP drafts, picked from the draft head's confidence, instead of a fixed 3. On a 30-prompt set that is +4.7 % tok/s at c=1. Greedy text can differ from 1.3.5.9 where two tokens are nearly tied, because the number of rows a step verifies changes; FLASH_MTP_CONF=0 goes back.
  • 27B on one card: an INT4 output head whose top candidates are rescored exactly from the BF16 rows (greedy output still takes the BF16 argmax, and every card keeps 0.64 GB more VRAM), a bigger DFlash2 draft vocabulary, padded verify kernels, and second-choice drafts when the drafter is unsure (QE_DF_SIB). Two conversations on one card now both draft with DFlash2 (QE_DF_BATCH).
  • Sampling with a presence penalty (the non-thinking default is presence 1.5) used to walk a step's verify rows one by one on one thread. It now does them in parallel, with the same output: Flash-Next with 8 agents went from 697 to 808 tok/s on random prompts and from 882 to 991 on real agent text.

Against 1.3.5 (the last release with a full set of runs), same machine:

point 1.3.5 1.3.6
Flash-Next 3 cards, code (club-3090 bench) 404.8 449.3
Flash-Next 3 cards, prefill 10K / 90K (club-3090 bench) 6,128 / 7,515 8,207 / 8,394
Flash-Next 3 cards, soak median decode 372.4 411.4
27B 3 cards, code / narrative (club-3090 bench.sh) 418.9 / 276.1 507.0 / 281.2
27B 3 cards, short replies code / prose / math (385 tokens, greedy) 440 / 247 / 346 (1.3.5.8) 474 / 264 / 401
27B 1 card, the same (--max-ctx 150K, DFlash2) 206 / 119 / 158 (1.3.5.8) 212 / 150 / 196
Flash-Next 8 agents at once, 3 cards 867 760 (852 without the presence penalty)

The 8-agent drop is the non-thinking default presence penalty 1.5 that came with 1.3.5.3 (taken from the model cards); clients that send their own sampling values aren't affected, and QE_PRESENCE_PENALTY=0 restores the old default.

Faster Flash-Next prefill

A series of smaller changes, each with the same output: prompt-row attention at two blocks per SM, a faster indexer scan, the next batch's n-gram rows built during the current one, no slow path for a 1-32-token tail chunk (a 4,097-token prompt: 707 to 457 ms), MTP work only where a draft needs it, and appends of 1,024 to 2,047 tokens on 3 cards with peer access staging their host-held experts by the copy engine (FLASH_PF_STAGE_MIN).

Behaviour changes

  • No conversation gets priority any more, the main agent thread included. VRAM, the host-RAM archive, pauses and the prefill queue treat every conversation alike: least recently used goes first. FLASH_MAIN_STALE_S and FLASH_EC_RECLAIM_MAIN_S are gone, --main-bonus is accepted and ignored.
  • usage.completion_tokens counts the end token the model generated, as vLLM does. A reply that ended on its own reports one token more than before, so tok/s from usage-based benchmarks is a little higher than in older releases for the same speed.
  • A tool call that does not parse reaches the client as text instead of an empty reply.
  • The thinking budget closes at exactly its count, whatever the drafts.

Fixes

  • tool_choice: none now bans the tool-call token inside the normal decode path, so such a reply no longer slows down itself or the replies beside it; a client stop string can no longer cut a forced call; none without the grammar tier follows a re-opened <think>.
  • Wrong-typed request fields (messages, system, tools, chat_template_kwargs) get HTTP 400 instead of 500, on both APIs.
  • The loop guard: sliding-window clients, reminders inside user turns, a note that would overflow the context, --loop-guard-turns range checks.
  • The 27B disk cache syncs and checksums its files (old files still load) and checks for cancellation per piece.
  • Downloads at start-up no longer race the environment setup.
  • Flash-Next: a reply near the context limit no longer stops a few tokens early; small graph budgets (FLASH_GRAPH_MB) no longer recapture every decode step; the FLASH_KV=ram decode calibration measures with the real cache; a batched step with a slot listed twice is folded into one; the start-up FAULT line lists the settings chosen at start; the archive retries sooner with --disk-cache.

Known limits and how to work around them

Flash-Next on four cards is no longer on this list: the long-context recall fix of 1.3.5.5 was confirmed on four real cards by a community run (4x RTX 5080, 1.3.5.7 / 1.3.5.9, see #24): needles found up to 240,664 tokens at 262K and up to 843,147 at 1M, the same level as 3 cards.

what you may see why what to do
temperature-0 Flash-Next text differs from 1.3.5.9 in places the confidence-based draft count changes how many rows a step verifies, and a near-tie can flip FLASH_MTP_CONF=0 gives the old drafting; for reproducible greedy runs keep using FLASH_ATTN_CHUNK=1
a single 27B card stays on MTP and prints For DFlash2: --max-ctx <= 165K ... the drafter doesn't fit next to the full context on one card ask for that --max-ctx, or set QE_DRAFTER=dflash2
an idle main agent conversation can lose its cache like any other no conversation gets priority now give the archive more RAM (FLASH_KV_ARCHIVE_GB, QE_KV_ARCHIVE_GB) if your main thread often comes back to a re-prefill

What I tested

On 3x RTX 3090, from a clean build of this release:

check result
greedy output against the references (Flash-Next 2 and 3 cards, batched, first start, 27B identity set) identical where the code paths are the same; the new single-stream drafting has its own references
262K-token stress with images, 3 and 2 cards pass, no errors, no graph fallbacks
VRAM taken by another process mid-run (both models), host RAM taken mid-run (16 / 32 / 48 GB) the engine shrinks and keeps serving; no failed request, no OOM kill
decay over a long agent soak (club-3090 #1928 shape) 0.96 / 0.99 of the fresh speed
conversation cases (side tasks, forks, two sessions, a new request that needs room) pass under the plain LRU rule
KL against BF16 on the public set 0.114 / 89.6 % / 0.125, same as 1.3.5
club-3090 8-pack, 3 full runs thinking off 125 / 121 / 125 (mean 123.7), thinking on 125 / 130 / 124 (mean 126.3)
club-3090 rebench-full and the 27B bench see the table above; verify-stress 8/8, soak pass
27B on 3 cards with an empty cache, on one card drafter downloaded and checked, loaded; correct answers
Flash-Next 2 cards, sub-agents next to a 115K main returning sub-agent turns start in 0.87 s (1.3.5.8: 1.21 s)
the llama-swap style 4M / 6M host-RAM KV commands ready in 63 / 67 s
Docker images: 27B on 3 cards with an empty qengine-cache volume, Flash-Next, 27B with --vision 1 drafter downloaded into the volume and loaded; all three answer correctly, return HTTP 400 for a too-long prompt, show 1.3.6 in /api/stats, no error lines; the image is described correctly