Skip to content

QEngine 1.3.5.8

Choose a tag to compare

@bpx03 bpx03 released this 12 Nov 07:34
· 5 commits to main since this release

QEngine 1.3.5.8

This is the new stable release: latest, vision and the main branch move here from 1.3.5.7. The main change: the 27B drafts with a DFlash2 block drafter when one conversation decodes alone, which makes short single-stream replies faster with unchanged output. Flash-Next on 2 cards no longer slows down when sub-agents run next to a long main conversation. tool_choice now works, NVLink bridges are picked up, and the licence is now MIT. The model weights are the same; the 27B downloads the drafter (1.28 GB) on its first start.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.8, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.8-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.8, then rebuild.

DFlash2 block drafter (27B)

Until now the 27B drafted with its MTP head (1 to 7 tokens per step). DFlash2 is a small block drafter that reads the hidden states of five layers of the model and proposes 7 tokens at once. We use the W4A16 build syvai/Qwen3.8-27B-DFlash2-W4A16 (Apache-2.0) of incoai/Qwen3.8-27B-DFlash2. The model verifies every draft as before, so greedy output is identical to MTP and sampled output follows the same distribution.

It is used when one conversation decodes alone. When two or more decode in the same step, or a reply is grammar-constrained (tool_choice, response_format), the step drafts with MTP as before, so batched throughput doesn't change.

Measured on this hardware, server, one conversation, greedy, 385 tokens, three short prompts (a Python function, a short story, a proof):

setup code prose math
3 cards, 1.3.5.8 (DFlash2) 440 tok/s 247 tok/s 346 tok/s
3 cards, MTP (QE_DRAFTER=mtp, measured at the drafter merge) 321 230 297
1 card, 1.3.5.8 (DFlash2, --max-ctx 150K, 1 slot) 206 119 158
1 card, 1.3.5.7 (MTP, 1 slot, QE_RESERVE_GB=0.5) 147 100 131

The gain depends on the text. Code and other predictable output gain the most. Long reasoning that MTP and the lookup drafts already predict well gains little. Agent turns that edit a file already in the context come out about 3 % slower: the model copies text from further back than the drafter's last ~2K positions. At 131K tokens of context the gain was +10 % on one card.

What it costs, and why one card stays on MTP by default. The drafter needs VRAM on every card: 0.70 GB per card on 3 cards, 0.82 GB on 2, 1.16 GB on one. On 2 and 3 cards the default 262K context still fits, so it is on. On a single card it costs about 38K tokens of context (one slot: 191K → 153K), so QE_DRAFTER=auto (the default) keeps a single card on MTP with its full context and prints what would fit:

DFlash2 drafter not loaded (QE_DRAFTER=auto): GPU0 has 8.52 GB free after the weights; ... For DFlash2: --max-ctx <= 150K or QE_DRAFTER=dflash2

To use it on one card, ask for that context, or force it and let the context shrink:

QE_DEVICES=0 QE_MAX_SEQS=1 server/run.sh 8080 --model models/Qwen3.8-27B-INT4 --max-ctx 150K
QE_DEVICES=0 QE_MAX_SEQS=1 QE_DRAFTER=dflash2 server/run.sh 8080 --model models/Qwen3.8-27B-INT4

With the default 16 slots one card holds only about 43K tokens per conversation, and the line says "fewer slots (QE_MAX_SEQS)". On one card DFlash2 is meant for 1 to 4 slots. QE_DRAFTER=mtp never loads it and downloads nothing.

The file. On the first start the server downloads the drafter from a pinned revision, checks its sha256 and caches it in ~/.cache/qengine/dflash2 (Docker: the qengine-cache volume). Later starts use it offline. Without network access and without the file the server starts on MTP and says so in one line; it never stops for it. QE_FETCH_ASSETS=0 turns the download off, QE_DFLASH_URL points it at a mirror, and QE_DFLASH=<dir or file> (or a dflash2/ directory inside the model directory) uses a local copy. Details: README, DFlash2.

Flash-Next on 2 cards: sub-agents no longer slow the main conversation

On 2 cards Flash-Next keeps its KV cache in VRAM by default, next to the expert cache. When a main conversation grew while sub-agents ran beside it, the main had to move above the sub-agents' regions to find room. That pushed the top of the KV pool up, and the expert cache lost the VRAM above it. In a Claude-Code-style run (a main conversation growing to 115K tokens, two sub-agents before each turn), decode fell to 146 tok/s against 181 with FLASH_KV=ram (thanks for the report in club-3090 #2002).

Now idle conversations move out of a growing conversation's way, and a finished sub-agent that sits above every generating conversation leaves VRAM for the host-RAM archive after 0.5 s under expert-cache pressure (FLASH_EC_RECLAIM_ABOVE_S; a negative value turns it off). Same run: 183 tok/s at 52-111K and 165 tok/s at 112-115K, within 1 % of FLASH_KV=ram per step, and each main turn is 7-8 % shorter than with FLASH_KV=ram, because prefill stays faster with the KV cache in VRAM. The 2-card default stays FLASH_KV=vram. Details: docs/advanced.md.

I also checked sub-agents that come back after a short tool call, which is what the 0.5 s rule could hurt. In this run two sub-agents worked side by side, 4 turns each, with 0.3 s or 1.5 s tool gaps, before every main turn, while the main grew to 115K tokens. With the KV cache in VRAM, 17 of 96 returning sub-agent turns were restored from the host-RAM archive (about 10 ms each). Returning turns started in 1.21 s on average against 1.18 s with FLASH_KV=ram. Main turns started in 1.5-1.7 s against 1.9 s.

NVLink

Cards joined by an NVLink bridge use it through CUDA peer access for peer copies and collectives, and the start-up link measurement picks the faster transport per operation. No flag is needed. The measurement is cached per link setup. Until now the cache key didn't include NVLink, so a bridge added after the first start kept the PCIe-era measurement for up to 30 days. The key now includes each card's active NVLink links and the card at the other end. The first start of this release measures again on every machine. Thanks to @Berberian007 for tracking it down (club-3090 #2000).

Fixes and smaller changes

  • tool_choice is honoured on both APIs. The server used to ignore it: none still returned a tool call, and required or a named function forced nothing. Now none (Anthropic {"type": "none"}) keeps the tools in the prompt but masks the tool-call token, so the model answers in text. required (Anthropic any) and a named function ({"type": "function", "function": {"name": ...}}, Anthropic {"type": "tool", "name": ...}) make the answer start with that call after any thinking. An unknown function name, or a forcing choice without tools, is a 400 invalid_request_error. A none reply decodes on the slower constrained path; a forced call does so only until the call header.
  • Prompts are no longer cut at a checkpoint's tokenizer limit. A tokenizer.json that ships a truncation setting (some finetunes set max_length: 4096) made the server cut every longer prompt silently. The server now ignores the file's truncation and padding settings.
  • W4A16 exports convert for Flash-Next. tools/convert_flash_next.py also takes auto_gptq dense INT4 exports with fused MTP experts. The new tools/convert_ngram_fp8.py turns an FP8 n-gram table into the INT8 one; run it first, then the converter with QE_NGRAM_EXTERNAL=1. Act-order exports are refused. Contributed by @Berberian007 in #23, with review changes.

Licence

QEngine is now under the MIT License. Versions up to 1.3.5.7 were released under the PolyForm Noncommercial License 1.0.0. The model weights and the downloaded drafter keep their own licences.

Known limits and how to work around them

what you may see why what to do
a single card stays on MTP and prints For DFlash2: --max-ctx <= 150K ... with the default context the drafter doesn't fit next to the KV cache on one card ask for that --max-ctx (the number comes from your free VRAM), or set QE_DRAFTER=dflash2 and accept the smaller context
agent turns that rewrite a file already in the context are about 3 % slower on the 27B the drafter sees only the last ~2K positions, while MTP reads the whole context nothing to do for most loads; QE_DRAFTER=mtp brings back the 1.3.5.7 drafting
the first turn of a 27B conversation restored from the archive drafts a little worse the drafter's recent-context ring isn't archived, so it starts empty it refills over the next ~2K tokens
temperature-0 27B replies with a thinking budget can end at a different token than with MTP the forced </think> lands at a step boundary, and DFlash2 takes longer steps the text before it is identical; set QE_DRAFTER=mtp if you compare against old outputs
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window, or keep 659K; the context: start-up line shows what you get
temperature-0 Flash-Next replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards for long context, and please report it with the start-up log

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
greedy output, Flash-Next on 2 and 3 cards, and the 27B identity set with QE_DRAFTER=mtp, against 1.3.5.7 identical
27B greedy output, DFlash2 against MTP, on 3 cards and on one card identical
27B on 3 cards with an empty cache drafter downloaded and checked, loaded; correct chat answer
27B on one card, default context / --max-ctx 150K MTP with the For DFlash2: --max-ctx <= 150K line / DFlash2 loaded, 153,600 tokens served
Flash-Next decode step, 3 cards, one stream 8.03 ms (1.3.5.7: 8.04)
27B single-stream decode without drafts, engine level 124.2 tok/s (1.3.5.7: 123.8)
Flash-Next 2 cards, multi-turn sub-agents next to a 115K main, FLASH_KV=vram against ram sub-agent turns 1.21 / 1.18 s to the first token, main turns 1.5-1.7 / 1.9 s, decode 177 / 180 tok/s
Docker images: 27B on 3 cards with an empty qengine-cache volume, Flash-Next, 27B with --vision 1 drafter downloaded into the volume and loaded; all three start with no error lines, answer a chat correctly, return HTTP 400 for a too-long prompt; the image is described correctly

What I didn't test: a machine with an NVLink bridge (there's none here; the cache key was checked on PCIe), 4 cards, a host short of RAM on this build, and the long release runs (262K stress, KL, the 8-pack). Those ran on 1.3.5, and the README numbers are from 1.3.5.

What I'd like to hear

If you run the 27B on one card, try --max-ctx at the value your start-up line names and tell me what you get. If you have an NVLink bridge, the [comm] start-up lines from the first start of this release would help. The issue tracker is the place.