Skip to content

QEngine 1.3.5

Choose a tag to compare

@bpx03 bpx03 released this 06 Nov 16:41
· 13 commits to main since this release

QEngine 1.3.5

This release is mostly about fixing things. I went through every open issue and every finding from my own code reviews,
and the list got long: a use-after-free in the archive, a disk cache that lost checkpoints or forgot itself, tool calls that
could end up hidden in the reasoning, stop strings leaking, one bad request failing everybody else's. There are also four
new things: long context on the 27B, sparse attention for long prompts, faster Flash-Next prefill, and a decode transport
that picks itself per machine.

The Flash-Next and 27B weights are the same as in 1.3.4.3, so there is nothing new to download. Please read
What behaves differently before you upgrade.

Closeness to BF16 (KL)

Flash-Next's next-token KL against the official BF16 model, on jyohukuchan's public reference set (21 cases, 32,359
positions), scored with the set's own compare.py, 3 cards, default settings. Lower is closer.

Flash-Next, default weights mean KL top-1 agreement WikiText-2 KL KL, context by batched prefill (last 512 positions)
1.3.5 0.114 89.6 % 0.125 0.126

The weights are the same as in 1.3.4.3, where the mean was 0.119, so I expected the number to stay put. It moved a little, mostly on WikiText-2, and the long code case got further away. I haven't traced this to one change, so please don't read it as a quality gain.

The last column comes from my own harness; the others you can reproduce with tools/flash_kl_ref.py or the Docker kl
command (README, "Quality").

case positions mean KL top-1 agreement KL, last 512 positions after batched prefill
en_prose_129 129 0.090 89.1 % 0.090
ja_prose_129 129 0.070 90.7 % 0.070
zh_tech_257 257 0.051 87.5 % 0.051
rust_code_257 257 0.088 90.3 % 0.088
python_code_513 513 0.138 89.5 % 0.133
sql_code_1025 1,025 0.189 87.6 % 0.168
arithmetic_65 55 0.033 83.6 % 0.033
structured_257 257 0.063 94.2 % 0.063
agent_code_513 513 0.084 91.8 % 0.081
natural_4097 4,097 0.051 93.5 % 0.077
code_4097 4,097 0.291 88.1 % 0.219
wiki2_r0 2,103 0.162 87.9 % 0.170
wiki2_r1 2,103 0.143 91.6 % 0.179
wiki2_r2 2,103 0.191 88.4 % 0.344
wiki2_r3 2,103 0.169 85.4 % 0.163
wiki2_r4 2,103 0.064 91.9 % 0.098
wiki2_r5 2,103 0.093 91.7 % 0.067
wiki2_r6 2,103 0.092 92.5 % 0.133
wiki2_r7 2,103 0.156 85.5 % 0.257
wiki2_r8 2,103 0.110 87.9 % 0.115
wiki2_r9 2,103 0.068 89.3 % 0.057

club-3090 bench

8-pack. Three full runs on the same build, each on a fresh server, Flash-Next on 3 cards, vision on, the club's launch
settings. Runs vary a lot from one to the next, so please read the means and the spread together.

pack N thinking off: runs mean pass@3 thinking on: runs mean pass@3
toolcall 15 13 / 12 / 13 12.7 14.0 13 / 13 / 13 13.0 14.0
instructfollow 15 14 / 12 / 14 13.3 14.0 14 / 15 / 15 14.7 15.0
structoutput 15 13 / 12 / 12 12.3 14.0 13 / 13 / 14 13.3 14.0
dataextract 15 12 / 12 / 10 11.3 12.7 13 / 13 / 13 13.0 13.7
reasonmath 15 13 / 14 / 15 14.0 15.0 13 / 15 / 13 13.7 15.0
bugfind 15 15 / 15 / 15 15.0 15.0 14 / 14 / 14 14.0 15.0
hermesagent 20 12 / 13 / 14 13.0 15.7 15 / 13 / 15 14.3 17.0
cli 40 28 / 31 / 28 29.0 32.3 32 / 29 / 31 30.7 33.7
total 150 120 / 121 / 121 120.7 132.7 127 / 125 / 128 126.7 137.3

The same weights scored 117.0 and 124.7 in the 1.3.4.1 runs (119 / 117 / 115, 128 / 122 / 124). The thinking-on ranges overlap and the thinking-off ones just don't, and three runs per version are too few for me to call that a gain. There were no engine errors in the three runs. One reply came back as a tool call that didn't parse and recovered on its retry, and the loop guard fired once.

rebench-full. rebench-full.sh plus report.sh --full at club commit 4862eb3b, stock club slug: 8 slots, thinking on,
effort low, vision on. 3× RTX 3090, P2P on, November 6, 2026.

Flash-Next 1.3.5
decode, narrative (tok/s) 313.6
decode, code (tok/s) 404.8
prefill, 10K / 90K tokens (tok/s) 6,128 / 7,515
decode at 10K / 90K depth (tok/s) 350.1 / 290.4
1 / 2 / 4 requests at once, total (tok/s) 293.4 / 429.6 / 651.5
soak, median decode (tok/s) 372.4
verify-stress 8/8
soak PASS, 0 of 50 silent empties

Decode at a given depth is one reply per run, so those two are the noisiest numbers here.

27B, club bench.sh, 3 cards, P2P 1.3.5
decode, narrative / code (tok/s) 276.1 / 418.9
prefill, 10K / 90K tokens (tok/s) 6,437 / 5,159

The 27B prefill row was measured on its own, on cooled cards, with 32 slots and vision on.

Release checks

Every check of this release, once. The memory, load and restart checks and the 27B rows ran on the build just before the final one, with FLASH_ATTN_CHUNK=1 on: the final build has the same engine source with that default off and two small server changes. The rows marked * ran on the final engine.

check what it checks result key number
Test suites unit tests, parser, mock server, loop guard, agents, thinking budget pass 0 failed
Same text, one request* Flash-Next greedy output is the expected one, with FLASH_ATTN_CHUNK=1 pass same as reference
Same text, 8 at once* the same, 8 short requests pass same as reference
First start* empty start-up cache, re-measured, cached: same text in all three pass 3 of 3
27B output 27B output matches the expected one over its modes pass same as reference
27B single stream decode speed with no drafts pass above the gate floor
Disk cache and archive 5 tests: restart, races, a full archive, eviction pass restart recomputes 18 of 7,113 tokens
Long prompt, idle server finds a needle in a long prompt, version line, CPU when idle pass 196,924 tokens, 0.025 cores
Stress, 3 cards* 262K-token prompts plus image traffic pass 0 errors
Stress, 2 cards the same pass 0 errors
Mode A on 2 cards, no P2P starts, finds a needle, same text with weights paged out or not pass 71.3 GB paged out in 1.6 s
GPU memory taken, Flash-Next another process takes 1, 2, 4 GB, then 2 GB on the other card, under load pass 86 requests, 0 failed
GPU memory taken, 27B another process takes 16 GB under load pass 0 failed
Host RAM taken another process takes 16, 32, 48 GB under load pass 214 requests, 0 failed, 0 OOM kills
Decode after a soak* speed stays after a long multi-turn run (2 cards, no P2P) pass 97.7 % and 99.0 % of fresh
A side task on a main thread main turn resumes after a side task pass 60,347 of 60,766 tokens reused
8 side turns on a main thread main turn resumes after 8 of them pass 60,360 of 60,779 tokens reused
Two long sessions both come back from the archive pass about 150,000 tokens reused each
A new request needs the room of an idle one 3 variants of it, also with a 1 GB archive pass 80,366 of 80,799 reused with a normal archive; 0 with 1 GB (lost, logged)
8 subagents with a 1 GB archive all served pass 8 of 8
Steering vector refusals on a safety probe, with the vector pass 0.8 % on safe prompts, 3.5 % on unsafe
Server start-up* both models, with and without vision: answer, too-long prompt, image pass 0 errors

New

27B: context up to 1M tokens (--max-ctx 512K or 1M, opt-in). Above 262K the 27B needs YaRN, the same way Flash-Next
does, and --max-ctx now turns it on with the factor context / 262,144. It's static: it applies to every request of that
server, short ones too, so I'd run a separate server for long documents. The default is unchanged, bit for bit. On 3 cards
on November 3 the needle was found in a 524K prompt and in a 1.04M prompt. That check is a needle and a few spot checks at 16K, not a quality benchmark.

27B: sparse attention for long prompts (--lsa MAP, opt-in). At these lengths most of the time goes to attention, so
--lsa makes each token read 2,048 to 4,096 of its keys instead of all of them. On the same November 3 runs 512K and 1M prompts
got faster, and so did decode at 1M. The price is exactness: those keys carry about 60-80 % of the full attention weight in my measurement. I checked
two retrieval tasks at 128K and 512K and one needle at 1M, and the answers matched full attention, but I did not run RULER or
measure reasoning quality over long documents. Only one conversation at a time benefits, it needs P2P, and the KV pool
gets about 0.5 GB per card smaller. Below a prompt position of about 64K and a context of about 192K nothing changes. You
have to fetch and convert the map yourself: see docs/advanced.md.

Flash-Next: faster prefill. The block scores of the attention index now run on tensor cores, the DeltaNet projection
kernel picks its width per shape, and short prompts take the faster path from 80 tokens per batch when peer access is
available. In quick flash_run checks during the work, against the master of that day and not against 1.3.4.3, prefill got faster at
90K and 240K on 3 and on 2 cards. There is an opt-in FLASH_GRAPH_PRIO that makes long prefills faster still, but it stays
off: in the server, decode stayed slower after a long prefill with it, and I haven't found out why.

Decode transport picks itself (QE_DECODE_COMM, both models). On some boards the small messages of a decode step are
much faster through host memory than as stores into the other card. A club-3090 user with an EPYC 7F52 and PCIe gen 3 x16 measured
this on 1.3.4.3 by forcing QE_P2P=0. Now the engine
times real decode steps on both paths at the end of the start-up, keeps the faster one, and prefill keeps its own path. It takes a
few seconds the first time and is cached per hardware and configuration (QE_COMM_CACHE=refresh measures again). I can only say
this much about it: on my machine it picks P2P on every mode and the output and speed stay the same. The board it was built for
hasn't run it yet, so please send the [comm] decode collectives line if you have a board like that.

Also new. A start-up line with the CUDA module loading, native or PTX per card, and the watchdog budget. A source build now
targets the GPUs in the machine (always including sm_86) instead of only sm_86. The Docker images have a kl command.

Flash-Next: the same text on repeat, if you ask for it (FLASH_ATTN_CHUNK=1). With this on, the same request gives the same text whatever shares a step and whatever state the machine is in. Turn it on when you compare runs, reproduce a report or run tests. It's off by default, because it costs speed, and it doesn't make answers closer to BF16.

By default, text at temperature 0 can differ between runs where two tokens are nearly tied. I saw that change with what shared a step and with what had just run on the cards. I measured no sign of lower quality from it, and the KL and 8-pack numbers above are at the default. What it costs, each row measured on one build with the setting on and off:

off (default) FLASH_ATTN_CHUNK=1 change
8 agents at once, 2 runs each (tok/s) 934.5 / 956.0 881.4 / 847.2 -8.6 %
8 short prompts at once, first token, median (ms) 184 / 183 308 / 311 about +125 ms
8 short prompts at once, all done, median (ms) 333 / 336 462 / 466 about +130 ms
8 short sequences, no server (tok/s) 1,095 1,070 -2.3 %

Short prompts that arrive together are prefilled one after another with it on, which is the first-token cost. It holds per setup: a different number of cards can give different text, and I didn't measure it across cache hits and misses. What I know of the cause is its class, float sums in the decode path. I can't explain why an earlier load on the cards changes those results, only that this setting removes the effect.

Fixes

Memory safety and wrong output

  • Use-after-free in the archive (27B). With a full archive, making room could delete an archive that the code still held
    and then use it. The default setup, with no disk cache, was the exposed one. In a test with freed memory overwritten
    it crashed before the fix and passes now.
  • Wrong KV after a refused write (Flash-Next). If a conversation's archive write was refused at eviction, a later turn could
    resume from stale state that still passed the position checks. A property test found this, and the older code had it too. It now cuts back
    to what is really there.
  • No P2P: a stale word could pass as new. On the host-memory path (stock driver, QE_P2P=0) nobody cleared the
    all-reduce words, so a late reader could take an old word whose 16-bit tag matched, which means a wrong sum on one row and
    possibly a wrong token or draft decision. My forced repro showed an error before the fix and none after. How often this hit
    real traffic I did not measure, because there's no counter for it. Speed on that path is unchanged in the gate pairs.
  • The sampler reused random numbers. With 17 or more rows in a step, a row could get the same random number as a row of the next
    step. That is fixed, and it changes sampled output (see below). I did not measure how much it skewed anything.

Disk cache (27B, --disk-cache, #14)

  • A conversation saved in the background lost the checkpoints of the file it replaced, so after a restart a new
    conversation with the same system prompt recomputed everything (7,113 of 7,113 tokens in the repro, now 18). The
    system-prompt state now stays through the cap on checkpoints.
  • The cache could forget itself: at every start where the decode calibration wasn't cached yet (a new build, or anything else on a card
    while it measured), the index was cleared after the files had loaded.
  • A covered file is now deleted only after its replacement is written, so a conversation always has a file. I tested that across
    turns, but I did not test a kill in the middle of a write.

Reply parser (both models)

  • Text and tool calls no longer depend on where a decode step happens to end. Before, a real <think> token after the start of a reply
    could by chance send the rest of the answer into reasoning, including a tool call that then never ran, or leave a second
    reasoning block in the answer.
  • A real tool call in an open reasoning block now runs, when it's in the right form and ends the reply with the model's own end of turn.
    This is rare and I don't know how rare; I didn't measure it on a real model.
  • A reply cut inside a character by max_tokens lost that step's whole text. It now keeps the text and drops only the partial character.
  • Stop strings no longer leak. A stop split across two steps left its first part in the reply, and a stop longer than 64 characters
    across steps was missed. The leftmost match now wins. One side effect: text that could start a stop reaches the client a little later.
  • A bad request no longer fails other people's. stop: [1] raised inside the shared step and failed every running request (that part
    is from reading the code and from a one-slot mock), and other bad fields gave a 500 after the prefill had started. These get a
    400 now, before anything starts. chat_template_kwargs that isn't an object still gives a 500, for that request only.
  • Template markers that a message only quotes (<|im_end|>, <think>, <tool_call>, and so on) are plain text now
    (#7). A conversation that pastes a chat template no longer splits into made-up turns and
    ends a reply early: in my check early stops went from 5 of 5 to 0 of 5 on both models, and prompts without markers tokenize the same as before.

Memory pressure and eviction

  • Decode slowing down after a while (#8). After a soak, idle conversations held VRAM
    that the Flash-Next expert cache needed, and single-stream decode ran on a fraction of the cache, slower on my machine at 8 slots. Now
    idle conversations move out when the cache is short.
  • VRAM taken by another program (both models). A browser growing on a card used to be able to take the engine down. Flash-Next now moves resident
    expert groups to host memory and back, and the 27B shrinks its KV pool and grows it back. The watchdog waits while a slow driver call
    runs. With another process taking 1, 2 and 4 GB under load, then 2 GB on the other card, 86 Flash-Next requests were served and none failed; the 27B with 16 GB taken also served everything. I could not reproduce the 2× RTX 5090 crash in club-3090
    #1918 that started this, so I can't say that exact crash is gone.
  • Host RAM taken by another program. The start-up plan follows the size of the machine and warns when MemAvailable is short, the n-gram table falls back to
    the page cache when it really doesn't fit, and while serving the archive gives memory back before the OOM killer acts. Below what the load needs,
    the server exits with a clear message and exit code 3 instead of an abort. With another process taking up to 48 GB mid-run, 214 requests were served, none failed and nothing was OOM-killed; free memory went down to 9 GB.
  • Small cards. A Flash-Next context that doesn't fit is lowered to the largest that does. The 27B lowers its slots and reserve so that the
    pool holds 32K tokens. Loading out of memory is a clear message, not an abort. I made no test runs on 1× RTX 3090 for this
    release, so I make no claim about it. Forced mode A on two 24 GB cards, which aborted at load in 1.3.4.3, now starts, serves and finds a needle at about 33K tokens.
  • One eviction order (both models). Every path that frees memory now uses plain least-recently-used over whole conversations, with the main thread used in
    the last 5 minutes after all the others. A generating conversation is never touched. This replaces older rules that could, for
    example, let an old main refuse other conversations' archive writes for good on the 27B. See below for what this means for you.
  • The 8-turn fork recompute. Eight side turns in place on a main thread pushed the main's own end-of-turn state out of its checkpoints, so
    the next main turn recomputed everything (0 of 60,777 tokens reused; now 60,360 of 60,779).
  • 27B out of VRAM for a graph no longer aborts. 1.3.4.3 aborted there. It now gives up a small reserve (QE_GRAPH_RESERVE_MB, 128) and runs the step without a graph, with one log line.
  • A request that can never fit ends with a clear error. After a KV pool shrank under VRAM pressure, a queued request that fits nowhere while nothing runs now gets its error after QE_NO_ROOM_S (10 s). The error says another process took the GPU memory, with the sizes, or that the request just doesn't fit. The server keeps running. I saw this error on a real 27B at a 20 s setting; the 10 s default itself has only a mock test.
  • The session tracker stopped keeping new sessions after 64 sessions in one server run: a new one was its own eviction victim, so its main thread was never recognised. Fixed.

Start-up crash, env race (#11). One report: a crash at load in 2 of 5 boots on 4 cards. The engine changed environment variables
with threads alive that read them. In an isolated test that pattern crashes on the glibc in our Docker image (2.39) and not on my host's 2.43. I fixed it, and a shim
that logs every write now shows one write, with one thread alive. I must be honest about what this shows: I did not reproduce the crash on the engine. 50 loads inside the image
didn't crash before the fix either, and I have no 4 cards. It's the likely cause and it had to go anyway, but I can't confirm it's what the reporter hit, and I'd like
to hear if it comes back.

Loop guard. The old guard only counted identical turns in a row, so a repeating cycle of a few calls got past it. A real session looped for minutes like that, around a script with an
unchanged result. It now sees short cycles and needs the tool results to repeat exactly, so polls whose output changes are not loops. A guarded turn now
reasons, which is what actually broke the loop in my replays (12 of 12), and it backs off at 3, 6, 12 turns, then every 24, so a long legitimate wait isn't guarded on
every turn.

What behaves differently

  • --disk-cache on Flash-Next now stops the start with an error (#15). Flash-Next never had a disk cache, and a warning in a log is easy to miss. If you pass the flag there, remove it, or the server won't start (under a Docker restart policy it keeps restarting on this error). If you used it only to place the start-up cache, set QE_DISK_CACHE instead. A real disk cache for Flash-Next is planned for a later release.
  • A guarded turn thinks even if the client turned thinking off. That is enable_thinking: false, thinking: {"type": "disabled"} and thinking_budget: 0. It costs at most 2,048 reasoning tokens on top of
    max_tokens. --loop-guard-think 0 turns this off, and "loop_guard": false in a request turns the guard off for that request. A false fire costs one turn.
  • Sampled output differs from 1.3.4.3, also with a fixed seed (the sampler fix above). For how far a repeat holds on Flash-Next, see FLASH_ATTN_CHUNK above. The 27B's greedy output is unchanged.
  • A full archive evicts older conversations. Under pressure, moving an idle conversation out of VRAM may now drop archived conversations older than it, and each loss is logged. Without this, decode fell
    by about 18 % once the archive was full. So on a busy server an old conversation can be recomputed when it returns. The output stays correct.
  • The main thread can leave too. It goes last, but when nothing else can go, it goes, with a log line. There is no guarantee. An idle main no longer makes a new request wait.
  • Bad request fields get a 400. Some of these used to give a 500. stop_sequences as one string is now one stop, instead of being split into characters.
  • A planned archive under 2 GB warns at start, and a small archive may hold nothing for long. The settings stay as they are.
  • On exactly 4 cards with P2P, try FLASH_PAR=tp (#12). auto always picks pp on 4 cards, and the peer access no in its log line doesn't mean your cards lack P2P. One user measured about 2× decode on
    one rig (4× RTX 5080, 116 / 123 to 237 / 303 tok/s at 8K / 32K), with prefill even at 8K and slower at 32K. That's one report on one machine. I can't test 4 cards, and tp couldn't load on another user's 4 × 20 GB cards. The fix
    to the rule itself comes in a later release.

Known limitations

Flash-Next under host RAM pressure can lose more archived conversations than the memory it gives back. Pinned archive
memory is held in 1 GB slabs, conversations share slabs, and a slab returns to the system only when it is completely
empty. When another program takes host RAM, giving back 1 GB can drop several conversations first (1.9 to 11.9 GB in
simulation). The engine keeps serving, each dropped conversation is logged and recomputed when it returns, and output
stays correct. A full archive with RAM to spare doesn't do this. To avoid it, set FLASH_KV_ARCHIVE_GB so the archive
stays below what your machine can spare with your other programs running. Details:
docs/advanced.md.

Flash-Next text between runs. At default settings greedy text can differ between runs at near-ties. FLASH_ATTN_CHUNK=1 removes this at the cost above, and I can't yet explain why an earlier load on the cards changes the default results.

27B disk cache. The files have no checksum and no fsync, so a power loss during a write could leave a file of the right size with bad contents. With the default shared prefix, a conversation attached to a pinned
prefix doesn't survive a restart; only the system-prompt state does. A write that has been superseded still runs to its end.

Conversations under memory pressure. A conversation the reclaim has just written to the archive can be evicted by a newer write, with a log line. With a full archive, under pressure on Flash-Next, the oldest idle conversation whose write is refused leaves VRAM anyway. If it has no earlier state in the archive it is lost, with a log line, and is recomputed when it returns; the output stays correct. Only an active main thread stays in VRAM in that case, also with a log line: larger than the archive, it would otherwise be recomputed on every turn.

27B out of graph memory. Only plain steps survive this. A batched or speculative step may still abort if a graph outgrows QE_GRAPH_RESERVE_MB.

A request that used to wait for memory can now fail. On the 27B, QE_NO_ROOM_S (10 s) ends a queued request that fits nowhere while nothing runs, for example when another process took the GPU memory and the KV pool shrank. It used to wait until the memory came back. A larger value makes it wait longer; 0 is no deadline.

Two reports I can't close. The FLASH_PAR=tp workaround for 4 cards (#12) is unverified by me, and the start-up crash (#11) wasn't reproduced on the engine.

Loop guard. It catches repeats with identical tool results. Plain-text loops aren't caught. An edit / test (same failure) / ls pattern can reach a guarded turn on each pass, which is another reason for the back-off.

Upgrading

docker pull ghcr.io/bpx03/qengine:1.3.5          # or :latest (text only)
docker pull ghcr.io/bpx03/qengine:1.3.5-vision   # or :vision (with the image encoder)

From source: git pull and rebuild. No new model download. The first start on a new build measures the decode paths once, which takes a few seconds, and then caches the result.