Skip to content

QEngine 1.3.5.6

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 18:39
· 7 commits to main since this release

QEngine 1.3.5.6

This is the new stable release: latest, vision and the main branch move here from 1.3.5. It fixes slow decode through host memory on boards where P2P stores are expensive, a 27B crash with images while the n-gram lookup was drafting, and 27B agent conversations under load losing their prompt cache between turns. It also contains everything from the experimental 1.3.5.1 to 1.3.5.5 builds. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.6, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.6-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.6, then rebuild.

Fixes

  • Decode through host memory is fast again (#21, thanks @jorgermp; #20, thanks @cristianflorett). Since 1.3.5 both engines time decode on two paths at start-up and keep the faster one. The host candidate in 1.3.5 to 1.3.5.5 was not the path 1.3.4.3 used without peer access: it ran the P2P kernels over per-receiver host buffers, and every attention block read its exchange rows from host memory. On 3 cards it took 12.1 ms per step against 8.3 ms for the old path. So on boards where P2P stores are expensive (some Gen3 and multi-socket machines), the start-up check could only choose between two slow options, and one stream decoded far slower than on 1.3.4.3 (210 against about 130 tok/s on one 4-card box). Now the host path is the old no-peer one: each card writes its words once into pinned host memory, the other cards read them, and Flash-Next copies its attention exchanges into device memory in one step. Peer access stays on, so prefill keeps its P2P or copy-engine path. On our 3-card box P2P still wins (8.0 against 8.3 ms per step), and the output is identical on both paths.
  • 27B crash with images (illegal memory access, Xid 13). The n-gram lookup drafts the next tokens from the conversation's own history. For image rows that history held the ids the server derives from the image content, not the image placeholder token, so a draft that copied an image could hand the embedding an index far past its table, and every card faulted at once. It showed up as cudaErrorIllegalAddress in a batched step, often while the KV pool was shrinking under VRAM pressure, but the pool was not the cause. Image rows now enter the lookup history as <|image_pad|>, as Flash-Next already did. Flash-Next was not affected. Text-only conversations give the same output as before.
  • 27B agent conversations kept their prompt cache. The store of recurrent-state snapshots, which lets a new turn resume where the previous prompt ended, had 32 entries for every slot count. With 8 agent conversations on 32 slots they pushed each other's states out, and turns resumed from the system prompt: in a test with a VRAM hog, about 308K tokens were prefilled again, against 49K now, and the agent throughput went from 94.8 to 148.8 tok/s. The store now holds 4 snapshots per sequence slot (at least 32; QE_SNAPSHOTS overrides it). The first 32 are pinned at start as before; the rest are pinned only as the store fills, and only while the host has room. When host RAM runs short, unused snapshot buffers are freed and the store stays at its size.

QE_DECODE_COMM: how decode picks its path

Decode sends many small messages per step (all-reduces, the attention exchanges). They go either as P2P stores into the other cards' memory, or through the host path described above.

  • auto (the default): at the end of the start-up the engine runs a few seconds of dummy decode on both paths, with one sequence and with four batched sequences, and keeps the faster path for each. The host path has to win clearly (by 5 %, in at least 3 of 4 rounds). The [comm] decode collectives line shows both timings and the pick. The result is cached in ~/.cache/qengine (the Docker volume) and measured again when the cards, the driver or the build change; QE_DECODE_COMM=remeasure measures again once.
  • QE_DECODE_COMM=host forces the host path for decode. Prefill keeps peer access.
  • QE_DECODE_COMM=p2p forces P2P stores for decode.
  • QE_P2P=0 still turns peer access off for everything, prefill included, which is usually slower for long prompts.

The old QE_DECODE_HOST_FENCE setting is gone with the old host path; if you set it, it is ignored.

Known limits and how to work around them

what you may see why what to do
the 27B host RAM plan start-up line shows larger engine buffers than 1.3.5.5 (13.3 against 7.9 GB on 3 cards with the default 16 slots) the plan counts the whole snapshot store (4 per slot, about 80 MB each), although entries past the first 32 are pinned only when needed; on a host short of RAM the KV archive starts smaller than it has to nothing on a large host; on a small one set QE_SNAPSHOTS=32 (the old size) or fewer slots (QE_MAX_SEQS)
a 27B turn after an image, or after a prompt under 4,096 tokens, prefills a few thousand tokens again the state at the very end of such a prompt isn't stored, so the turn resumes from the nearest checkpoint before it nothing to do; this is the same as in earlier releases
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, the pool can't reach the full window keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
Flash-Next on a host with little RAM pages to swap (versions up to 1.3.5.4) the KV pool was counted twice (fixed in 1.3.5.5) upgrade; on older versions lower FLASH_KV_ARCHIVE_GB (for example by the size of the KV pool) or turn swap off
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
an OpenRouter client sends reasoning: {"enabled": false} and still gets thinking reasoning.enabled and reasoning.exclude aren't read yet send reasoning: {"effort": "none"}, reasoning_effort: "none" or chat_template_kwargs: {"enable_thinking": false}
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
a small host still pages out a little while the model loads the model files fill the page cache next to the pinned memory, and the kernel reclaims hard when both reach the limit raise QE_MEM_MARGIN_GB by a few GB, or lower FLASH_KV_RAM_TOKENS; turning swap off avoids paging the server out
decode is slower than expected on a board with peer access the start-up check picked P2P or the host path from a few timed steps, and your real load may favour the other one look at the [comm] decode collectives start-up line (both timings and the pick); compare QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and tell me which is faster, with that line

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards ready, no error lines
a chat on each correct answer
greedy output on 2 and 3 cards (Flash-Next) and the 27B identity set, against 1.3.5.5 identical
the same greedy runs with QE_DECODE_COMM=host identical to P2P
Flash-Next decode step, 3 cards, one stream P2P 7.99 ms, host path 8.36 ms (the 1.3.5.5 host path: 12.1 ms)
auto on this box picks P2P for one and for four sequences, on both models
reasoning_effort: "none", reasoning: {"effort": "none"}, and none with enable_thinking: true no reasoning on both models
context: and host RAM plan start-up lines present on both models
start with --vision 1 and an image question correct description
the image crash: 27B on 2 cards, 32 slots, vision on, 4 parallel requests that make the lookup copy an image, 10 rounds 40 of 40 correct with lookup drafts active, no error lines (1.3.5.5 crashed on the first request)
27B single-stream decode, 1.3.5.5 / 1.3.5.6, 3 and 2 cards, alternating runs (club-3090 bench) unchanged within run-to-run noise (3 cards: narrative 261 / 266, code 409 / 402 tok/s; 2 cards: narrative 204 / 205, code 315 / 318 tok/s; engine-level decode 124.0 / 124.1 tok/s, 124.3 after the decode path change)

What I didn't test: a real 4-card machine or a Gen3 board where the host path should win (the reporters' boxes), a host short of RAM on this build (the snapshot store's host-short path was checked on its own branch), the images themselves on a GPU, and the long release runs (262K stress, KL, the 8-pack): they ran on 1.3.5, and the numbers in the README are from 1.3.5.

What I'd like to hear

If your board picks the host path, or you compare QE_DECODE_COMM=host and p2p, the [comm] decode collectives line and your tok/s help a lot. If you run the 27B with agents or images, a start-up log and a note on whether later turns start fast helps. The issue tracker is the place.