Skip to content

QEngine 1.3.5.7

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 22:10
· 6 commits to main since this release

QEngine 1.3.5.7

This is the new stable release: latest, vision and the main branch move here from 1.3.5.6. The main change: --lsa, the sparse attention for long 27B prompts, now works with no manual steps. It also drops the model files from the page cache while the model loads, and fixes a few small things found in code reviews. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.7, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.7-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.7, then rebuild.

--lsa without manual steps (27B)

At 512K or 1M tokens most of the 27B prompt time goes to attention, and that share grows with depth. --lsa makes that attention sparse, but until now you had to download the index map and convert it with torch yourself, so few people tried it. Now this is all it takes:

server/run.sh 8080 --model models/Qwen3.8-27B-INT4 --max-ctx 1M --lsa
# Docker: -e QE_EXTRA_ARGS="--max-ctx 1M --lsa" (or -e QE_LSA=auto), with the qengine-cache volume mounted
  • On the first start the server downloads the released index map (6.3 MB, from the authors' GitHub release d-lsa-weights-v1 of Prefrontal AI's latent-sparse-attention repository) from a pinned URL, checks its sha256 and converts it to safetensors. The conversion runs in plain Python: the file's pickle goes through a restricted reader, so nothing in the file runs, and torch isn't needed. The result is checked against a pinned sha256 too and cached in ~/.cache/qengine/lsa (in Docker, the qengine-cache volume). Later starts use the cached file without network.
  • QEngine doesn't ship the map; it is fetched from upstream. The start-up log names the map, its source and the cache path.
  • Offline, or with QE_FETCH_ASSETS=0 and no cached map, the server starts with full attention and logs a curl line to fetch the file by hand. It never refuses to start because of the map.
  • --lsa MAP (or QE_LSA=MAP) still takes a map file, and a released .pt is converted into the cache. The documented forms are now just --lsa / --lsa MAP and QE_LSA=auto / QE_LSA=MAP; older spellings (--lsa auto, QE_LSA=1) still work.

Measured on 3× RTX 3090, --max-ctx 1M, one needle prompt, engine-level prompt time without / with --lsa:

prompt without --lsa with --lsa needle
92,160 tokens 20.9 s 20.6 s found / found
262,144 tokens 78.6 s 57.4 s found / found
524,288 tokens 232 s 117 s found / found

With --lsa the prompt speed stays about flat with depth (about 4,480 tok/s at 90K, 262K and 512K), while with full attention it falls from 4,400 to 2,260 tok/s. It is lossy, and only one long conversation at a time uses it; the trade-off and its limits are in docs/advanced.md.

The model files leave the page cache during the load

While the model loads, the engine reads tens of GB of model files. Their pages stayed in the page cache next to the memory the engine pins at the same time, so on a host or in a container short of RAM the kernel had to reclaim hard, and it could page other things out. Now each layer's file pages are dropped once every card has read that layer (QE_LOAD_DROP).

  • Flash-Next: on by default. The n-gram table keeps its pages. In a 110 GB container the load went from 28 million directly reclaimed pages to none, with the same load time and the same output. QE_LOAD_DROP=0 (or FLASH_WEIGHTS_COLD=0) turns it off.
  • 27B: automatic. It is on only when free RAM is short of the files plus the default archive and pinned prompts; on a large host the files stay cached, so a restart stays quick. QE_LOAD_DROP=1 forces it on. The start-up line load: model file pages ... says which it picked.

Small fixes

  • Grammar-constrained image prompts with the opt-in grammar lookup (QE_LK_GRAMMAR=1) no longer fail. A lookup draft that ran into an image row indexed past the grammar's masks and the request failed. Image rows now enter that lookup as <|image_pad|>, as the engines' own lookups do since 1.3.5.6.
  • 27B: two image prompts prefilling at once each keep their own image rows. Up to two prompts can be in the prefill at the same time, and one image table held only the newest prompt's rows, so the first could embed the second one's image. In a test that overlaps two image prompts, the first one's logits now match the run alone (before: up to 0.44 apart).
  • 27B host-RAM plan counts only the snapshots it pins at start. The plan counted the whole snapshot store, while only the first 32 entries are pinned at start (the rest only when the store fills and the host has room). The 1.3.5.6 known limit about larger engine buffers in the host RAM plan line is gone (QE_SNAPSHOTS=128: 23.9 against 7.9 GB), so a small host gives that room to the KV archive.
  • OpenRouter reasoning.enabled and reasoning.exclude are read. reasoning: {"enabled": false} turns thinking off when the request sends no other thinking setting; "enabled": true turns it on over the server default at the effort the request names. reasoning: {"exclude": true} on /v1/chat/completions: the model reasons as usual (the reasoning tokens are counted in usage), but the reply carries no reasoning_content, streamed or not.
  • Flash-Next with FLASH_KV_RAM_LAZY=1 reports the pool it can really use. The context: start-up line and the no-room error now show what is pinned plus what the host can still pin, not the full reserved pool. One full conversation window is pinned from the start, so a conversation of the full --max-ctx always fits.

Known limits and how to work around them

what you may see why what to do
a 27B turn after an image, or after a prompt under 4,096 tokens, prefills a few thousand tokens again the state at the very end of such a prompt isn't stored, so the turn resumes from the nearest checkpoint before it nothing to do; this is the same as in earlier releases
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
--lsa logs LSA: off: ... and the server runs with full attention no network and no cached map, a checkpoint without a released map, or a setup the sparse path doesn't support (no peer access between the cards, an uneven card split, a KV pool too small for one conversation above 262K) the log line says which; for the map, fetch it with the curl line from the log into the cache directory and start again
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it beyond one conversation lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, more conversations can't get room keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
Flash-Next on 4 cards still misses a needle the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
decode is slower than expected on a board with peer access the start-up check picked P2P or the host path from a few timed steps, and your real load may favour the other one look at the [comm] decode collectives start-up line (both timings and the pick); compare QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and tell me which is faster, with that line

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards, also Flash-Next with --vision 1 ready, no error lines
a chat on each, a too-long prompt correct answer; HTTP 400 for the too-long prompt
greedy output on 2 and 3 cards (Flash-Next), also with QE_DECODE_COMM=host, and the 27B identity set, against 1.3.5.6 identical
Flash-Next decode step, 3 cards, one stream P2P 8.04 ms, host path 8.33 ms (1.3.5.6: 7.99 / 8.36)
27B single-stream decode, engine level 123.8 tok/s (1.3.5.6: 124.3)
--max-ctx 1M --lsa on the 27B with an empty cache map downloaded and checked, sparse attention on; needle found at 80K tokens (15.2 s) and at 266K tokens (56.3 s, whole request)
load-time page-cache drop Flash-Next dropped 67 GB of file ranges during the load; 27B kept the files cached (auto, large host)
reasoning_effort: "none" in its three forms, effort low no reasoning / short reasoning on both models
context: and host RAM plan start-up lines present on both models
an image question with --vision 1 correct description
the 1.3.5.6 image crash repro (27B, 2 cards, 32 slots, vision on, 4 parallel requests, 10 rounds) 40 of 40 correct, lookup drafts active

What I didn't test: a real 4-card machine, a host or container short of RAM on this build (the load-time drop was measured on its own branch), the Docker images beyond a start and a chat per image, and the long release runs (262K stress, KL, the 8-pack). Those ran on 1.3.5, and the README numbers are from 1.3.5.

What I'd like to hear

If you run the 27B on long documents, try --max-ctx 1M --lsa and tell me how long your prompts take and whether the answers hold up. If your host is short of RAM, the host RAM plan and load: start-up lines help. The issue tracker is the place.