Repository navigation
QEngine 1.3.5.7
QEngine 1.3.5.7
This is the new stable release: latest, vision and the main branch move here from 1.3.5.6. The main change: --lsa, the sparse attention for long 27B prompts, now works with no manual steps. It also drops the model files from the page cache while the model loads, and fixes a few small things found in code reviews. The weights are the same, so there's nothing new to download.
docker pull ghcr.io/bpx03/qengine:latest # = 1.3.5.7, text only
docker pull ghcr.io/bpx03/qengine:vision # = 1.3.5.7-vision, with the image encoderFrom source: git fetch --tags && git checkout v1.3.5.7, then rebuild.
--lsa without manual steps (27B)
At 512K or 1M tokens most of the 27B prompt time goes to attention, and that share grows with depth. --lsa makes that attention sparse, but until now you had to download the index map and convert it with torch yourself, so few people tried it. Now this is all it takes:
server/run.sh 8080 --model models/Qwen3.8-27B-INT4 --max-ctx 1M --lsa
# Docker: -e QE_EXTRA_ARGS="--max-ctx 1M --lsa" (or -e QE_LSA=auto), with the qengine-cache volume mounted- On the first start the server downloads the released index map (6.3 MB, from the authors' GitHub release d-lsa-weights-v1 of Prefrontal AI's latent-sparse-attention repository) from a pinned URL, checks its sha256 and converts it to safetensors. The conversion runs in plain Python: the file's pickle goes through a restricted reader, so nothing in the file runs, and torch isn't needed. The result is checked against a pinned sha256 too and cached in
~/.cache/qengine/lsa(in Docker, theqengine-cachevolume). Later starts use the cached file without network. - QEngine doesn't ship the map; it is fetched from upstream. The start-up log names the map, its source and the cache path.
- Offline, or with
QE_FETCH_ASSETS=0and no cached map, the server starts with full attention and logs acurlline to fetch the file by hand. It never refuses to start because of the map. --lsa MAP(orQE_LSA=MAP) still takes a map file, and a released.ptis converted into the cache. The documented forms are now just--lsa/--lsa MAPandQE_LSA=auto/QE_LSA=MAP; older spellings (--lsa auto,QE_LSA=1) still work.
Measured on 3× RTX 3090, --max-ctx 1M, one needle prompt, engine-level prompt time without / with --lsa:
| prompt | without --lsa |
with --lsa |
needle |
|---|---|---|---|
| 92,160 tokens | 20.9 s | 20.6 s | found / found |
| 262,144 tokens | 78.6 s | 57.4 s | found / found |
| 524,288 tokens | 232 s | 117 s | found / found |
With --lsa the prompt speed stays about flat with depth (about 4,480 tok/s at 90K, 262K and 512K), while with full attention it falls from 4,400 to 2,260 tok/s. It is lossy, and only one long conversation at a time uses it; the trade-off and its limits are in docs/advanced.md.
The model files leave the page cache during the load
While the model loads, the engine reads tens of GB of model files. Their pages stayed in the page cache next to the memory the engine pins at the same time, so on a host or in a container short of RAM the kernel had to reclaim hard, and it could page other things out. Now each layer's file pages are dropped once every card has read that layer (QE_LOAD_DROP).
- Flash-Next: on by default. The n-gram table keeps its pages. In a 110 GB container the load went from 28 million directly reclaimed pages to none, with the same load time and the same output.
QE_LOAD_DROP=0(orFLASH_WEIGHTS_COLD=0) turns it off. - 27B: automatic. It is on only when free RAM is short of the files plus the default archive and pinned prompts; on a large host the files stay cached, so a restart stays quick.
QE_LOAD_DROP=1forces it on. The start-up lineload: model file pages ...says which it picked.
Small fixes
- Grammar-constrained image prompts with the opt-in grammar lookup (
QE_LK_GRAMMAR=1) no longer fail. A lookup draft that ran into an image row indexed past the grammar's masks and the request failed. Image rows now enter that lookup as<|image_pad|>, as the engines' own lookups do since 1.3.5.6. - 27B: two image prompts prefilling at once each keep their own image rows. Up to two prompts can be in the prefill at the same time, and one image table held only the newest prompt's rows, so the first could embed the second one's image. In a test that overlaps two image prompts, the first one's logits now match the run alone (before: up to 0.44 apart).
- 27B host-RAM plan counts only the snapshots it pins at start. The plan counted the whole snapshot store, while only the first 32 entries are pinned at start (the rest only when the store fills and the host has room). The 1.3.5.6 known limit about larger engine buffers in the
host RAM planline is gone (QE_SNAPSHOTS=128: 23.9 against 7.9 GB), so a small host gives that room to the KV archive. - OpenRouter
reasoning.enabledandreasoning.excludeare read.reasoning: {"enabled": false}turns thinking off when the request sends no other thinking setting;"enabled": trueturns it on over the server default at the effort the request names.reasoning: {"exclude": true}on/v1/chat/completions: the model reasons as usual (the reasoning tokens are counted inusage), but the reply carries noreasoning_content, streamed or not. - Flash-Next with
FLASH_KV_RAM_LAZY=1reports the pool it can really use. Thecontext:start-up line and the no-room error now show what is pinned plus what the host can still pin, not the full reserved pool. One full conversation window is pinned from the start, so a conversation of the full--max-ctxalways fits.
Known limits and how to work around them
| what you may see | why | what to do |
|---|---|---|
| a 27B turn after an image, or after a prompt under 4,096 tokens, prefills a few thousand tokens again | the state at the very end of such a prompt isn't stored, so the turn resumes from the nearest checkpoint before it | nothing to do; this is the same as in earlier releases |
27B with --max-ctx 1M on 2 cards serves about 659K tokens |
its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights | use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get |
--lsa logs LSA: off: ... and the server runs with full attention |
no network and no cached map, a checkpoint without a released map, or a setup the sparse path doesn't support (no peer access between the cards, an uneven card split, a KV pool too small for one conversation above 262K) | the log line says which; for the map, fetch it with the curl line from the log into the cache directory and start again |
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it beyond one conversation |
lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, more conversations can't get room | keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit |
| temperature-0 replies differ slightly between runs | the default attention path sums in an order that depends on timing, for speed | set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed |
| Flash-Next on 4 cards still misses a needle | the 1.3.5.5 fix was verified with four ranks on three cards, not yet on a real 4-card machine | run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log |
| decode is slower than expected on a board with peer access | the start-up check picked P2P or the host path from a few timed steps, and your real load may favour the other one | look at the [comm] decode collectives start-up line (both timings and the pick); compare QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and tell me which is faster, with that line |
What I tested
On 3× RTX 3090, from a clean build of this release:
| check | result |
|---|---|
Flash-Next and 27B start, 3 cards, also Flash-Next with --vision 1 |
ready, no error lines |
| a chat on each, a too-long prompt | correct answer; HTTP 400 for the too-long prompt |
greedy output on 2 and 3 cards (Flash-Next), also with QE_DECODE_COMM=host, and the 27B identity set, against 1.3.5.6 |
identical |
| Flash-Next decode step, 3 cards, one stream | P2P 8.04 ms, host path 8.33 ms (1.3.5.6: 7.99 / 8.36) |
| 27B single-stream decode, engine level | 123.8 tok/s (1.3.5.6: 124.3) |
--max-ctx 1M --lsa on the 27B with an empty cache |
map downloaded and checked, sparse attention on; needle found at 80K tokens (15.2 s) and at 266K tokens (56.3 s, whole request) |
| load-time page-cache drop | Flash-Next dropped 67 GB of file ranges during the load; 27B kept the files cached (auto, large host) |
reasoning_effort: "none" in its three forms, effort low |
no reasoning / short reasoning on both models |
context: and host RAM plan start-up lines |
present on both models |
an image question with --vision 1 |
correct description |
| the 1.3.5.6 image crash repro (27B, 2 cards, 32 slots, vision on, 4 parallel requests, 10 rounds) | 40 of 40 correct, lookup drafts active |
What I didn't test: a real 4-card machine, a host or container short of RAM on this build (the load-time drop was measured on its own branch), the Docker images beyond a start and a chat per image, and the long release runs (262K stress, KL, the 8-pack). Those ran on 1.3.5, and the README numbers are from 1.3.5.
What I'd like to hear
If you run the 27B on long documents, try --max-ctx 1M --lsa and tell me how long your prompts take and whether the answers hold up. If your host is short of RAM, the host RAM plan and load: start-up lines help. The issue tracker is the place.