Repository navigation
QEngine 1.3.5.5 (experimental)
Pre-releaseQEngine 1.3.5.5 (experimental)
This build fixes four things reported in club-3090#1966: Flash-Next on 4 cards losing details in long prompts, reasoning_effort: "none" still thinking, a served context larger than the KV pool, and Flash-Next pushing a host short of RAM into swap with FLASH_KV=ram. It also contains everything from 1.3.5.4. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.
docker pull ghcr.io/bpx03/qengine:1.3.5.5-vision # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.5 # text onlyFrom source: git fetch --tags && git checkout v1.3.5.5, then rebuild.
Fixes
- Flash-Next on 4 cards lost details in long prompts. With four GPUs, one small DeltaNet projection wrote its output rows wider than they were spaced, so part of the gate values of the next token were overwritten, in a different order on every run. Long-context recall suffered first: the right phrase came back without its trailing number. Fixed. 2 and 3 cards were never affected and give the same output as before.
reasoning_effort: "none"now turns thinking off. Before, the server read it as an unknown effort and renderedmediumwith thinking on.none(any case) now works likeenable_thinking: falsein every place an effort can come from:chat_template_kwargs.reasoning_effort, the top-levelreasoning_effort, the Responses / OpenRouter-stylereasoning: {"effort": ...}, Anthropicoutput_config.effortand the server default (--reasoning-effort none/QE_REASONING_EFFORT=none). When a request carries bothnoneandenable_thinking: true, thinking is off.- Served context never exceeds the KV pool. When the pool is smaller than
--max-ctx, the server still starts and serves at most the pool's length. The start-up log printscontext: requested ..., KV pool ..., served ...for both models, and/v1/modelsreports the served length. - Flash-Next no longer overcommits host RAM with
FLASH_KV=ram. The KV pool in host RAM was left out of the host-RAM plan, so the KV archive was given the same memory a second time (on a 134.5 GB host the plan added up to 148 GB). Now the experts, the pinned n-gram table and the pool must fitMemAvailableminus the margin at start. When they don't, the archive goes first, then the pool shrinks to seats × window, then to one window, then the n-gram table moves to the page cache. Experts are never reduced. Thehost RAM planstart-up line shows the pool and what stays free. The runtime guard also respects a cgroup limit (docker --memory, systemdMemoryMax), for both models.
The default KV pool in host RAM (Flash-Next, FLASH_KV=ram)
A start-up line such as KV pool 1,000,448 tokens is the pool shared by all conversations together, not the longest prompt you can send. Its default is FLASH_KV_RAM_TOKENS=1M (rounded to whole 1,024-token blocks), the same on 2, 3 or 4 cards. One conversation is still limited by the window (--max-ctx, 262,144 by default, up to 1M), and the new context: requested ..., KV pool ..., served ... line shows both numbers side by side.
What the pool costs and how to size it:
FLASH_KV_RAM_TOKENS |
pinned host RAM (all cards together) | VRAM per card for indexer keys, 3 / 2 cards |
|---|---|---|
| 1M (default) | 13.5 GB | 277 / 416 MB |
| 4M | 54 GB | 1.1 / 1.7 GB |
| 10M | 135 GB | 2.8 / 4.2 GB |
- More long conversations at once: raise it (for example
4M). Each conversation keeps its own tokens in the shared pool, so 4M holds four 1M conversations or many shorter ones. - A host with little RAM: lower it to what you use, for example
FLASH_KV_RAM_TOKENS=300Kfor one 262K conversation plus a little room. Since this release the engine also fits the pool, the KV archive and the pinned n-gram table underMemAvailableat start, so it no longer pushes the host into swap. On 1.3.5.4 and older, set it by hand and lowerFLASH_KV_ARCHIVE_GBas well. - Pin only what you use:
FLASH_KV_RAM_LAZY=1pins the pool in slabs as conversations grow (on GPUs that can use registered host memory directly). See the limit in the table below. - Without
FLASH_KV=ramthe KV pool lives in VRAM (262,144 tokens shared) and none of this applies.
Known limits and how to work around them
| what you may see | why | what to do |
|---|---|---|
27B with --max-ctx 1M on 2 cards serves about 659K tokens |
its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights | use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get |
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it |
lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, the pool can't reach the full window | keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit |
| Flash-Next on a host with little RAM pages to swap (versions up to 1.3.5.4) | the KV pool was counted twice (see Fixes) | upgrade; on older versions lower FLASH_KV_ARCHIVE_GB (for example by the size of the KV pool) or turn swap off |
| temperature-0 replies differ slightly between runs | the default attention path sums in an order that depends on timing, for speed | set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed |
an OpenRouter client sends reasoning: {"enabled": false} and still gets thinking |
reasoning.enabled and reasoning.exclude aren't read yet |
send reasoning: {"effort": "none"}, reasoning_effort: "none" or chat_template_kwargs: {"enable_thinking": false} |
| Flash-Next on 4 cards still misses a needle after this release | the fix was verified with four ranks on three cards, not yet on a real 4-card machine | run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log |
| a small host still pages out a little while the model loads | the model files fill the page cache next to the pinned memory, and the kernel reclaims hard when both reach the limit | raise QE_MEM_MARGIN_GB by a few GB, or lower FLASH_KV_RAM_TOKENS; turning swap off avoids paging the server out |
on some multi-socket / Gen3 boards one stream decodes slower than with QE_P2P=0 on 1.3.4.3 |
the host-memory decode path adds a memory fence per thread (QE_DECODE_HOST_FENCE=1) for PCIe ordering; under investigation (#20) |
try QE_DECODE_COMM=host QE_DECODE_HOST_FENCE=0 and check that the output looks right; QE_P2P=0 turns peer access off for everything (prefill gets slower) |
What I tested
On 3× RTX 3090, from a clean build of this release:
| check | result |
|---|---|
| Flash-Next and 27B start, 3 cards | ready, no error lines |
| a chat on each | correct answer |
| greedy output on 2 and 3 cards (Flash-Next) and the 27B identity set, against 1.3.5.4 | identical, so speed is unchanged too; one decode run per model matched 1.3.5.4 |
reasoning_effort: "none", reasoning: {"effort": "none"}, and none with enable_thinking: true |
no reasoning on both models |
context: and host RAM plan start-up lines |
present on both models |
start with --vision 1 and an image question |
correct description |
| the 4-card fix | four ranks on three cards: the needle is recalled, 2- and 3-card output unchanged |
What I didn't test: a real 4-card machine, a host short of RAM on this build (the plan was checked on its own branch), the images themselves on a GPU, speed suites, KL or the 8-pack on this build.
Thanks
Thanks to @Hazelhof for the detailed report in club-3090#1966, which led to all four fixes, and to @cristianflorett for the decode report in #20.
What I'd like to hear
If you have 4 cards, please try a long-context recall on Flash-Next and send the start-up log. If you run with FLASH_KV=ram on a smaller host, the host RAM plan line helps. The issue tracker is the place.