Skip to content

QEngine 1.3.5.5 (experimental)

Pre-release
Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 09 Nov 13:24
· 8 commits to main since this release

QEngine 1.3.5.5 (experimental)

This build fixes four things reported in club-3090#1966: Flash-Next on 4 cards losing details in long prompts, reasoning_effort: "none" still thinking, a served context larger than the KV pool, and Flash-Next pushing a host short of RAM into swap with FLASH_KV=ram. It also contains everything from 1.3.5.4. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.

docker pull ghcr.io/bpx03/qengine:1.3.5.5-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.5          # text only

From source: git fetch --tags && git checkout v1.3.5.5, then rebuild.

Fixes

  • Flash-Next on 4 cards lost details in long prompts. With four GPUs, one small DeltaNet projection wrote its output rows wider than they were spaced, so part of the gate values of the next token were overwritten, in a different order on every run. Long-context recall suffered first: the right phrase came back without its trailing number. Fixed. 2 and 3 cards were never affected and give the same output as before.
  • reasoning_effort: "none" now turns thinking off. Before, the server read it as an unknown effort and rendered medium with thinking on. none (any case) now works like enable_thinking: false in every place an effort can come from: chat_template_kwargs.reasoning_effort, the top-level reasoning_effort, the Responses / OpenRouter-style reasoning: {"effort": ...}, Anthropic output_config.effort and the server default (--reasoning-effort none / QE_REASONING_EFFORT=none). When a request carries both none and enable_thinking: true, thinking is off.
  • Served context never exceeds the KV pool. When the pool is smaller than --max-ctx, the server still starts and serves at most the pool's length. The start-up log prints context: requested ..., KV pool ..., served ... for both models, and /v1/models reports the served length.
  • Flash-Next no longer overcommits host RAM with FLASH_KV=ram. The KV pool in host RAM was left out of the host-RAM plan, so the KV archive was given the same memory a second time (on a 134.5 GB host the plan added up to 148 GB). Now the experts, the pinned n-gram table and the pool must fit MemAvailable minus the margin at start. When they don't, the archive goes first, then the pool shrinks to seats × window, then to one window, then the n-gram table moves to the page cache. Experts are never reduced. The host RAM plan start-up line shows the pool and what stays free. The runtime guard also respects a cgroup limit (docker --memory, systemd MemoryMax), for both models.

The default KV pool in host RAM (Flash-Next, FLASH_KV=ram)

A start-up line such as KV pool 1,000,448 tokens is the pool shared by all conversations together, not the longest prompt you can send. Its default is FLASH_KV_RAM_TOKENS=1M (rounded to whole 1,024-token blocks), the same on 2, 3 or 4 cards. One conversation is still limited by the window (--max-ctx, 262,144 by default, up to 1M), and the new context: requested ..., KV pool ..., served ... line shows both numbers side by side.

What the pool costs and how to size it:

FLASH_KV_RAM_TOKENS pinned host RAM (all cards together) VRAM per card for indexer keys, 3 / 2 cards
1M (default) 13.5 GB 277 / 416 MB
4M 54 GB 1.1 / 1.7 GB
10M 135 GB 2.8 / 4.2 GB
  • More long conversations at once: raise it (for example 4M). Each conversation keeps its own tokens in the shared pool, so 4M holds four 1M conversations or many shorter ones.
  • A host with little RAM: lower it to what you use, for example FLASH_KV_RAM_TOKENS=300K for one 262K conversation plus a little room. Since this release the engine also fits the pool, the KV archive and the pinned n-gram table under MemAvailable at start, so it no longer pushes the host into swap. On 1.3.5.4 and older, set it by hand and lower FLASH_KV_ARCHIVE_GB as well.
  • Pin only what you use: FLASH_KV_RAM_LAZY=1 pins the pool in slabs as conversations grow (on GPUs that can use registered host memory directly). See the limit in the table below.
  • Without FLASH_KV=ram the KV pool lives in VRAM (262,144 tokens shared) and none of this applies.

Known limits and how to work around them

what you may see why what to do
27B with --max-ctx 1M on 2 cards serves about 659K tokens its KV cache lives in VRAM, and two 24 GB cards hold about 659K tokens next to the weights use 3 or more cards for the full window (a 4-card run reached a 1.09M pool), or keep 659K; the context: start-up line shows what you get
a very long request is refused with FLASH_KV_RAM_LAZY=1 although the window allows it lazy pinning grows the host-RAM pool as conversations grow; if other programs took the RAM meanwhile, the pool can't reach the full window keep the default FLASH_KV_RAM_LAZY=0 (the whole pool is pinned at start), or free host RAM; the error says how many tokens fit
Flash-Next on a host with little RAM pages to swap (versions up to 1.3.5.4) the KV pool was counted twice (see Fixes) upgrade; on older versions lower FLASH_KV_ARCHIVE_GB (for example by the size of the KV pool) or turn swap off
temperature-0 replies differ slightly between runs the default attention path sums in an order that depends on timing, for speed set FLASH_ATTN_CHUNK=1 for reproducible greedy runs (benchmarks, evals); it costs some speed
an OpenRouter client sends reasoning: {"enabled": false} and still gets thinking reasoning.enabled and reasoning.exclude aren't read yet send reasoning: {"effort": "none"}, reasoning_effort: "none" or chat_template_kwargs: {"enable_thinking": false}
Flash-Next on 4 cards still misses a needle after this release the fix was verified with four ranks on three cards, not yet on a real 4-card machine run 3 cards (--devices with three GPUs) for long context, and please report it with the start-up log
a small host still pages out a little while the model loads the model files fill the page cache next to the pinned memory, and the kernel reclaims hard when both reach the limit raise QE_MEM_MARGIN_GB by a few GB, or lower FLASH_KV_RAM_TOKENS; turning swap off avoids paging the server out
on some multi-socket / Gen3 boards one stream decodes slower than with QE_P2P=0 on 1.3.4.3 the host-memory decode path adds a memory fence per thread (QE_DECODE_HOST_FENCE=1) for PCIe ordering; under investigation (#20) try QE_DECODE_COMM=host QE_DECODE_HOST_FENCE=0 and check that the output looks right; QE_P2P=0 turns peer access off for everything (prefill gets slower)

What I tested

On 3× RTX 3090, from a clean build of this release:

check result
Flash-Next and 27B start, 3 cards ready, no error lines
a chat on each correct answer
greedy output on 2 and 3 cards (Flash-Next) and the 27B identity set, against 1.3.5.4 identical, so speed is unchanged too; one decode run per model matched 1.3.5.4
reasoning_effort: "none", reasoning: {"effort": "none"}, and none with enable_thinking: true no reasoning on both models
context: and host RAM plan start-up lines present on both models
start with --vision 1 and an image question correct description
the 4-card fix four ranks on three cards: the needle is recalled, 2- and 3-card output unchanged

What I didn't test: a real 4-card machine, a host short of RAM on this build (the plan was checked on its own branch), the images themselves on a GPU, speed suites, KL or the 8-pack on this build.

Thanks

Thanks to @Hazelhof for the detailed report in club-3090#1966, which led to all four fixes, and to @cristianflorett for the decode report in #20.

What I'd like to hear

If you have 4 cards, please try a long-context recall on Flash-Next and send the start-up log. If you run with FLASH_KV=ram on a smaller host, the host RAM plan line helps. The issue tracker is the place.