Skip to content

QEngine 1.3.5.9

Choose a tag to compare

@bpx03 bpx03 released this 12 Nov 08:59
· 4 commits to main since this release

QEngine 1.3.5.9

A one-fix release on top of 1.3.5.8: latest, vision and the main branch move here. It fixes slow or stuck Flash-Next starts on a host whose RAM is fragmented, which happened most with a large FLASH_KV=ram pool. The model weights and everything else are the same as in 1.3.5.8.

docker pull ghcr.io/bpx03/qengine:latest   # = 1.3.5.9, text only
docker pull ghcr.io/bpx03/qengine:vision   # = 1.3.5.9-vision, with the image encoder

From source: git fetch --tags && git checkout v1.3.5.9, then rebuild.

The fix: host-backed experts are pinned on 4 KB pages

Flash-Next keeps the experts that don't fit in VRAM in pinned host memory. Up to 1.3.5.8 it asked the kernel for 2 MB pages for that buffer. The model files it copies into pinned memory already used 4 KB pages. With the usual defrag=madvise setting, a 2 MB page fault on a host whose RAM is fragmented runs direct compaction before it falls back to 4 KB pages. A large page cache plus a big pinned FLASH_KV=ram pool is enough to cause that. On my machine a start with a 4M-token pool spent 182-197 s loading the experts instead of the usual 27-39 s. A 6M-token pool start ran past a 5-minute health check and got killed.

Now both buffers use 4 KB pages by default, which never need compaction. On this build a 4M-token start reached ready in 66 s, with 38 s of expert loading (that run did not have the fragmented RAM of the slow starts). FLASH_PIN_THP=1 brings back the 2 MB pages, but only to compare with an older build (docs/advanced.md).

What I tested

On 3× RTX 3090, from a clean build of this release: Flash-Next on 3 cards with FLASH_KV=ram FLASH_KV_RAM_TOKENS=4M, --max-ctx 1M and --vision 1 started without error lines and answered a chat correctly. No other checks ran on this build; everything else is unchanged from 1.3.5.8, whose notes list what was tested there.

The known limits of 1.3.5.8 still apply.