Skip to content

QEngine 1.3.5.3 (experimental)

Pre-release
Pre-release

Choose a tag to compare

@bpx03 bpx03 released this 07 Nov 00:49
· 13 commits to main since this release

QEngine 1.3.5.3 (experimental)

This build gives Qwen3.8-Flash-Next the disk cache the 27B already has, which people kept asking me for (#15). With --disk-cache DIR, Flash-Next conversations survive a restart. It's off by default, and without the flag nothing changes. It also contains everything from 1.3.5.2. The stable version is still 1.3.5: latest, vision and the main branch stay there.

It also changes the sampling defaults, including for people who send nothing. That part is new and unmeasured; see below.

docker pull ghcr.io/bpx03/qengine:1.3.5.3-vision   # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.3          # text only

From source: git fetch --tags && git checkout v1.3.5.3, then rebuild.

Using it

server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4 --disk-cache /path/to/a/roomy/disk

In Docker, mount a volume for that directory and give docker stop more time (-t 120), or the save at shutdown gets cut off. The files from the last idle save are still there in that case.

The flag and its defaults are the 27B's: 200 GB at most (--disk-cache-gb), and a save after 120 idle seconds (--persist-idle). Until now the flag stopped a Flash-Next start with an error.

What you'd see

A follow-up to a 95K-token conversation after a restart, on my 3× RTX 3090:

time to the reply
recomputed from nothing, first turn 13.3 s
read back from the file (2.3 GB; the read took 0.5 s of that) 0.7 s

That's one short run on a fast NVMe drive, not a benchmark. The file is also the cost: about 2.3 GB for that conversation, because it holds the KV and the recurrent states. Conversations under 2048 tokens aren't written.

A new conversation with the same system prompt reads only the part it needs: the 95K system prompt came back in 0.3 s, from 1.5 GB of a 2.3 GB file.

How it behaves

  • Writes run in the background when a conversation leaves VRAM (at most once a minute per conversation), after the idle time, and at shutdown. Requests don't wait for them.
  • Reads run in the background too. Other conversations keep generating, and after 30 s the prompt gives up on the file and prefills.
  • A file is only used by the same setup: same build, same weights (path, size and date of each file), same number of cards, same layout (tp and pp don't share files), same YaRN factor. An upgrade starts with an empty cache. Old files stay until room is needed.
  • A truncated or damaged file is refused and removed, and the prompt prefills. Every read is checked with a checksum.
  • A disk that fails costs the cache, not the server. A directory it can't write, a full disk or a write error gives a line in the log, and it keeps serving from memory. It also leaves 1 GB free on the disk.

Steered conversations are written too, unlike the 27B: a file records its vector by name and contents, plus mode, layers and scale, so it serves only the same steering. Details and the extra variables are in docs/advanced.md.

What's different from the 27B

The flag is the same, but the code isn't shared, because the two engines archive conversations differently. Flash-Next checks more: the 27B compares file sizes and its own model identity, while Flash-Next also compares the build and uses checksums. Its idle save doesn't block requests, and its reads don't stop the other conversations. The 27B path is untouched, and its greedy output matches the reference.

Sampling defaults from the model cards

When a request doesn't send a value, the server now uses what the Qwen3.8 model cards recommend for that reply's mode:

mode temperature top_p top_k min_p presence penalty
thinking 1.0 0.95 20 0 0
non-thinking 0.7 0.80 20 0 1.5

What changes if you send nothing:

  • Non-thinking replies, both models: temperature 0.7, top_p 0.80 and presence 1.5. Before, both models used top_p 0.95 and no penalty, with temperature 1.0 on Flash-Next and 0.6 on the 27B.
  • 27B thinking replies: temperature 1.0 instead of 0.6.
  • Flash-Next thinking replies: the same as before.

Values you send still win, one by one. So a client that sends only a temperature still gets presence 1.5 on a non-thinking reply. Greedy requests (temperature 0) stay without a penalty. --temperature, --top-p, --top-k, --min-p and --presence-penalty set one value for both modes.

The cards warn that a higher presence penalty may occasionally mix languages and slightly lower quality. To turn it off, start with --presence-penalty 0 or send presence_penalty: 0 in a request.

I haven't measured these defaults. No GPU run on this build used them, and I didn't start the Docker images, so I don't know the speed cost of presence 1.5, how tool calls behave under it, or what it does to quality. The code itself got only the mock server test and the parser test.

What I tested

These runs were on the disk cache branch just before the final build, on 3× RTX 3090. The final build and the images themselves weren't started.

A light pass on 3× RTX 3090, with a needle question in each conversation:

check result
flag off: Flash-Next greedy c=1, 27B greedy all items same as the references
tp: write, clean restart, follow-up turn 95,011 of 95,046 tokens from the file, needle correct
tp: new conversation with the same system prompt 94,978 of 95,007 from the file, needle correct
tp: follow-up after kill -9, from the idle save 91,109 of 91,144 from the file, needle correct
tp: follow-up with an image in the conversation 92,547 of 92,582 from the file, needle correct
tp: follow-up read while another reply is streaming both complete and correct
pp: write, restart, follow-up and new conversation 95,011 of 95,046 and 94,978 of 95,007, needles correct
steered (strata_l8_33): restart, follow-up 95,011 of 95,046 from the file, needle correct; steering off, another scale, another default vector or a changed sidecar don't load it
tp files on a pp server refused
truncated file refused and removed at start, correct reply after a prefill
one flipped byte in a file refused at the read and removed, correct reply after a prefill
another model directory, or one weight file replaced refused, correct reply after a prefill
unwritable directory warning, serves
1 GB disk, no file fits log line, serves, nothing left behind
a write failing partway (file size limit) log line, serves, nothing left behind

The format and the writer also have a CPU test: truncation at many points, flipped bytes, a full device, replaced and cancelled writes.

What I didn't run: speed with the flag on, the 8-pack and KL, 2 or 4 cards, FLASH_KV=ram, contexts above 262K, a slow or hanging disk, the budget's removal of old files, long runs, images with steering, per-request steering mode or layers in a file, and the Docker images with the flag. Nobody but me has reviewed this code yet. I only had one set of converted weights here, so the "another checkpoint" check used the same bytes under another path and a recopied file. The refusal depends only on path, size and date, not on the bytes.

Known limits: a conversation that doesn't fit the host archive isn't written (older conversations are evicted first, so this needs one larger than the archive, or an archive held by pinned prefixes and writes in progress), each save rewrites the whole file, and one directory serves one server (a second one runs without the cache and says so). A write in progress holds its memory until the file is done, so with a small archive and a slow disk a conversation leaving VRAM can be refused a place it would get moments later. It then resumes from an earlier state or prefills again, and the log says so. A conversation evicted from RAM before its save is queued may have no file or an older one. I read these two from the code and didn't test them.

A log line on the 27B

The 27B start-up line now says P2P: peer access between all N GPUs ... instead of naming a decode path before it's chosen, as reported on issue #19. Only the log text changed.

What I'd like to hear

If you run it, please send your start-up log, your disk type, your card count and anything that looks wrong on the issue tracker. This one depends on your testing: I'd especially like to hear about slow disks and long-running agents.