Repository navigation
QEngine 1.3.5.3 (experimental)
Pre-releaseQEngine 1.3.5.3 (experimental)
This build gives Qwen3.8-Flash-Next the disk cache the 27B already has, which people kept asking me for (#15). With --disk-cache DIR, Flash-Next conversations survive a restart. It's off by default, and without the flag nothing changes. It also contains everything from 1.3.5.2. The stable version is still 1.3.5: latest, vision and the main branch stay there.
It also changes the sampling defaults, including for people who send nothing. That part is new and unmeasured; see below.
docker pull ghcr.io/bpx03/qengine:1.3.5.3-vision # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.3 # text onlyFrom source: git fetch --tags && git checkout v1.3.5.3, then rebuild.
Using it
server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4 --disk-cache /path/to/a/roomy/diskIn Docker, mount a volume for that directory and give docker stop more time (-t 120), or the save at shutdown gets cut off. The files from the last idle save are still there in that case.
The flag and its defaults are the 27B's: 200 GB at most (--disk-cache-gb), and a save after 120 idle seconds (--persist-idle). Until now the flag stopped a Flash-Next start with an error.
What you'd see
A follow-up to a 95K-token conversation after a restart, on my 3× RTX 3090:
| time to the reply | |
|---|---|
| recomputed from nothing, first turn | 13.3 s |
| read back from the file (2.3 GB; the read took 0.5 s of that) | 0.7 s |
That's one short run on a fast NVMe drive, not a benchmark. The file is also the cost: about 2.3 GB for that conversation, because it holds the KV and the recurrent states. Conversations under 2048 tokens aren't written.
A new conversation with the same system prompt reads only the part it needs: the 95K system prompt came back in 0.3 s, from 1.5 GB of a 2.3 GB file.
How it behaves
- Writes run in the background when a conversation leaves VRAM (at most once a minute per conversation), after the idle time, and at shutdown. Requests don't wait for them.
- Reads run in the background too. Other conversations keep generating, and after 30 s the prompt gives up on the file and prefills.
- A file is only used by the same setup: same build, same weights (path, size and date of each file), same number of cards, same layout (
tpandppdon't share files), same YaRN factor. An upgrade starts with an empty cache. Old files stay until room is needed. - A truncated or damaged file is refused and removed, and the prompt prefills. Every read is checked with a checksum.
- A disk that fails costs the cache, not the server. A directory it can't write, a full disk or a write error gives a line in the log, and it keeps serving from memory. It also leaves 1 GB free on the disk.
Steered conversations are written too, unlike the 27B: a file records its vector by name and contents, plus mode, layers and scale, so it serves only the same steering. Details and the extra variables are in docs/advanced.md.
What's different from the 27B
The flag is the same, but the code isn't shared, because the two engines archive conversations differently. Flash-Next checks more: the 27B compares file sizes and its own model identity, while Flash-Next also compares the build and uses checksums. Its idle save doesn't block requests, and its reads don't stop the other conversations. The 27B path is untouched, and its greedy output matches the reference.
Sampling defaults from the model cards
When a request doesn't send a value, the server now uses what the Qwen3.8 model cards recommend for that reply's mode:
| mode | temperature | top_p | top_k | min_p | presence penalty |
|---|---|---|---|---|---|
| thinking | 1.0 | 0.95 | 20 | 0 | 0 |
| non-thinking | 0.7 | 0.80 | 20 | 0 | 1.5 |
What changes if you send nothing:
- Non-thinking replies, both models: temperature 0.7, top_p 0.80 and presence 1.5. Before, both models used top_p 0.95 and no penalty, with temperature 1.0 on Flash-Next and 0.6 on the 27B.
- 27B thinking replies: temperature 1.0 instead of 0.6.
- Flash-Next thinking replies: the same as before.
Values you send still win, one by one. So a client that sends only a temperature still gets presence 1.5 on a non-thinking reply. Greedy requests (temperature 0) stay without a penalty. --temperature, --top-p, --top-k, --min-p and --presence-penalty set one value for both modes.
The cards warn that a higher presence penalty may occasionally mix languages and slightly lower quality. To turn it off, start with --presence-penalty 0 or send presence_penalty: 0 in a request.
I haven't measured these defaults. No GPU run on this build used them, and I didn't start the Docker images, so I don't know the speed cost of presence 1.5, how tool calls behave under it, or what it does to quality. The code itself got only the mock server test and the parser test.
What I tested
These runs were on the disk cache branch just before the final build, on 3× RTX 3090. The final build and the images themselves weren't started.
A light pass on 3× RTX 3090, with a needle question in each conversation:
| check | result |
|---|---|
| flag off: Flash-Next greedy c=1, 27B greedy all items | same as the references |
tp: write, clean restart, follow-up turn |
95,011 of 95,046 tokens from the file, needle correct |
tp: new conversation with the same system prompt |
94,978 of 95,007 from the file, needle correct |
tp: follow-up after kill -9, from the idle save |
91,109 of 91,144 from the file, needle correct |
tp: follow-up with an image in the conversation |
92,547 of 92,582 from the file, needle correct |
tp: follow-up read while another reply is streaming |
both complete and correct |
pp: write, restart, follow-up and new conversation |
95,011 of 95,046 and 94,978 of 95,007, needles correct |
steered (strata_l8_33): restart, follow-up |
95,011 of 95,046 from the file, needle correct; steering off, another scale, another default vector or a changed sidecar don't load it |
tp files on a pp server |
refused |
| truncated file | refused and removed at start, correct reply after a prefill |
| one flipped byte in a file | refused at the read and removed, correct reply after a prefill |
| another model directory, or one weight file replaced | refused, correct reply after a prefill |
| unwritable directory | warning, serves |
| 1 GB disk, no file fits | log line, serves, nothing left behind |
| a write failing partway (file size limit) | log line, serves, nothing left behind |
The format and the writer also have a CPU test: truncation at many points, flipped bytes, a full device, replaced and cancelled writes.
What I didn't run: speed with the flag on, the 8-pack and KL, 2 or 4 cards, FLASH_KV=ram, contexts above 262K, a slow or hanging disk, the budget's removal of old files, long runs, images with steering, per-request steering mode or layers in a file, and the Docker images with the flag. Nobody but me has reviewed this code yet. I only had one set of converted weights here, so the "another checkpoint" check used the same bytes under another path and a recopied file. The refusal depends only on path, size and date, not on the bytes.
Known limits: a conversation that doesn't fit the host archive isn't written (older conversations are evicted first, so this needs one larger than the archive, or an archive held by pinned prefixes and writes in progress), each save rewrites the whole file, and one directory serves one server (a second one runs without the cache and says so). A write in progress holds its memory until the file is done, so with a small archive and a slow disk a conversation leaving VRAM can be refused a place it would get moments later. It then resumes from an earlier state or prefills again, and the log says so. A conversation evicted from RAM before its save is queued may have no file or an older one. I read these two from the code and didn't test them.
A log line on the 27B
The 27B start-up line now says P2P: peer access between all N GPUs ... instead of naming a decode path before it's chosen, as reported on issue #19. Only the log text changed.
What I'd like to hear
If you run it, please send your start-up log, your disk type, your card count and anything that looks wrong on the issue tracker. This one depends on your testing: I'd especially like to hear about slow disks and long-running agents.