Repository navigation
QEngine 1.3.5.4 (experimental)
Pre-releaseQEngine 1.3.5.4 (experimental)
This build closes most of the open issues: logprobs (#10), the 4-card mode pick (#12), a possible crash during the load (#11), a clearer decode-path log line (#19), and NUMA placement on boards that don't report a card's node (the idea from #16). It also contains everything from 1.3.5.3. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.
docker pull ghcr.io/bpx03/qengine:1.3.5.4-vision # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.4 # text onlyFrom source: git fetch --tags && git checkout v1.3.5.4, then rebuild.
logprobs and top_logprobs (#10)
Both models accept logprobs: true and top_logprobs (0-20) on /v1/chat/completions, also when streaming. The numbers come from the target head's raw logits, before temperature, penalties and the sampling filters, as in the OpenAI API.
- Only the reply's content tokens get entries; the reasoning part doesn't.
- A request with
logprobsruns about 3× slower than without, because it steps one token at a time with no drafts. Requests running next to it slow down too, to a bit under half their speed alone in my runs. Requests without the field are unchanged. - In
tp(Flash-Next) and on the 27B, the values matched a no-draft reference exactly on my 3 cards. Inppthe chosen token always is the top entry, but the low-probability entries of the top 20 carrypp's own rounding, the same differenceppandtpalready have without logprobs. Usetpif you compare distributions. - No prompt logprobs. One request per position with
max_tokens: 1gives you the same.
4 cards: auto picks tp when it can (#12)
On exactly 4 equal cards, FLASH_PAR=auto now picks tp when peer access between all of them passes the data check at start and tp fits the free VRAM. Otherwise it picks pp, as before. With 5 or more cards it stays pp. If tp can't start (out of device or host memory during the load, a CUDA error, or a stall in the check), the server restarts itself in pp and says so in the log. A mode you set yourself is never changed.
I don't have 4 cards. The rule passed its unit tests, and 2- and 3-card starts give the same output and speed as before, but no 4-card machine has run it. Reports from 4-card machines are the most useful thing you can send me for this one.
Smaller changes
- Load (#11): a setting was written while the load threads were already reading settings, which can crash the load. It's written before any thread starts now, and not at all when CUDA is already up (as with
--vision 1). - Log (#19): the
[qflash] parallel modeline names the prefill path and, when it's known, decode's path. Withautoit says the decode path is timed at the end of the start-up, and a closing line gives the result. - NUMA: on a machine with several NUMA nodes where the system doesn't report a card's node (
GPU NUMA ID: N/Ainnvidia-smi topo -m, seen on Threadripper Pro boards), pinned host memory is now spread over all nodes instead of wherever the first touch puts it.QE_NUMA=0,1,2,3sets the nodes by hand. Machines with one node, or with the nodes reported, are unchanged. My machine has one node, so this path hasn't run here. - Logs:
FLASH_SLOT_LOGis documented, the idle reclaim line shows the longer wait of an active main thread, andflash_runprintspeer accessinstead of a path it hadn't picked yet.
What I tested
On 3× RTX 3090, from a clean build of this release:
| check | result |
|---|---|
start with --vision 1 |
ready, no error lines |
| an image question | correct description |
logprobs with top_logprobs: 3 |
entries for every token |
| a non-thinking request on the default sampling | normal reply |
| greedy output, same request twice | identical |
2 and 3 cards, tp: greedy output and decode speed against 1.3.5.3 |
same output, speed within run-to-run noise |
What I didn't test: 4 cards, the NUMA fallback, the images themselves on a GPU, speed beyond the pairs above, KL or the 8-pack on this build.
What I'd like to hear
If you have 4 cards, please send the start-up log: the [qflash] parallel mode line and anything about a restart in pp. The issue tracker is the place.