Repository navigation
QEngine 1.3.5.1 (experimental)
Pre-releaseQEngine 1.3.5.1 (experimental)
This is an experimental build for people who asked for images in pipeline mode. It's lightly tested on purpose, and I'd like your feedback before it becomes the normal release. The stable version is still 1.3.5: latest, vision and the main branch stay there. The weights are the same, so there's nothing new to download.
docker pull ghcr.io/bpx03/qengine:1.3.5.1-vision # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.1 # text onlyFrom source: git fetch --tags && git checkout v1.3.5.1, then rebuild.
Image input in FLASH_PAR=pp
Flash-Next in pp now serves images. Start the server with FLASH_PAR=pp and --vision 1, and send images the same way as in tp. pptp still refuses them.
What I tested, once each, on 3× RTX 3090:
| check | result |
|---|---|
an image question in pp |
correct reply, same content as tp |
| 2 images of different sizes plus 2 text requests sent at once | all correct |
| a follow-up turn on an image conversation | correct, 1,452 of 1,478 tokens reused |
| a 27K-token image prompt with 3 text requests sent during it | all correct |
greedy text in tp with FLASH_ATTN_CHUNK=1 |
same as the reference |
| 0 error lines in the server logs | yes |
What I didn't test: 4 or more cards, 2 cards, long runs under load, an image with structured output, how much the encoder's VRAM takes from the expert cache, or speed. I didn't run KL or the 8-pack on this build. auto picks pp on 4 or more cards, so that's where your reports will help most.
When the image table has to grow, the engine first waits for prompts in flight. Through the server this wait shouldn't normally trigger, because an image prompt starts only when no other prompt is prefilling.
The copy engine and pp: what the logs got wrong
There's no copy engine mode to add to plain pp, and the old start-up log made it look otherwise.
FLASH_COMM=ce picks the path of the TP exchanges in prefill, the ones that happen between cards at every layer. Plain pp doesn't have those. Its stages hand rows from one card to the next, as plain copies over P2P or through pinned host memory (FLASH_PP_P2P). I checked this with traces on 3 cards: pp ran the same kernels and copies with FLASH_COMM=ce and FLASH_COMM=p2p, while tp with ce showed the exchange kernels and about 8.7 GB of peer copies, and pp showed none of that. The hops are device-to-device peer copies; the trace doesn't show which hardware engine runs them, though I expect it's the copy engine.
The log still said this in pp, because the probe runs for the mode pick:
[comm] prefill transfers: copy engine (FLASH_COMM=ce, forced)
[qflash] parallel mode: pp (...); probe: 3 cards, peer access yes, TP exchanges copy engine (...)
Now it says:
[comm] prefill transfers: no per-layer exchanges in the pipeline stages (rows hop over the boundaries below); TP would use copy engine (FLASH_COMM=ce, forced)
[qflash] parallel mode: pp (...); probe: 3 cards, peer access yes, TP exchanges (not used in this mode) would be copy engine (...)
[comm] decode collectives: none across cards, no calibration (pipeline mode B: its stages do not use the TP collectives)
The line that shows the real path of a hop is [flash] pipeline boundary ..., and it was right all along. If you read the old line on your machine and concluded that pp was running on the copy engine, that's why.
pptp is different, because its groups are tensor-parallel inside. FLASH_COMM=ce works there, and I ran it for the first time on record: 3 cards, FLASH_PAR=pptp FLASH_PP_MAP='0,1|2' FLASH_COMM=ce, an 8K-token prompt and a short request, both correct, and the trace shows the group really using the copy path. It also printed peer access no, TP exchanges host-staged in its parallel mode line, which was wrong too. It says per group now, and the PP x TP stage lines give the real state. I have no 4 cards, so real pair layouts are untested.
Only the log text changed: the same paths get picked as in 1.3.5.
What I'd like to hear
If you run this in pp, please tell me if images work on your machine, and send the start-up log and any errors on the issue tracker. The [qflash] parallel mode, [flash] pipeline boundary and host RAM plan lines are the ones I need most.