Repository navigation
QEngine 1.3.5.2 (experimental)
Pre-releaseQEngine 1.3.5.2 (experimental)
This build adds INT8 layer weights for Qwen3.8-27B, as an option. INT4 stays the default, and INT8 runs only if you point the server at an 8-bit checkpoint. It also contains everything from 1.3.5.1. The stable version is still 1.3.5: latest, vision and the main branch stay there.
The INT8 weights are at bpx03/Qwen3.8-27B-INT8-QEngine, and INT4 users need nothing new.
docker pull ghcr.io/bpx03/qengine:1.3.5.2-vision # with the image encoder
docker pull ghcr.io/bpx03/qengine:1.3.5.2 # text onlyFrom source: git fetch --tags && git checkout v1.3.5.2, then rebuild.
Running the 27B on INT8
hf download bpx03/Qwen3.8-27B-INT8-QEngine --local-dir models/Qwen3.8-27B-INT8
server/run.sh 8080 --model models/Qwen3.8-27B-INT8 # Docker: -e QE_MODEL=/app/models/Qwen3.8-27B-INT8Or build the same kind of checkpoint yourself from the official BF16 weights. It takes no GPU and no calibration data, and it needs the Python packages in requirements-ref.txt:
hf download Qwen/Qwen3.8-27B --local-dir models/Qwen3.8-27B-BF16
python3 tools/quant_int8.py --src models/Qwen3.8-27B-BF16 --out models/Qwen3.8-27B-INT8 \
--kv-from models/Qwen3.8-27B-INT4 # copies the FP8 KV scales, which keep the full KV poolThree cards are what I'd use. On 2 cards it needed 16 slots or fewer on November 1, and one card can't hold the weights.
What it costs and what it gives
It's closer to the BF16 model and it's slower, because every decode step reads twice the weight bytes. The INT8 KL and top-1 are from this build, on 3 cards. Everything else in the table is from November 1, on older code, and I didn't rerun it, including the INT4 KL and top-1 (the INT4 greedy output on this build matches the 1.3.5 reference). The November 1 speed and pool numbers are from one day and one build.
| metric | INT8 | INT4 | INT8 / INT4 |
|---|---|---|---|
| KL vs BF16, mean (320 positions) | 6.66e-4 | 1.47e-2 | 22× closer |
| top-1 match with BF16 | 99.7 % | 95.3 % | |
| decode narrative, tok/s | 184.6 | 266.6 | 0.69 |
| decode code, tok/s | 276.7 | 387.7 | 0.71 |
| prefill 10K, tok/s | 4,816 | 6,077 | 0.79 |
| prefill 90K, tok/s | 4,098 | 4,866 | 0.84 |
| agent c=8, output tok/s | 810 | 1,215 | 0.67 |
| KV pool, 32 slots, 262K context, tokens | 566,272 | 932,864 | 0.61 |
I don't know yet whether the lower KL gives better answers in real tasks, because the 8-pack has never run on INT8. If you compare the two on your own work, I'd like to hear what you find.
Using your own INT8 checkpoint
Pass its directory with --model. The start-up check reads the file headers, and a format the engine would misread stops the start with the reason (exit code 3) instead of giving wrong output. What the engine accepts:
- one
model.safetensors, not shards - compressed-tensors pack-quantized, symmetric, group 128 along the columns
weight_packedas int32[N][K/4], bytes LSB-first, stored valuew + 128weight_scaleas bf16[N][K/128]- the same tensor names as the 27B INT4 checkpoint
The check can't prove everything: it looks at tensor types, shapes and a sample of the bytes.
- No FP8 KV scales: the server still starts, on a bf16 KV cache with half the pool and no position split of the KV heads, and it says so.
QE_KV_SCALES=<a 27B INT4 model.safetensors>borrows the scales. I checked this logic with a CPU test, but I haven't run this fallback on cards. - MTP head: the engine looks for
model_mtp.safetensorsnext to the weights. An INT8 checkpoint uses that head and skips the fine-tuned one, which was trained on INT4 outputs (QE_INT8_MTP_FT=1uses it anyway).--mtp-filealways wins. lued/Qwen3.8-27B-INT8-W8A16-MTP: I haven't tried it. It's sharded, so the start refuses it as it is. As far as I know, its tensor layout matches once merged into one file, and it has no KV scales, so the point above would apply.
What I tested on this build
A light pass, on 3× RTX 3090:
| check | result |
|---|---|
| Flash-Next greedy, c=1 | same as the 1.3.5 reference |
| 27B INT4 greedy, all items | same as the 1.3.5 reference |
| INT8 start on 3 cards, 16 slots | starts, 0 error lines |
| INT8 replies through the server | a short sum, a 5.4K-token prompt with a needle and a code question, all correct |
| the Docker images: 27B INT4, Flash-Next, 27B with an image, 27B INT8 | start and correct replies, 0 error lines |
| INT8 KL vs BF16, 320 positions | mean 6.66e-4, p50 1.40e-5, p99 7.68e-3, top-1 99.7 % |
| INT8 under pool pressure, default path, 4 sequences | batched == alone for 1200/1200 tokens each |
| INT8 resume after eviction from the archive | 6,557 of 6,560 tokens reused, logits match the live continuation (cos 1.000000) |
| INT8 follow-up turn through the server | found the needle, 7,757 of 7,783 tokens reused |
| INT8, 2 bursts of 8 parallel requests with known answers | 16/16 correct, 0 error lines |
| rejected formats (shards, group 64, zero point, signed bytes, wrong packed type) | clear refusal, CPU test on made-up headers |
What I didn't run on this build: speed, the 8-pack (never run on INT8), 2 cards, images, steering, the disk cache, long runs on INT8, and the lookup-draft path under QE_GEMV_H16=0.
The INT8 kernels also fix one INT4 detail with QE_GEMV_H16=0, the off-switch path: a sequence verifying lookup drafts alone could differ from the same sequence in a batch. The default path isn't affected. That fix was checked on November 1 on the older code; the pool test above ran the default path only.
Thanks
The idea of INT8 layer weights came from lh2892 in PR #1. This is a separate implementation with a different weight layout.
What I'd like to hear
If you run it, please send your start-up log, your card count and anything that looks wrong on the issue tracker.