Mach X v0.2.13

Fifth tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.13 makes the Qwen models read prompts 1.4–3.7× faster, makes speculative decoding on Qwen3.8-27B usable in normal serving, and tests GPU memory at load for a page fault that corrupted one model on one of the test cards.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Every figure was measured on October 4, 2026 (October 2 where noted), on community fine-tunes of the models, old path against new path on the same day. "Bench" is ie-prompt-bench (prompt cache off, greedy) with prompts of 2,404 and 10,693 tokens.

Highlights

Prefill on the Qwen models

A per-card profile of the 27B's prefill showed attention at 47 % of the time and the per-chunk expansion of the 8-bit weights to fp16 at 25 %; the matrix multiply itself was 17 %. Three changes, none of them lower precision: a chunk's attention as two oneDNN matmuls on the XMX engines with a softmax kernel between them; Q8_0 weights read in place by the prefill matmul; the DeltaNet scan with its state in registers.

model	measure	before	v0.2.13
Qwen3.8-27B Q8_0	bench prefill, 2.4K / 10.7K tokens	1,037 / 820 tok/s	2,962 / 3,001 tok/s
	ie serve, a 20,248-token turn	33.0 s	7.8 s
Qwen3.8-27B Q6_K	bench prefill	987 / 794 tok/s	1,378 / 1,549 tok/s
35B-A3B class Q8_0	bench prefill	1,816 / 1,256 tok/s	2,896 / 2,808 tok/s
	ie serve --parallel 4, a 20,210-token turn	16.4 s	5.8 s
	the 15-agent replay (tools/swarm_replay.py)	116.1 s	77.1–77.2 s
	replay: a worker's decode, median (5th percentile)	9.89 (7.13) tok/s	14.8–15.2 (12.1) tok/s
	replay: a worker's first token, median; the wave's	2.79 s; 11.2 s	1.6–1.7 s; 5.8 s
Qwen3.8-Flash (Flash-Next)	bench prefill, text never seen before / a repeated prompt	241 / 487 tok/s	470 / 688 tok/s
	ie serve, a 20,248-token turn	39.4 s	28.1 s

Flash-Next also reads its 26.8 GiB per-layer token table into RAM in the background at load (it was read from the file mapping one page fault at a time), and its sparse attention runs as the same matmuls with the selected keys as a mask, up to 65,536 tokens of context.

Perplexity moved by −0.14 % to +0.21 % across the three models and is identical run to run. Decode speed without speculation is unchanged.

Speculative decoding on Qwen3.8-27B in normal serving

--spec used to run only at temperature 0, on prompts up to 4,096 tokens, and re-read the whole prompt. It is now a step of the normal decode loop: the engine samples every token and the step supplies already-verified logits whenever the sampled token equals the draft. It works with any sampler, the prompt cache and any prompt length; four ie serve replies at temperature 0.7 with a fixed seed were byte-equal to a plain server's.

Qwen3.8-27B Q8_0, --spec --spec-k 3	plain	speculative
ie serve, temperature 0.7, short prompt	17.1 tok/s	24.3
ie serve, temperature 0.7, 10,891-token prompt	15.9	17.8
bench, greedy, five short prompts (median)	17.1	29.7

A memory-page fault is caught at load

On one of the two test cards, two 2 MiB pages of a single GPU allocation can be backed by one physical page. On Flash-Next this put one expert's weights in another expert's cache slot and replies ended after a few tokens. Every device allocation of 64 MiB or more is now tested before use (about 0.5 s for 49 GiB across both cards); a faulty allocation is kept aside and another is made. Flash-Next's expert caches get their own test with slot quarantine.

Also since v0.2.8

• Image input on the Qwen3.8-27B and 35B-A3B two-card splits at --parallel 1 with --mmproj.
• /props prompt_cache_slots reports the two Qwen splits' real prompt-cache capacity.
• Flash-Next's expert-cache budget counts the context's KV and workspaces: at --ctx 250000 decode went from 0.2 to 23 tok/s (October 2).

Behaviour changes

• Greedy replies of the three Qwen models can differ from v0.2.8's at near-ties (the prefill paths round differently). Each new path has a switch that restores the previous one: IE_QWEN35_ATTN_GEMM, IE_QWEN35_S8_PREFILL, IE_QWEN35_DN_SCAN, IE_Q35MOE_ATTN_GEMM, IE_Q35MOE_S8_PREFILL, IE_Q35MOE_DN_SCAN, IE_Q4E_ATTN_GEMM, IE_Q4E_S8_PREFILL, IE_Q4E_DN_SCAN (=0).
• With --spec, the Qwen3.8-27B split speculates on sampled requests and long prompts too (IE_QWEN35_SPEC_STEP=0 = the previous greedy-only path).
• Flash-Next's load reads 26.8 GiB into the page cache in the background (IE_Q4E_PLE_PREFETCH=0 off).
• Every load tests its large device allocations (IE_NO_ALLOC_ALIAS_TEST=1 skips).

Known limitations

• Flash-Next's prefill is now 82 % expert layers; past 65,536 tokens of context its previous attention kernel runs.
• Plain decode on the 27B is unchanged (17 tok/s on Q8_0); --spec is slower than plain on the Q6_K file.
• The speculative gain depends on the draft head's hit rate (1.35–1.7 tokens per forward at temperature 0.7).
• ie serve --help still describes --spec as greedy-only.
• Not run for this release: image requests on the new prefill paths, --int8-kv, the 35B's Q6_K / Q5_K files.
• The allocation test cannot see a page shared between two different allocations.
• In the release smoke one 256-token DeepSeek-V4.1 reply differed between two runs of the previous build (the new build reproduced its own); cause not investigated.