Mach X v0.2.27

Mach X v0.2.27

Everything since v0.2.26. Measured
October 20–21, 2026, on two Arc Pro B70 cards.

Qwen3.8-27B: MTP drafts on the request lanes (--spec --parallel N)

Until now --spec turned the 27B's request lanes off: the model's MTP draft head ran only at --parallel 1. Now the
lanes draft with it. While at most one lane a card decodes, a lane's next step checks the head's draft in the same
step as its next token (--spec-k rows); with more lanes decoding they share the cards in row groups, as without
--spec.

greedy, 256 tokens a reply, two cards	plain	--spec
Q4_K_M, one request	27.2 tok/s	36.0–40.2
Q4_K_M, two at once, together	51.8	63.1
Q4_K_M, four at once, together	91.1	90.7 (row groups)
Q8_0, one request	17.4	26.7–28.0
Q8_0, four at once, together	64.8	63.9 (row groups)

• Checks: every reply above has the same text and token count as the plain server's: 39 replies, pictures
  included (the streamed chunks differ: a round sends two tokens at once). --spec at --parallel 1 gives the same
  replies at the same speed as before the change (the code it shares with the lanes moved). 68–71 % of the drafts were
  kept. No kernel GPU fault in any run.
• --spec is near-lossless, not lossless: on a rare near-tie the checking step can pick a different token than plain
  decoding (see v0.2.25). Run without --spec for bit-exact output.
• About 0.14 GiB a lane on each card (checked against free memory at load; if it does not fit, the lanes run without
  drafts and the load says why). Switches: IE_QWEN35_LANES_MTP=0 turns the rounds off, IE_QWEN35_LANES_MTP_MAX
  sets how many decoding lanes may run them (default: the card count).

Qwen3.8-27B: Q4_K_M on the fast path

The 27B's Q4_K files ran on an older kernel; they now take the same path as Q8_0, Q6_K and Q5_K.

Qwen3.8-27B Q4_K_M	v0.2.26	v0.2.27
decode, two cards	22.8 tok/s	27.5
decode, one card	19.1	27.8
reading a 3,382-token prompt, two cards	1,232 tok/s	1,393
reading a 3,382-token prompt, one card	390	1,045
decode with --spec (one request)		37.1–39.6

• Q6_K reads that prompt at 1,330 tok/s (was 965) with the same replies; Q8_0 and the 35B-A3B class give the same
  replies at the same speed.
• Checks: perplexity on 4,096 wikitext tokens 5.8107 (old kernel 5.8133); one of three test replies differs from
  v0.2.26 after a near-tie (a different summation order); one card and two cards give the same replies.
• Measured on a Q4_K_M made from the Q8_0 file (llama-quantize --allow-requantize). Pictures on it are answered right
  in our probes, including after 16.7K tokens; a 640 px synthetic picture got "Sorry, I need..." on 3 of 9 (the Q8_0
  file 9 of 9). That is a near-tie in the twice-quantized file: the GPU encoder matches the CPU one, and the old kernel
  fails a different picture.

Pictures on the request lanes: Qwen3.8-27B and the 35B-A3B class

ie serve --parallel N --mmproj <projector> now takes images on every lane (before: --parallel 1 only, and /props
said vision was not ready). Each lane keeps its own picture rows and M-RoPE positions. An image request skips the
prompt cache and the reuse paths (sticky lane checkpoints, restart points).

• Checks (four lanes, Q4_K_M and Q8_0 27B, 35B-A3B Q6_K): every answer right; four at once give the same replies as
  each alone; a lane's picture reply is byte-identical to --parallel 1's; text requests unchanged.
• The vision encoder's scratch leaked on every growth (270 MiB on card 1 after four picture sizes), a failed allocation
  passed a null pointer to a kernel, and its copies were pageable. All three fixed (pinned copies, as in issue #2).

A VRAM leak on every request (v0.2.26 and earlier)

The oneDNN products used for prompt reading were cached per server thread and per row count, and each kept its own
12–20 MiB scratch buffer on the card for the life of the server. On the 27B that was about 100 MiB a card per request
(Q4_K_M, 12 requests: 9,648 → 10,939 MiB on card 0, still climbing). Each card queue now shares one scratch buffer:
after the first request the same 12 requests stay at 9,688–9,689 MiB, with the same replies and the same kernels
(oneDNN picks the same implementation for all 720 shapes checked). Every model whose prompt reading goes through
oneDNN gets the fix. DeepSeek-V4.1's disk prompt cache is keyed by its numerics files, so its first turn on a cached
prompt after the upgrade is read cold once.

DeepSeek-V4.1-Flash on more than two cards, and tensor parallel

What the v0.2.27-rc1 test build
carried:

• More than two cards: v0.2.26 had to be told --cards 0,1 (past two cards, prompts over 2,048 tokens failed and
  attention past 16K tokens was wrong). The pipeline now carries everything a later card needs across a card boundary;
  integrated graphics are skipped and --gpus N takes the first N discrete Arc cards; the CPU expert leg picks its
  cores from the machine, and each card's link is measured at load. Checked by emulating four cards as four pipeline
  stages on two B70s: 15 of 15 traces byte-identical to two cards (prompts up to 19,783 tokens; CPU expert leg off).
  Two-card replies are identical to v0.2.26's.
• Tensor parallel, opt-in (IE_DS41_TP=1): every card runs all 40 layers on its share of each layer. On two B70s
  it decodes 17.2 tok/s on a short prompt (layer split 14.6) and 12.3 at 23.5K tokens of context (10.5), and reads a
  23,581-token prompt at 301 tok/s (layer split 513). Perplexity on 4,096 wikitext tokens 1.51707 (layer split
  1.51685). Switches: IE_DS41_TP_1BAR=0, IE_DS41_TP_AFFINITY=0, IE_DS41_DISK_CACHE, IE_DS41_STEP_DETAIL=1.

Also

• dequant_q4_K_to_Bt / dequant_q5_K_to_Bt skipped the last columns of a matrix whose width is not a multiple of 64
  (no shipped model has one); fixed, with a test that fails on the old kernels.

What is still slow, and not checked

• Real four-card DeepSeek-V4.1 speed is not measured (emulation shares one card's memory and link between two
  stages). Tensor parallel reads long prompts about 40 % slower than the layer split on two cards.
• The 27B's MTP rounds help one or two decoding lanes on two cards; at four and more, row groups are faster and the
  rounds stay off. The 35B-A3B class and Flash-Next keep their lookup drafts.
• The other models (Flash-Next, the DeepSeeks, MiMo, GLM, Gemma and more) still copy host memory directly, mostly
  whole tensors at load. GLM-5.3-Flash prefill is bound by the host expert bank over PCIe; ie serve runs GLM one
  request at a time.
• DeepSeek-V4.1 with its CPU expert leg on (the default) is not repeatable token for token at long context; with
  IE_DS41_CPU_MISS=0 it is.