Mach X v0.2.8

Fourth tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.8 makes v0.2.6's agent swarm faster: a lead and up to 15 sub-agents that read files and write at the same time, on the Qwen3.6-35B-A3B class (Q8_0) split over two Arc Pro B70 cards. The 15-agent replay takes 116.3–116.5 s where v0.2.6 took 131.3–132.4 s in the same day's runs (−12 %), an agent's decode goes from a median 8.65–8.80 to 9.83–9.86 tok/s, and its first token comes in 2.58–2.83 s instead of 3.79–4.79 s. Three changes do it: the engine's serial turns run beside the lane pipe instead of pausing it, the decoding lanes regroup, and the tiled prefill attention starts at 512 positions instead of 6,144. There is no new kernel and no change to the request API; the changes are for the 35B-A3B class.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Every figure was measured on October 2, 2026, on the driver stack v0.2.6 was measured on, on a community fine-tune of the model. Each change was measured by an independent run of its build against the build before it, two runs per build where a line gives two numbers, and each build's replies were compared byte for byte with the build before it.

Highlights

The agent replay: 12 % faster, a worker's first token in 2.6–2.8 s

The replay is v0.2.6's (tools/swarm_replay.py): 15 worker conversations for 3 turns each (prompts growing from 1.4K to about 8K tokens), then a wave of 15 new conversations that share one system prompt of about 12K tokens, with one follow-up each: 75 requests against ie serve --gpus 2 --ctx 262144 --parallel 16 --max-queue 16 --thinking on. Every run finished 75 of 75 requests with 0 errors. Each row compares two runs of a build with two runs of the build before it: alternating in one session for the first row, at most three hours apart for the other two (the same prompts). v0.2.6's release notes give 139.7 s for this replay, measured on October 1; the same binary measured 131.3 and 132.4 s on October 2, so compare within a row.

change	whole replay	workers' decode per request, median (5th percentile)	workers' first token, median
serial turns beside the lane pipe	132.4 / 131.3 → 127.7 / 127.7 s	8.65 / 8.80 (6.88 / 6.50) → 8.59 / 8.61 (7.66 / 7.65) tok/s	3.79 / 4.79 → 4.16 / 4.00 s
decoding lanes regrouped; a lone prompt's remainder re-cut, a prompt's tail first	127.8 / 128.9 → 122.2 / 122.2 s	8.76 / 8.61 (7.68 / 7.24) → 8.85 / 8.93 (7.87 / 7.84)	4.15 / 4.15 → 4.06 / 4.01 s
tiled prefill attention from 512 positions	122.2 / 122.2 → 116.3 / 116.5 s	8.85 / 8.93 (7.87 / 7.84) → 9.83 / 9.86 (7.12 / 7.13)	4.06 / 4.01 → 2.83 / 2.58 s
v0.2.6 → v0.2.8	131.3–132.4 → 116.3–116.5 s (−12 %); the workers phase 82.0–82.8 → 70.3 s	8.65–8.80 (6.50–6.88) → 9.83–9.86 (7.12–7.13)	3.79–4.79 → 2.58–2.83 s

Serial turns beside the lane pipe

Between a lane's prefill pieces and its decode steps the server takes short exclusive turns for that lane: restoring a cached prefix, marking a shared prefix, snapshotting the conversation, ending the prompt. In v0.2.6 every turn paused the two-card pipe and waited for every step in flight to land, because the prompt-end turn ran the prompt's last 3–6 rows (the generation prompt) as a full two-card forward on the cards' shared workspace. In one trace of the replay on v0.2.6 that was 84 pauses and 16.5 s of waiting, 89 % of it the prompt-end turn's.

Now those rows go through the pipe as the prompt's last piece, each turn's copies address their lane by index, and the turns run while the other lanes keep stepping. Only a lost lane's reset, and the snapshot of a reply that ended at its length limit (taken only on templates without thinking), still pause the pipe.

the replay, two runs of each build	v0.2.6	serial turns beside the pipe
pauses; the pipe's paused time	84–86; 13.3–13.4 s	0; 0.7 s
card 0 idle in the workers phase	19.9–20.5 s (24.5–25.0 %)	10.7–10.8 s (13.7–13.8 %)
card 0's longest idle gap before a decode step	2,068–2,070 ms	291–292 ms
the 15 workers' first tokens at the start (median)	5.17–5.19 s	4.22–4.23 s
workers' decode per request, 5th percentile	6.50–6.88 tok/s	7.65–7.66
workers' decode per request, median	8.65–8.80 tok/s	8.59–8.61

• The median decode did not move, and the trace shows why: without the pauses' release bursts the decoding lanes formed smaller groups (3.20 rows per card step against 4.53). The decode regroup, next, fixes that.
• With an 80K-token lead and six workers nothing regressed: the workers' first token 12.2 s against 12.6, the lead's 169.0 s against 169.5, the same replies.
• The replies of all five lane models (35B-A3B class, Qwen3.8-27B, Qwen3.8-Flash, MiMo-V2.6-Flash, DeepSeek-V4.1-Flash) equalled v0.2.6's byte for byte: 156 checks, 0 differences. In the replay the 45 worker replies equalled v0.2.6's; 28 of the 30 wave replies differ, because the wave's first prompt took the prefill path of a prompt that starts alone (its first 8,192 rows in one piece) in every run of the new builds. v0.2.6 takes that path too, depending on arrival timing (1 of its 4 runs), and the pieces decide the bits.
• IE_Q35MOE_TURN_DRAIN=1 restores v0.2.6's protocol; on the build with only this change the replay's 75 replies then equalled v0.2.6's.

Decoding lanes regroup

The decoding lanes step in groups. A group is one card step that reads the weights once, so a row costs 5.7–5.8 ms in a 1-row step and 2.2 ms in an 8-row step (17.5–17.7 ms). The pipe merges only the lanes it finds queued when it forms a group, and a landed group's lanes resubmit together, so three or more groups rotating over the two cards never met: one lands per step, one forms per step. v0.2.6's paused turns had been merging them by accident, releasing every parked lane at once.

Now, before forming a group that would leave three or more rotating, card 0 waits up to 20 ms for the decode group landing on card 1 and forms from everything queued. One or two decoders never wait, a prefill piece at the front of the queue never waits, and a merge is paid once.

workers phase of the replay, card 0	decode steps	rows per step	1-row steps	decode card time
v0.2.6	1,595–1,599	4.53–4.54	164–184	19.6 s
serial turns beside the pipe	2,184–2,196	3.30–3.31	792–819	21.9–22.4 s
with the regroup	1,586–1,587	4.56	175–177	19.6–19.7 s

The waits held card 0 for 636–644 ms over the replay (37–41 waits, 33–37 of them brought lanes, 4–5 timed out). The replay's replies are 75 of 75 identical to the build before, and 124 identity checks on the 35B-A3B class and Qwen3.8-27B found 0 differences: a row's bytes do not depend on its group. IE_Q35MOE_REGROUP=0 turns it off; IE_Q35MOE_GROUP_WAIT_US sets the wait.

• A lone prompt's remainder is re-cut, and a prompt's tail goes first. The first change had moved the first token of the wave's first prompt from 8.57 s (v0.2.6 when it interleaved with its followers from the start) or 10.22 s (v0.2.6 when it prefilled alone first) to 12.4 s: released into the running pipe, its remaining 3,275 rows ran as one piece, and its 7-row tail queued behind the 14 followers' first pieces. Now a prompt that started alone has its remaining pieces re-cut into 512-row pieces as soon as another request waits for its turn (that prompt's prefill 11.1 → 8.9 s; v0.2.6 alone took 8.8), and a prompt's tail piece goes to the front of the queue (one piece of at most 512 rows per prompt, so nothing starves). The first prompt's first token 12.38–12.43 → 11.17–11.31 s; the 14 followers' 12.41–12.42 → 11.29–11.31 s. The first prompt stays behind v0.2.6: see Known limitations.

Tiled prefill attention from 512 positions

v0.2.6 took the tiled attention kernel for a prefill piece whose last position reached 6,144 and the previous kernel below, a threshold copied from the 27B and never measured on this class. Measured on the replay's 512-row pieces (the size a prompt is read in beside running lanes; card 0 medians):

start of the piece	0–2K	2–4K	4–6K	6–8K	8–16K	fit below 6,144
threshold 6,144 (v0.2.6)	208.0 ms	247.1	295.8	276.6	366.1	155 ms + 30 per 1,000 positions
threshold 512 (v0.2.8)	190.1 ms (−9 %)	215.0 (−13 %)	248.8 (−16 %)	276.0	366.5	159 + 18, the same line as from 6,144 up

The tile has no overhead at depth 0 (a 512-row piece over 512 keys: 156.5 ms against 158.0), so the old threshold made a piece at 4–6K slower than one at 6–8K.

• In the replay: the workers' first token 4.06 / 4.01 → 2.83 / 2.58 s (median), their prefill 1,085 → 1,171 tok/s together, their decode median 8.85 / 8.93 → 9.83 / 9.86 tok/s, the workers phase 75.9 → 70.3 s. The wave, all of it deeper than 8K, is unchanged.
• One request at a time: a 2,038-token cold prompt 1.41 → 1.28 s (server-side 1,395 → 1,271 ms; measured with the threshold set by environment on the v0.2.6 build). 8,065-, 16,369- and 32,161-token prompts are unchanged within 0.4 %: a prompt alone is read in 8,192-row pieces, which were on the tile already. Needle prompts of 2,043 and 3,580 tokens got their first token in 1.36–1.42 s instead of 1.47–1.50 and in 2.30–2.31 s instead of 2.65–2.67, found 6 of 6 needles, and gave byte-identical replies.
• Numerics: perplexity on the 12,288-token anchor is 2.3529 at both thresholds. On prefill chunks of 4,096 / 2,048 / 1,024 / 512 rows, which change kernel, it moves −0.064 / +0.183 / −0.497 / +0.028 % (wikitext-2, 511 scored tokens each; the sign changes between windows).
• Bits: a prompt whose prefill pieces end between 512 and 6,143 positions computes other bits than on v0.2.6: 39 of the replay's 45 worker replies differ, 6 are identical, and the 30 wave replies, deeper than 8K, are identical. Prompts shorter than 512 tokens and pieces ending at 6,144 or deeper are unchanged.
• One dependence on timing is gone: on v0.2.6 a prompt crossing 6,144 computed its rows below 6,144 with the other kernel if it had been re-cut into 512-row pieces beside a running lane (28 of 30 wave replies differed between two v0.2.6 runs at the same threshold). From 512, every piece that reaches 512 takes the tile whatever the cut.
• IE_Q35MOE_FA2_TILE_MINCTX=6144 gives v0.2.6's kernel choice: with it the release build reproduced v0.2.6's recorded replies on the check's short prompts, tool call and batch of four, and v0.2.6's perplexity to six decimals on the chunks that change kernel. The 27B's own threshold is unchanged.

Step-level tables, the expectations that did not hold, what was not re-measured, and how to reproduce are in PERFORMANCE.md.

Serving

• The load log says which protocol a run used: [qwen35moe] serial turns run BESIDE the lane pipe ... (or DRAIN the lane pipe (IE_Q35MOE_TURN_DRAIN=1)), [qwen35moe] decode regroup ON: stage 0 waits up to 20000 us ... (or OFF), and [qwen35moe_split] prefill attention: tiled (FA2 wide tile) from 512 positions, naive below ....
• /health gains turns_nodrain (serial turns that ran beside the pipe), turns_boundary_first, and on the 35B-A3B class pipe: groups (the decode groups formed so far, by size), regroup_waits, regroup_merges, regroup_timeouts, regroup_wait_ms. The shutdown line's groups by size gains the regroup counts.
• A stop is as fast as before.

stop request to process exit	v0.2.6	this release
16 lanes at --ctx 262144, the 80K-token lead and six workers, a 512-row piece at 42K positions in flight	0.45–0.53 s	0.45 s on the release build; 0.53 s on the build before the attention change
4 lanes, four ~18.8K-token prompts mid-prefill	0.43 s	0.38 s on the first build of the series; not re-measured on the release build

Every stop left 0 GPU fault lines, and a server started right after it loaded and gave the recorded reply. All 40 GPU jobs that checked this release left 0 GPU fault lines.

The models it is tuned for

model	prefill	decode
Qwen3.6-35B-A3B class (Q8_0)	one request, cold: 2K tokens in 1.28 s (October 2, with the new threshold set by environment on the v0.2.6 build; 1.41 s before), 8K in 5.96 s, 32K in 48.4 s (October 1)	unchanged for one request and for 16 that only write (85.7 tok/s; 414.1 tok/s together, 25.9 each; October 1); in the 15-agent replay a median 9.83–9.86 tok/s per agent (October 2; v0.2.6: 8.65–8.80)
Qwen3.8-27B, MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, DeepSeek-V4-Flash, GLM-5.3-Flash, Qwen3.8-Flash	unchanged from v0.2.6	unchanged from v0.2.6

Conditions and methods are in the README's Benchmarks section and in "New in v0.2.8". These come from one machine; your hardware will differ.

Upgrading from v0.2.6

• Nothing changes for a client. No flag or request field changed and the lane count is picked as before; the three changes apply to the Qwen3.6-35B-A3B class on its two-card split.
• Output changes on the 35B-A3B class for prompts with prefill pieces ending between 512 and 6,143 positions: the tiled prefill attention now starts at 512 positions (IE_Q35MOE_FA2_TILE_MINCTX, was 6144), and it is not bit-identical to the kernel it replaces. Reply checksums recorded on v0.2.6 for such prompts have to be recorded again. Prompts shorter than 512 tokens, and a prompt alone whose pieces all reach 6,144, give the same bytes as before. IE_Q35MOE_FA2_TILE_MINCTX=6144 gives v0.2.6's kernel choice; IE_Q35MOE_NO_FA2_TILE (set to any value) still gives the previous kernel at every depth.
• The serial turns no longer pause the lane pipe on the 35B-A3B class. /health drains, drain_ms, drain_max_ms and paused_ms now count only the turns that paused the pipe and read 0 or close to it (0 pauses and 0.7 s in the replay, where v0.2.6 read 84–86 and 13.3–13.4 s); the turns that ran beside the pipe are counted in turns_nodrain. IE_Q35MOE_TURN_DRAIN=1 returns to v0.2.6's protocol.
• The decode regroup is on by default on the 35B-A3B class (IE_Q35MOE_REGROUP=0 turns it off). It does not change any reply.
• A burst of conversations with one shared prompt: the first one's first token comes later than on v0.2.6 (11.3 s against 8.6 or 10.2 s for 15 conversations and a prompt of about 12K tokens); the others' come at 11.3 s, as in v0.2.6's better outcome (11.3 or 13.0 s).
• /health gains turns_nodrain, turns_boundary_first and pipe; pipe.groups is an empty list until the first decode group forms. Existing fields keep their names.
• The other models keep their behaviour. Their greedy replies equalled v0.2.6's after the first change (Qwen3.8-27B, Qwen3.8-Flash, MiMo-V2.6-Flash, DeepSeek-V4.1-Flash) and after the second (Qwen3.8-27B); the third changes only the 35B-A3B class's model file. On Qwen3.8-27B and Qwen3.8-Flash /health also gains turns_nodrain and turns_boundary_first.

New switches

switch	default	what the other value does
IE_Q35MOE_TURN_DRAIN	unset: the serial turns (restore, shared-prefix mark, conversation snapshot, prompt end, the snapshot of a reply that ended with a stop) run beside the lane pipe, each waiting only for its own lane's steps	=1 every turn pauses the pipe and the prompt's last rows run inside the prompt-end turn, as in v0.2.6
IE_Q35MOE_REGROUP	on	=0 no wait before a decode group forms; the groups stay as the lanes' arrival aligned them
IE_Q35MOE_GROUP_WAIT_US	20000 (1–1,000,000)	the most card 0 waits for the decode group landing on card 1 before forming a group that would leave three or more rotating
IE_Q35MOE_FA2_TILE_MINCTX	512 (6144 in v0.2.6): a prefill piece whose last position reaches this takes the tiled attention kernel	=6144 v0.2.6's kernel choice (the same bytes for the same prefill pieces)

Each is read once per process and applies to the 35B-A3B class's two-card split; the first three need more than one lane, the threshold applies to --parallel 1 as well. v0.2.6's switches are unchanged.

Requirements

Intel oneAPI 2026.x and an Intel Arc GPU. The agent-swarm configuration needs two Arc Pro B70 cards (64 GB VRAM); the largest models want about 256 GB of system RAM. Measured on Intel compute-runtime 26.35.39758.10 with the kernel's xe driver, the stack v0.2.0 and v0.2.6 shipped on.

Getting started

source scripts/env.sh
cmake -S . -B build -G Ninja && cmake --build build -j
./build/src/ie serve <35B-A3B-Q8_0.gguf> --gpus 2 --ctx 262144 --thinking on   # no --parallel: the load picks the lanes
curl -s localhost:11435/health                                                 # turns_nodrain, drains, pipe

ie <command> --help lists every option. QUICKSTART.md has the container route; the prebuilt image dates from July 2026 and predates this release, so build from source. The Dream Agent Harness drives this engine as a local agent, with its sub-agents on the request lanes.

Known limitations

• Many agents reading at once. With 15 agents that each append 1–3K tokens of tool output per turn, an agent decodes at a median 9.8 tok/s (5th percentile 7.1), against 25.9 tok/s per agent when 16 only write. The cards still spend the difference reading the agents' prompts. The replay's conversations stay under about 8K tokens and it has no lead; a live session with a lead was not re-measured on this release.
• The first prompt of a shared-prompt burst is slower than on v0.2.6. When 15 conversations arrive together with one shared prompt, the first one's first token comes at 11.3 s, against 8.6 or 10.2 s on v0.2.6 (by the timing of the followers' arrival). The 14 followers' first tokens are unchanged (11.3 s). The cause is not established: the leader's prefill is as fast as v0.2.6's (8.9 s), but 2.3–2.4 s pass between its first sampled token and the first byte the client receives, against 1.3–1.4 s on v0.2.6; a test build that ran its first decode step ahead of the followers' pieces did not change its first token. Not fixed in this release.
• The slowest 5 % of agent replies decode slower than on the build before the attention change: 7.87 / 7.84 → 7.12 / 7.13 tok/s, still above v0.2.6's 6.50–6.88. Per-step times are unchanged; the slowest replies are 18–55 tokens long and decode during the replay's opening while 14 other lanes read their prompts, and the faster prefill pieces leave more 1-row decode steps (285–288 against 175–177 in the workers phase).
• Bits change for prompts with prefill pieces ending between 512 and 6,143 positions on the 35B-A3B class; perplexity on such chunks moved by at most 0.5 % in either direction, on 511 scored tokens each. IE_Q35MOE_FA2_TILE_MINCTX=6144 gives v0.2.6's kernel choice.
• The regroup wait holds card 0 for up to 20 ms per merge: 636–650 ms over the replay (37–46 waits, 4–8 of them timed out at the bound).
• The changes are for the 35B-A3B class only. Qwen3.8-27B and Qwen3.8-Flash keep their paused turns; on them, on MiMo-V2.6-Flash and on DeepSeek-V4.1-Flash a long new prompt still slows or pauses the other lanes while it prefills.
• Not re-measured on the release build: the one-request 2,038-token prefill (measured with the environment knob on the v0.2.6 build), the lead-and-six-workers runs (measured on the first build of the series only), and the four-lane mid-prefill stop. The replies of Qwen3.8-Flash, MiMo-V2.6-Flash and DeepSeek-V4.1-Flash were compared with v0.2.6's after the first change only; the second was checked on the 35B-A3B class and Qwen3.8-27B, the third on the 35B-A3B class.
• The figures come from a community fine-tune of the model, from one machine, and from two runs per build; only the first change was measured with both builds alternating in one session.
• Verified on Arc Pro B70. Other Arc cards are untested by the maintainer. The limitations listed for v0.2.6 still apply where this release does not say otherwise.
• This is a 0.x release: interfaces may change.

Full changelog: v0.2.6...v0.2.8