Mach X v0.2.23

Everything since v0.2.21 (v0.2.22 was not published): MiMo-V2.6-Flash decodes 23-26 % faster with the same tokens, reads long prompts in ie serve up to 43 % faster (37,810 tokens: 92.5 -> 53.4 s), and DeepSeek-V4.1-Flash decodes about 7 % faster.

Build gate: the public tree built from scratch gives an ie byte-identical to the tested, installed one (md5 36d8a154); nine host suites pass (CPU expert kernels, OpenAI protocol, ChatML tools, server capabilities / controls / admission, CLI help, MiMo host rules).

Mach X v0.2.23

Everything since v0.2.21 (v0.2.22 was not
published). Every figure was measured on October 12, 2026, on two Arc Pro B70 cards with MiMo-V2.6-Flash-RL, the new code
against the same build with its switches set to the old behaviour; greedy unless a line says otherwise.

MiMo-V2.6-Flash writes 23–26 % faster, with the same tokens

MiMo's decode is bound by memory traffic: the routed experts that are not in VRAM either cross PCIe or run on the CPU's
E-cores, and both share the same RAM. The CPU half got cheaper: a layer's whole list of CPU experts runs in one parallel
region with one barrier per expert (it was about five regions per expert), the MXFP4 weights decode through an fp16 table
with a 2 KB prefetch, the elementwise work is split over the team, and the hand-offs between the GPU thread and the CPU
team spin for up to 2 ms before they sleep. The FP8 dense projections of a decode step read each weight once for up to
eight rows. Which experts go to the CPU and which to the GPU is unchanged, so every expert is rounded where it was.

MiMo-V2.6-Flash	old code	v0.2.23
decode with its DFlash drafter, 7 agent-style prompts x 128 tokens (two A-B-A runs)	21.8–23.3 tok/s	27.8–28.3
decode without the drafter (2 prompts)	14.1–14.7 tok/s	16.9–18.9
four requests at once in ie serve, decode together	20.6 tok/s	25.6

The generated tokens equal v0.2.21's in every arm (7 x 128). DeepSeek-V4.1-Flash runs the same CPU expert code: decode about
7 % faster (11.3 → 11.8–12.3 tok/s on a 5.7K-token prompt), prefill about 4 %, the text byte-identical in four runs.

MiMo-V2.6-Flash reads long prompts up to 43 % faster in ie serve

A prompt is read in 2,048-token pieces, and the two cards used to take turns on each piece. Now card 0 reads the next piece
while card 1 finishes the current one; every piece is computed as before, so a request's reply is the same.

ie serve, one request at a time	old	v0.2.23
6,003-token prompt, request lanes (the default)	13.4 s	9.6 s
5,466 / 5,403 tokens, lanes	12.9 / 12.2 s	9.2 / 8.7 s
37,810 tokens, lanes	92.5 s	53.4 s
6,003 / 37,810 tokens, --parallel 1	12.9 / 91.3 s	9.5 / 53.0 s

The replies were identical, and the lane test tool, which runs with the CPU expert path off, found the logits equal byte
for byte (two lanes and three prompts; one lane and a 37.8K-token prompt). The overlap is used while one request is reading; when a second request
is busy the pieces go one at a time, as before. Two requests at the same time can get other replies than in v0.2.21:
the overlap changes the order of the two requests' steps, and v0.2.21 gives the same replies when the second request
arrives a few seconds later (checked). Image prompts keep the old path.

Switches

Each restores the old behaviour: IE_MIMO26_PIPE_PREFILL=0 (the prompt overlap), IE_DS41_CPU_FUSED=0,
IE_CPU_MXFP4_PF=0, IE_CPU_MXFP4_F16=0, IE_CPU_MXFP4_TEAM=0, IE_DS41_CPU_SPIN_US=0, IE_DS41_CPU_BLOCKTIME=0 (the CPU
expert code), IE_MIMO26_FP8_LUT=1 (the FP8 projections).

What is still slow, and not checked

• Long-context decode on MiMo: at 30K tokens a full-attention layer takes 3.65 ms for the eight rows of a drafted step
  (0.55 for one row), bound by the per-key softmax chain. A matrix-engine version would be faster but changes the rounding;
  not done.
• Not checked: a client that leaves in the middle of an overlapped prompt read (in the test the server noticed only at
  the end of the prompt, old and new alike).
• Other models: their code did not change. The 35B-A3B class (16 lanes, 262K context), Flash-Next and the 27B were
  loaded on this build: replies with thinking on and off and tool calls as before, and on the 35B four requests at once equal
  to one at a time. One sampled 27B request showed the known mid-sentence </think> phantom tool call.
• Everything listed under New in v0.2.21 still holds.

Mach X v0.2.21

Everything since v0.2.20: Flash-Next now reads long image prompts with the image, restart points reach Flash-Next and image conversations, --spec is offered for the 35B-A3B class and Flash-Next, and Flash-Next's MTP head is available as an opt-in.

Build gate: the public tree built from scratch gives an ie byte-identical to the tested one (md5 57b6f835); six host suites pass.

Mach X v0.2.21

Everything since v0.2.20. Every figure was
measured on October 5, 2026, on two Arc Pro B70 cards, on community fine-tunes of the models, old path against new path on
the same day; "serve" is ie serve with an OpenAI client, greedy unless a line says otherwise.

Flash-Next reads long image prompts with the image

An image prompt of more than 2,048 tokens was answered by Flash-Next as if there were no image (v0.2.10 through v0.2.20). A
picture of a green square was described correctly at 281 and 1,318 prompt tokens and as "a red circle" or "a blue circle" at
2,199, 5,697 and 16,747. The pipelined prefill (the path for prompts over 2,048 tokens) resets both stages and did not stage
the picture again afterwards, so the model saw image-pad embeddings at plain positions; IE_Q4E_NO_PIPELINE=1 had been right
before the fix, which is how it was isolated. Now both test images are right at all four lengths and in a two-picture
conversation. The indexer's pooled block keys also take the picture's position streams, as its query already did
(IE_Q4E_QSA_MROPE_KEYS=0 reverts that part; the evidence for it is weak).

A conversation edited in the middle restarts from a saved point on Flash-Next, and image conversations keep theirs

The restart points of v0.2.18 now cover Flash-Next at one request lane. Its recurrent state (DeltaNet state and conv, the PLE
conv and history, the indexer's block count) is copied to pinned host memory, because the cards are full of expert cache,
at the conversation boundary, at the reply's end and wherever a 2,048-token chunk of the pipelined prefill passes a multiple
of the step; each stage takes its copy right after its own part of the chunk, so the pipeline is not cut. On a 15.7K-token
prompt (ie serve --ctx 65536): one line changed two thirds in 28.5 → 18.4 s, back to the first prompt 28.5 → 18.4 s, the
edited prompt with another question 26.4 → 16.2 s; the first read costs nothing (29.1 s against 29.0). All six replies equal
with the restart points on and off. IE_DN_LADDER=0 turns them off.

Image conversations keep their restart points too (27B and 35B-A3B splits, Flash-Next): the token record carries the picture's
hash at every image position, so the same picture shares its start and a different picture of the same size ends the shared
part where it starts. The second turn of a 16.7K-token image conversation:

second turn of a 16.7K-token image conversation	before	v0.2.21
Qwen3.8-27B	10.2 s	4.5 s
35B-A3B class	7.2 s	1.0 s
Flash-Next	24.8 s	3.0 s

The 35B-A3B replies were byte-equal (five requests, two pictures) and so was Flash-Next's first conversation; Flash-Next's
other follow-up was token-equal for 151 tokens of reasoning and then words its answer differently (the 32-token tail is read
in another chunk shape). IE_DN_LADDER_VISION=0 turns the image part off.

--spec is offered for the 35B-A3B class and Flash-Next

ie capabilities reported features.speculative only for the dense 27B and Gemma-4, so a client's load screen had no switch
for the other two even though --spec works there. It now reports it for the 35B-A3B class (lookup drafts across the request
lanes) and Flash-Next (lookup drafts at one request lane: with --parallel above 1 they are off, and the load says so; they
are not used on image requests). What the switch does is described in New in v0.2.18.

Flash-Next's MTP draft head: opt-in, and measured as not worth turning on

--spec --spec-head <mtp-head.gguf> (or IE_Q4E_MTP_HEAD) at one request lane loads the model's multi-token-prediction head
onto the tail stage (5.2 GB, a separate file taken from the base model's repository; it is not found by itself). Every committed
forward is fed to it, and where there is no repeat to copy a round drafts three tokens and checks four rows with the exact
verify kernels. The tokens are plain decoding's (800 greedy bench tokens and the restore / restart test equal); a draft of five
gave different tokens and was slower, so it stays at three. Measured on the UD-Q4_K_XL file, both cards:

Flash-Next, MTP head on, against off	off	on
bench decode, greedy	28.7 / 28.3 tok/s	30.8 / 33.7 (2.7–3.0 tokens a round)
bench decode at temperature 0.7		no gain
prefill	502 tok/s	412
serve, two long reasoning replies	36.5 / 36.0 s	34.1 / 33.8
serve, an ordinary question at temperature 0.7	9.0 s	9.4
serve, a 1,800-token file edit with --spec	30.7 s	33.5

A four-row check costs two plain steps and the head takes about 4.7 GiB of expert cache, so it is a small greedy gain against
a slower read; it is off unless a path is given. IE_Q4E_MTP=0 keeps it off even with a path.

What is still slow, and not checked

• Re-checked for this release on October 7, on the build that was published: Flash-Next with the vision tower answered both pictures correctly at 281, 1,318, 2,386, 6,617 and 16,747 prompt tokens (10 of 10); a 15.6K-token text prompt took 21.0 s to read and 12.4 s after one line changed two thirds in (a restart in place at token 8,192, 4.6 ms); ie capabilities reports speculative for Flash-Next; the capabilities, CLI-help and ChatML host tests pass. The 27B and 35B-A3B class were not loaded again: their figures above are the October 5 ones.
• The standalone runner's Flash-Next figures with the head (37.1–38.2 chat and 44.4–44.6 code tok/s, September 27, greedy,
  96 tokens, the base model) were not reproduced in ie serve. A side-by-side run of both on the same served file would say
  why; until then the table above is what the server does.
• Flash-Next's restart points and lookup drafts are for one request lane; the lanes do not have them. The MTP head is one
  lane, greedy, and is not combined with images.
• Not covered: a copy cursor or change list on the 27B and 35B-A3B class (small expected gain, skipped).
• Everything listed under New in v0.2.20 still holds.

Mach X v0.2.20

Seventh tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.20 is for people with one card: one Arc Pro B70 now runs the Qwen models on the same code as two. It also stops refusing loads from memory estimates and brings the in-place restart points to the lead request lane.

Everything since v0.2.18. Every figure was measured on October 5, 2026, on Arc Pro B70 cards, on community fine-tunes of the models; "bench" is ie-prompt-bench (prompt cache off, greedy, prompts of 2,404 and 10,693 tokens), "serve" is ie serve with an OpenAI client.

Highlights

One card

Most people have one card. Until now --gpus 1 put the Qwen models on older code: the 27B on an fp16 path, and the
35B-A3B class on a loader that did not accept a Q5_K_M or Q6_K file at all (token_embd: unsupported dtype). A Q8_0,
Q6_K or Q5_K file on one card now runs the two-card model's code on a single stage, so one card gets the same prefill and
decode kernels, the prompt cache, --spec with its lookup drafts, the in-place restart points and the request lanes.

one Arc Pro B70	prefill, tok/s (2.4K / 10.7K)	decode, tok/s	with --spec
Qwen3.8-27B Q6_K, before	190 / 173	13.1 / 12.5
Qwen3.8-27B Q6_K, now	823 / 830	22.0 / 20.4	32.6 / 19.8 tok/s
35B-A3B class Q5_K_M, before	did not load
35B-A3B class Q5_K_M, now	1,337 / 1,304	87.2 / 82.9	a 1,800-token file edit 22.2 s → 9.4 s

• Replies are the two-card ones: three greedy file edits gave byte-equal replies on one card and on two, on both models.
• Several requests at once on one card. The 35B-A3B class picks its request lanes by what the card has free: 3 with
  the Q5_K_M file (four 1,800-token edits and a question at once: 64.9 s, 45.1 s with --spec). The 27B takes one lane
  unless --parallel N says more.
• What fits one 32 GB card: the model file plus the context. The 27B at Q6_K (22.4 GB) or Q5_K (19.5 GB) and the
  35B-A3B class at Q5_K_M (25.3 GB) were run; the 27B at Q8_0 (29.0 GB) is tight and was not run; the 35B at Q8_0 (37.8 GB)
  does not fit. Other files (Q4_K_M) keep the previous one-card path.
• IE_QWEN35_ONE_CARD_SPLIT=0 / IE_Q35MOE_ONE_CARD_SPLIT=0 restore the previous one-card paths.

A load is tried as asked

The engine no longer refuses a load from a memory estimate.

• Qwen3.8-27B: its load-time memory check is gone. It loads at --ctx 250000 with the prompt cache and --spec on.
• The prompt cache gets what is free. On the 27B and the 35B-A3B class the cache's room is the VRAM each card reports
  free after the model, the context and the lanes are allocated (less 2 GiB for the first request's working buffers):
  6.1 GB a card on the 27B at --ctx 200000, 12.4 GB on the 35B-A3B class at one lane. IE_PROMPT_CACHE_VRAM_MIB sets it.
• Estimates warn. A lane count the free-memory measure says is too large (27B, 35B-A3B class), Flash-Next's and GLM's
  system-RAM checks, DeepSeek-V4.1's pinned-RAM floor and MiMo's free-memory reserve print a WARNING and the load goes
  on. Pinned system RAM cannot be swapped: a model that pins more than the machine has can stall it, and an over-filled
  card can take the display driver down. That choice is now the user's.
• What still stops a load: an allocation that really fails, and the two minimums without which a step cannot run
  (MiMo's smallest expert tier, DeepSeek-V4.1's stream partition).

Restart points on the lead lane

With request lanes on the 35B-A3B class, the lane that holds the long conversation (lane 0, the one at --ctx) now keeps
the in-place restart points of v0.2.18: after one line changed two thirds into a 29,300-token conversation the next
request took 7.8 s instead of 10.8 s, and the edited prompt with another question 5.9 s instead of 9.0 s; all six
replies byte-equal with them on and off. Sixteen busy lanes cost nothing measurable (56–58 s with, 55–58 s without).
IE_Q35MOE_LANE_LADDER=0 turns them off. A conversation short enough for a small lane (--slot-ctx) is served there,
without them.

Known limitations

• Not run on one card: Flash-Next, the DeepSeek, GLM and MiMo models (they lean on two cards and a lot of RAM), the
  27B at Q8_0, images.
• Not run: a load that truly does not fit, and the warning paths themselves (they need a machine short of memory).
• An explicit --parallel N on one card is tried even when it will not fit; the automatic count is the safe one.
• Everything listed under New in v0.2.18 still holds.

Mach X v0.2.18

Sixth tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.18 makes replies that copy text already in the conversation several times faster on the three Qwen models (with --spec), lets a conversation edited in the middle restart from a saved point instead of being read again, fixes --spec on the 27B's Q6_K / Q5_K files, and lets the 27B load at long context with the prompt cache on.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Everything since v0.2.13 (versions 0.2.14 to 0.2.18). Every figure was measured on October 4–5, 2026, on community fine-tunes of the models, old path against new path on the same day; "bench" is ie-prompt-bench (prompt cache off, greedy), "serve" is ie serve with an OpenAI client.

Highlights

Replies that copy are written several times faster (--spec)

An agent's reply often repeats text that is already in the conversation: a file returned with one change, a quoted
function, a tool result read back. With --spec the engine keeps an index of the conversation's tokens (built per
request, in memory, discarded after). When the last tokens of the reply repeat an earlier span, the tokens that followed
that span are taken as a draft and checked by the model in ONE forward; the reply follows the draft for as long as the
model's own sampled token is the drafted one, and the rest is rolled back. The model still decides every token, with
the request's sampler; nothing is added to the weights or the prompt.

a reply that returns a file with one identifier renamed (serve, temperature 0.7, 1,800 tokens)	plain	with --spec
Qwen3.8-27B Q8_0	108.5 s	28.4 s
35B-A3B class Q8_0, one request	24.3 s	7.1 s
Flash-Next UD-Q4_K_XL	68.7 s	30.7 s

bench, greedy	plain	with --spec
27B, a file copied with comments added (400 tokens)	16.9 tok/s	146 (58–67 with exact rows only)
27B, a documentation page repeated with rewording	17.0	37.3
35B-A3B class, the same copy prompt	78.4	281
35B-A3B class, the reworded page	84.4	118.7
Flash-Next, the copy prompt (600 tokens)	26.9	40.7
ordinary prompts (2.4K / 10.7K tokens): 35B-A3B class	78.2 / 72.5	78.1 / 74.8
ordinary prompts: Flash-Next	28.7 / 28.2	28.7 / 28.2

On the 35B-A3B class it works across the request lanes (ie serve --parallel N --spec): one lookup round runs at a
time and the other lanes make two steps between rounds.

35B-A3B class, ie serve, 1,800-token file edits at temperature 0.7	plain	with --spec
8 lanes: 4 edits and 1 ordinary question at once	36.8 s (202 tok/s together)	22.6 s (330)
... the ordinary question beside them	7.6 s	10.4 s
16 lanes: 12 edits and 1 ordinary question at once	74.3 s (294 tok/s together)	53.6 s (408)
... the ordinary question beside them	15.7 s	19.6 s

What it changes in the text. A long draft is checked through the prefill kernels (the ones that read every prompt),
not the decode kernels, so a token at a near-tie can come out differently from plain decoding, and the text after it
then differs. Measured: Flash-Next's greedy 900-token edit was byte-equal to plain decoding and the first 394 of 600
tokens of its copy bench were equal; on the 35B-A3B class one of three greedy edits was byte-equal and the first
different token of two copy benches came after 143 and 160 of 400 tokens. On the 27B, IE_QWEN35_LOOKUP_FAST=0 keeps
every row exact (text equal to plain decoding; 58 tok/s on the copy bench instead of 146). No task benchmark was run
with the feature on against off.

On Flash-Next a lookup round costs far more than on the other two (200 ms for 8 rows, about 2.3 s for 512, against
35–42 ms for a plain token), so it waits for a repeat of 32 tokens and starts at 64 rows; a reply that picks up the
copy again after a changed word resumes four tokens later, and a draft stops just before a word pair the reply has
been changing.

--spec on the 27B's Q6_K and Q5_K files

Three faults in the draft head off the Q8_0 path made --spec slower than plain decoding on these files (every draft
was noise). Fixed; greedy text equal to plain decoding on both files.

Qwen3.8-27B, --spec --spec-k 3, bench (2.4K / 10.7K-token prompts)	plain	speculative
Q6_K	21.5 / 19.8 tok/s	26.9 / 21.6
Q5_K	23.3 / 21.4	26.3 / 22.9
Q6_K, five short prompts (median)		33.2

--spec-k 3 measured best (Q8_0 / Q6_K, short prompts: K=3 29.7 / 33.2, K=4 29.3 / 32.4, K=5 28.6 / 28.5 tok/s).

A conversation edited in the middle restarts from a saved point

These models keep a running state over everything they have read (the DeltaNet layers), which cannot be rewound, so a
change in the middle of a context used to mean reading it again from the last saved prompt. On the Qwen3.8-27B and
35B-A3B splits at one request lane, with the prompt cache on, the engine now keeps small copies of that state every
8,192 tokens while a prompt is read (every 16,384 at --ctx 200000), one at the conversation boundary and one at the
reply's end. A next prompt that shares its beginning with the live conversation restarts from the deepest copy before
the change, in place: no KV rows are copied.

ie serve --ctx 65536, greedy, a 29,300-token prompt	before	now
27B: one line changed two thirds in	22.9 s	17.5 s (12,949 tokens read instead of 29,333)
35B-A3B class: the same	15.2 s	9.0 s
27B: next turns of a conversation too large for the prompt cache	22.9–23.0 s	11.1–17.5 s

Every reply was byte-equal with the copies on and off (six requests on each model). A copy is 73 MiB a card on the 27B
and 30 MiB on the 35B-A3B class, allocated on first use.

The 27B loads at long context with the prompt cache on

The load-time memory check on the 27B counted twelve saved prompts each as large as the whole context (99 GB a card
at --ctx 250000) and refused the load, although the context itself fit (24.5 of 34.2 GB). The check no longer
refuses: the prompt cache gets the memory the cards have left, drops its oldest entries to stay inside it, and skips a
snapshot that alone does not fit. A card whose weights and context alone pass 92 % prints a warning.

Switches

On by default where a line says so; =0 restores the previous path.

switch	applies to
--spec	turns the lookup drafts on (with the MTP draft head on the 27B); without it nothing here changes decoding
IE_QWEN35_LOOKUP, IE_QWEN35_LOOKUP_FAST (=0: exact rows only), IE_QWEN35_LOOKUP_MIN (8)	Qwen3.8-27B split, one request lane
IE_Q35MOE_LOOKUP, IE_Q35MOE_LANES_LOOKUP, IE_Q35MOE_LOOKUP_MIN (16), IE_Q35MOE_LOOKUP_GAP (2)	35B-A3B class: one lane; the lanes; the repeat that starts a lookup; the other lanes' steps between rounds
IE_Q4E_LOOKUP, IE_Q4E_LOOKUP_MIN (32), IE_Q4E_LOOKUP_ROWS (512)	Flash-Next, one request lane
IE_DN_LADDER, IE_DN_LADDER_STEP (8192)	the in-place restart points (27B and 35B-A3B splits, one lane, prompt cache on)

Known limitations

• Lookup needs --spec, and one request lane on the 27B and Flash-Next (the 27B's lanes are off under --spec).
• Only one lookup round runs at a time across the lanes: one edit gets 3.4×, twelve at once 1.4×, and a lane that
  is not copying waits longer beside them.
• Ordinary writing is unchanged: the 27B is still at the memory wall (about 17 tok/s plain, 20–25 with the draft
  head), and at temperature 1.0 fewer drafts are followed than at 0.7 (not measured here).
• Flash-Next often answers an edit with reasoning only, in plain decoding too; lookup does nothing for those replies.
• The restart points are not on the request lanes or Flash-Next; a prompt identical to the live one still restarts
  from the conversation boundary, not from its last token.
• ie serve --help does not describe the lookup drafts.
• Not run for this release: image requests (they take plain decoding and no restart points), --int8-kv, two
  clients sharing one lane during a lookup round, a client that disconnects during one, the 15-agent replay with
  --spec, a load of the 27B above --ctx 200000.

Mach X v0.2.13

Fifth tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.13 makes the Qwen models read prompts 1.4–3.7× faster, makes speculative decoding on Qwen3.8-27B usable in normal serving, and tests GPU memory at load for a page fault that corrupted one model on one of the test cards.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Every figure was measured on October 4, 2026 (October 2 where noted), on community fine-tunes of the models, old path against new path on the same day. "Bench" is ie-prompt-bench (prompt cache off, greedy) with prompts of 2,404 and 10,693 tokens.

Highlights

Prefill on the Qwen models

A per-card profile of the 27B's prefill showed attention at 47 % of the time and the per-chunk expansion of the 8-bit weights to fp16 at 25 %; the matrix multiply itself was 17 %. Three changes, none of them lower precision: a chunk's attention as two oneDNN matmuls on the XMX engines with a softmax kernel between them; Q8_0 weights read in place by the prefill matmul; the DeltaNet scan with its state in registers.

model	measure	before	v0.2.13
Qwen3.8-27B Q8_0	bench prefill, 2.4K / 10.7K tokens	1,037 / 820 tok/s	2,962 / 3,001 tok/s
	ie serve, a 20,248-token turn	33.0 s	7.8 s
Qwen3.8-27B Q6_K	bench prefill	987 / 794 tok/s	1,378 / 1,549 tok/s
35B-A3B class Q8_0	bench prefill	1,816 / 1,256 tok/s	2,896 / 2,808 tok/s
	ie serve --parallel 4, a 20,210-token turn	16.4 s	5.8 s
	the 15-agent replay (tools/swarm_replay.py)	116.1 s	77.1–77.2 s
	replay: a worker's decode, median (5th percentile)	9.89 (7.13) tok/s	14.8–15.2 (12.1) tok/s
	replay: a worker's first token, median; the wave's	2.79 s; 11.2 s	1.6–1.7 s; 5.8 s
Qwen3.8-Flash (Flash-Next)	bench prefill, text never seen before / a repeated prompt	241 / 487 tok/s	470 / 688 tok/s
	ie serve, a 20,248-token turn	39.4 s	28.1 s

Flash-Next also reads its 26.8 GiB per-layer token table into RAM in the background at load (it was read from the file mapping one page fault at a time), and its sparse attention runs as the same matmuls with the selected keys as a mask, up to 65,536 tokens of context.

Perplexity moved by −0.14 % to +0.21 % across the three models and is identical run to run. Decode speed without speculation is unchanged.

Speculative decoding on Qwen3.8-27B in normal serving

--spec used to run only at temperature 0, on prompts up to 4,096 tokens, and re-read the whole prompt. It is now a step of the normal decode loop: the engine samples every token and the step supplies already-verified logits whenever the sampled token equals the draft. It works with any sampler, the prompt cache and any prompt length; four ie serve replies at temperature 0.7 with a fixed seed were byte-equal to a plain server's.

Qwen3.8-27B Q8_0, --spec --spec-k 3	plain	speculative
ie serve, temperature 0.7, short prompt	17.1 tok/s	24.3
ie serve, temperature 0.7, 10,891-token prompt	15.9	17.8
bench, greedy, five short prompts (median)	17.1	29.7

A memory-page fault is caught at load

On one of the two test cards, two 2 MiB pages of a single GPU allocation can be backed by one physical page. On Flash-Next this put one expert's weights in another expert's cache slot and replies ended after a few tokens. Every device allocation of 64 MiB or more is now tested before use (about 0.5 s for 49 GiB across both cards); a faulty allocation is kept aside and another is made. Flash-Next's expert caches get their own test with slot quarantine.

Also since v0.2.8

• Image input on the Qwen3.8-27B and 35B-A3B two-card splits at --parallel 1 with --mmproj.
• /props prompt_cache_slots reports the two Qwen splits' real prompt-cache capacity.
• Flash-Next's expert-cache budget counts the context's KV and workspaces: at --ctx 250000 decode went from 0.2 to 23 tok/s (October 2).

Behaviour changes

• Greedy replies of the three Qwen models can differ from v0.2.8's at near-ties (the prefill paths round differently). Each new path has a switch that restores the previous one: IE_QWEN35_ATTN_GEMM, IE_QWEN35_S8_PREFILL, IE_QWEN35_DN_SCAN, IE_Q35MOE_ATTN_GEMM, IE_Q35MOE_S8_PREFILL, IE_Q35MOE_DN_SCAN, IE_Q4E_ATTN_GEMM, IE_Q4E_S8_PREFILL, IE_Q4E_DN_SCAN (=0).
• With --spec, the Qwen3.8-27B split speculates on sampled requests and long prompts too (IE_QWEN35_SPEC_STEP=0 = the previous greedy-only path).
• Flash-Next's load reads 26.8 GiB into the page cache in the background (IE_Q4E_PLE_PREFETCH=0 off).
• Every load tests its large device allocations (IE_NO_ALLOC_ALIAS_TEST=1 skips).

Known limitations

• Flash-Next's prefill is now 82 % expert layers; past 65,536 tokens of context its previous attention kernel runs.
• Plain decode on the 27B is unchanged (17 tok/s on Q8_0); --spec is slower than plain on the Q6_K file.
• The speculative gain depends on the draft head's hit rate (1.35–1.7 tokens per forward at temperature 0.7).
• ie serve --help still describes --spec as greedy-only.
• Not run for this release: image requests on the new prefill paths, --int8-kv, the 35B's Q6_K / Q5_K files.
• The allocation test cannot see a page shared between two different allocations.
• In the release smoke one 256-token DeepSeek-V4.1 reply differed between two runs of the previous build (the new build reproduced its own); cause not investigated.

Mach X v0.2.8

Fourth tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.8 makes v0.2.6's agent swarm faster: a lead and up to 15 sub-agents that read files and write at the same time, on the Qwen3.6-35B-A3B class (Q8_0) split over two Arc Pro B70 cards. The 15-agent replay takes 116.3–116.5 s where v0.2.6 took 131.3–132.4 s in the same day's runs (−12 %), an agent's decode goes from a median 8.65–8.80 to 9.83–9.86 tok/s, and its first token comes in 2.58–2.83 s instead of 3.79–4.79 s. Three changes do it: the engine's serial turns run beside the lane pipe instead of pausing it, the decoding lanes regroup, and the tiled prefill attention starts at 512 positions instead of 6,144. There is no new kernel and no change to the request API; the changes are for the 35B-A3B class.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Every figure was measured on October 2, 2026, on the driver stack v0.2.6 was measured on, on a community fine-tune of the model. Each change was measured by an independent run of its build against the build before it, two runs per build where a line gives two numbers, and each build's replies were compared byte for byte with the build before it.

Highlights

The agent replay: 12 % faster, a worker's first token in 2.6–2.8 s

The replay is v0.2.6's (tools/swarm_replay.py): 15 worker conversations for 3 turns each (prompts growing from 1.4K to about 8K tokens), then a wave of 15 new conversations that share one system prompt of about 12K tokens, with one follow-up each: 75 requests against ie serve --gpus 2 --ctx 262144 --parallel 16 --max-queue 16 --thinking on. Every run finished 75 of 75 requests with 0 errors. Each row compares two runs of a build with two runs of the build before it: alternating in one session for the first row, at most three hours apart for the other two (the same prompts). v0.2.6's release notes give 139.7 s for this replay, measured on October 1; the same binary measured 131.3 and 132.4 s on October 2, so compare within a row.

change	whole replay	workers' decode per request, median (5th percentile)	workers' first token, median
serial turns beside the lane pipe	132.4 / 131.3 → 127.7 / 127.7 s	8.65 / 8.80 (6.88 / 6.50) → 8.59 / 8.61 (7.66 / 7.65) tok/s	3.79 / 4.79 → 4.16 / 4.00 s
decoding lanes regrouped; a lone prompt's remainder re-cut, a prompt's tail first	127.8 / 128.9 → 122.2 / 122.2 s	8.76 / 8.61 (7.68 / 7.24) → 8.85 / 8.93 (7.87 / 7.84)	4.15 / 4.15 → 4.06 / 4.01 s
tiled prefill attention from 512 positions	122.2 / 122.2 → 116.3 / 116.5 s	8.85 / 8.93 (7.87 / 7.84) → 9.83 / 9.86 (7.12 / 7.13)	4.06 / 4.01 → 2.83 / 2.58 s
v0.2.6 → v0.2.8	131.3–132.4 → 116.3–116.5 s (−12 %); the workers phase 82.0–82.8 → 70.3 s	8.65–8.80 (6.50–6.88) → 9.83–9.86 (7.12–7.13)	3.79–4.79 → 2.58–2.83 s

Serial turns beside the lane pipe

Between a lane's prefill pieces and its decode steps the server takes short exclusive turns for that lane: restoring a cached prefix, marking a shared prefix, snapshotting the conversation, ending the prompt. In v0.2.6 every turn paused the two-card pipe and waited for every step in flight to land, because the prompt-end turn ran the prompt's last 3–6 rows (the generation prompt) as a full two-card forward on the cards' shared workspace. In one trace of the replay on v0.2.6 that was 84 pauses and 16.5 s of waiting, 89 % of it the prompt-end turn's.

Now those rows go through the pipe as the prompt's last piece, each turn's copies address their lane by index, and the turns run while the other lanes keep stepping. Only a lost lane's reset, and the snapshot of a reply that ended at its length limit (taken only on templates without thinking), still pause the pipe.

the replay, two runs of each build	v0.2.6	serial turns beside the pipe
pauses; the pipe's paused time	84–86; 13.3–13.4 s	0; 0.7 s
card 0 idle in the workers phase	19.9–20.5 s (24.5–25.0 %)	10.7–10.8 s (13.7–13.8 %)
card 0's longest idle gap before a decode step	2,068–2,070 ms	291–292 ms
the 15 workers' first tokens at the start (median)	5.17–5.19 s	4.22–4.23 s
workers' decode per request, 5th percentile	6.50–6.88 tok/s	7.65–7.66
workers' decode per request, median	8.65–8.80 tok/s	8.59–8.61

• The median decode did not move, and the trace shows why: without the pauses' release bursts the decoding lanes formed smaller groups (3.20 rows per card step against 4.53). The decode regroup, next, fixes that.
• With an 80K-token lead and six workers nothing regressed: the workers' first token 12.2 s against 12.6, the lead's 169.0 s against 169.5, the same replies.
• The replies of all five lane models (35B-A3B class, Qwen3.8-27B, Qwen3.8-Flash, MiMo-V2.6-Flash, DeepSeek-V4.1-Flash) equalled v0.2.6's byte for byte: 156 checks, 0 differences. In the replay the 45 worker replies equalled v0.2.6's; 28 of the 30 wave replies differ, because the wave's first prompt took the prefill path of a prompt that starts alone (its first 8,192 rows in one piece) in every run of the new builds. v0.2.6 takes that path too, depending on arrival timing (1 of its 4 runs), and the pieces decide the bits.
• IE_Q35MOE_TURN_DRAIN=1 restores v0.2.6's protocol; on the build with only this change the replay's 75 replies then equalled v0.2.6's.

Decoding lanes regroup

The decoding lanes step in groups. A group is one card step that reads the weights once, so a row costs 5.7–5.8 ms in a 1-row step and 2.2 ms in an 8-row step (17.5–17.7 ms). The pipe merges only the lanes it finds queued when it forms a group, and a landed group's lanes resubmit together, so three or more groups rotating over the two cards never met: one lands per step, one forms per step. v0.2.6's paused turns had been merging them by accident, releasing every parked lane at once.

Now, before forming a group that would leave three or more rotating, card 0 waits up to 20 ms for the decode group landing on card 1 and forms from everything queued. One or two decoders never wait, a prefill piece at the front of the queue never waits, and a merge is paid once.

workers phase of the replay, card 0	decode steps	rows per step	1-row steps	decode card time
v0.2.6	1,595–1,599	4.53–4.54	164–184	19.6 s
serial turns beside the pipe	2,184–2,196	3.30–3.31	792–819	21.9–22.4 s
with the regroup	1,586–1,587	4.56	175–177	19.6–19.7 s

The waits held card 0 for 636–644 ms over the replay (37–41 waits, 33–37 of them brought lanes, 4–5 timed out). The replay's replies are 75 of 75 identical to the build before, and 124 identity checks on the 35B-A3B class and Qwen3.8-27B found 0 differences: a row's bytes do not depend on its group. IE_Q35MOE_REGROUP=0 turns it off; IE_Q35MOE_GROUP_WAIT_US sets the wait.

• A lone prompt's remainder is re-cut, and a prompt's tail goes first. The first change had moved the first token of the wave's first prompt from 8.57 s (v0.2.6 when it interleaved with its followers from the start) or 10.22 s (v0.2.6 when it prefilled alone first) to 12.4 s: released into the running pipe, its remaining 3,275 rows ran as one piece, and its 7-row tail queued behind the 14 followers' first pieces. Now a prompt that started alone has its remaining pieces re-cut into 512-row pieces as soon as another request waits for its turn (that prompt's prefill 11.1 → 8.9 s; v0.2.6 alone took 8.8), and a prompt's tail piece goes to the front of the queue (one piece of at most 512 rows per prompt, so nothing starves). The first prompt's first token 12.38–12.43 → 11.17–11.31 s; the 14 followers' 12.41–12.42 → 11.29–11.31 s. The first prompt stays behind v0.2.6: see Known limitations.

Tiled prefill attention from 512 positions

v0.2.6 took the tiled attention kernel for a prefill piece whose last position reached 6,144 and the previous kernel below, a threshold copied from the 27B and never measured on this class. Measured on the replay's 512-row pieces (the size a prompt is read in beside running lanes; card 0 medians):

start of the piece	0–2K	2–4K	4–6K	6–8K	8–16K	fit below 6,144
threshold 6,144 (v0.2.6)	208.0 ms	247.1	295.8	276.6	366.1	155 ms + 30 per 1,000 positions
threshold 512 (v0.2.8)	190.1 ms (−9 %)	215.0 (−13 %)	248.8 (−16 %)	276.0	366.5	159 + 18, the same line as from 6,144 up

The tile has no overhead at depth 0 (a 512-row piece over 512 keys: 156.5 ms against 158.0), so the old threshold made a piece at 4–6K slower than one at 6–8K.

• In the replay: the workers' first token 4.06 / 4.01 → 2.83 / 2.58 s (median), their prefill 1,085 → 1,171 tok/s together, their decode median 8.85 / 8.93 → 9.83 / 9.86 tok/s, the workers phase 75.9 → 70.3 s. The wave, all of it deeper than 8K, is unchanged.
• One request at a time: a 2,038-token cold prompt 1.41 → 1.28 s (server-side 1,395 → 1,271 ms; measured with the threshold set by environment on the v0.2.6 build). 8,065-, 16,369- and 32,161-token prompts are unchanged within 0.4 %: a prompt alone is read in 8,192-row pieces, which were on the tile already. Needle prompts of 2,043 and 3,580 tokens got their first token in 1.36–1.42 s instead of 1.47–1.50 and in 2.30–2.31 s instead of 2.65–2.67, found 6 of 6 needles, and gave byte-identical replies.
• Numerics: perplexity on the 12,288-token anchor is 2.3529 at both thresholds. On prefill chunks of 4,096 / 2,048 / 1,024 / 512 rows, which change kernel, it moves −0.064 / +0.183 / −0.497 / +0.028 % (wikitext-2, 511 scored tokens each; the sign changes between windows).
• Bits: a prompt whose prefill pieces end between 512 and 6,143 ...

Mach X v0.2.6

Third tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.6 is about agent swarms on one engine: a lead and up to 15 sub-agents that read files and write at the same time, on the Qwen3.6-35B-A3B class (Q8_0) split over two Arc Pro B70 cards. A replay of that workload which did not finish in 336 s now takes 139.7 s. The release also returns tool calls and reasoning in the OpenAI fields on the Qwen templates, moves the Q8_0 kernels to the compiler's native integer dot (bit-identical; Qwen3.8-27B at 16 lanes 113.8 → 153.5 tok/s), lets the engine pick the number of request lanes, makes an orderly stop take about a second, and runs Q6_K and Q5_K natively on the 27B and 35B-A3B splits.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM). Figures were measured September 29 – October 1, 2026 on the driver stack v0.2.0 shipped on; the 35B-A3B-class figures, and the Qwen3.8-27B's 16-lane and Q6_K / Q5_K figures, are on community fine-tunes of those models. One run per figure unless a line says A-B-A.

Highlights

Agent swarms: up to 16 agents that read and write at once

v0.2.0's 16-lane figure was measured with short prompts and long replies. Agents do the opposite: each turn appends 1–3K tokens of tool output and writes a short reply, so the cards spend most of their time reading. The engine's replay of that workload (tools/swarm_replay.py) runs 15 worker conversations for 3 turns each (prompts growing from 1.4K to about 8K tokens), then a wave of 15 new conversations that share one 12.3K-token system prompt, with one follow-up each: 75 requests against ie serve --gpus 2 --ctx 262144 --parallel 16 --thinking on, the flags the Dream Agent Harness starts it with. Its worker phase reads 87.8K new prompt tokens and writes 8.6K.

build	whole replay	decode, summed over the run	decode per request, workers (median)	the shared-prompt wave
September 30, before this work	not finished at the 336 s cut (45 of 75 requests)	25.4 tok/s	3.87 tok/s	0 of 30 requests done: each of the 15 agents read the shared prompt itself
v0.2.6's scheduler and kernels, October 1	139.7 s, 75 of 75	101.2 tok/s	8.10 tok/s (5th percentile 6.64)	30 of 30 done

Step by step, each pair of runs made back to back (the same build measured 146 s and 158.0 s in two sessions, so compare within a row):

change	whole replay	decode, summed over the run
a shared prompt read once, conversations kept per lane, tiled prefill attention (September 30)	not finished at 336 s → 212.7 s	25.4 → 67.0 tok/s
native int-dot Q8_0 MoE kernels (October 1)	214.9 → 169.6 s	66.3 → 84.2
both cards busy during a deep prefill, short requests first (October 1)	166.2 → 154.3 s	86.3 → 92.7
512-row prefill pieces beside running lanes, were 2,048 (October 1)	156.1 → 144.6 s	91.6 → 99.1
native int-dot Q8_0 GEMVs (October 1)	144.8 → 139.7 s	99.0 → 101.2

• A shared prompt is read once. When several agents arrive with the same system prompt and tools, the first one reads it and the others wait for it and restore it (the prefill FIFO, now on by default for the 35B-A3B class). In the replay's wave, all 15 agents used to read the 12.3K-token prompt themselves and none had finished a reply after 142 s; now 14 of 15 restore 12,279 tokens and the wave's 30 requests finish in 69 s (September 30; 53.2–58.4 s on the October 1 builds).
• A conversation resumes in its lane. Each lane keeps a checkpoint of its conversation and restores the next turn in place; the prompt cache is sized for the lanes within a VRAM budget, and a conversation's new snapshot replaces its older ones. Worker follow-ups that restored their previous turn: 6 of 30 before, 30 of 30 now; the worker phase reads 87.8K new tokens instead of 126.6K. A new user query after a tool loop restores up to the previous query's end (373 cached tokens instead of 0 in a five-round test).
• Long prompts prefill faster at depth. Tiled prefill attention from 6,144 positions: a cold 55,923-token prompt in 94.5 s instead of 200.4 s; a follow-up with 1,164 new tokens at 56K depth in 6.38 s instead of 11.70 s. Not bit-identical: perplexity 2.3529 against 2.3525 (+0.017 %), needles 6 of 6 at 32K and 54K.
• Both cards busy during a deep prefill. A lane's next piece enters card 0 while card 1 runs the piece before it. A 51,459-token prompt beside a decoding lane: first token in 64.1 s instead of 119.3 s (×1.86), the cards 93 / 96 % busy instead of 50 / 51 %, the lane decoding beside it at 18.1 tok/s instead of 10.9, with the same bytes.
• Short requests go first. A prompt with at most 16,384 tokens left to read goes ahead of a long prompt's next piece; the long prompt waits for at most 2 short pieces or 10 s. With an 80K-token lead prefilling and six ~2.2K-token workers arriving 2 s later, the workers' first tokens came at 53 s; with the lane pipeline alone at 208 s, with neither at 267 s. The lead's own first token came at 175.9 s, against 208.0 s and 267.7 s.
• Workers keep writing during a deep prefill. A long prompt that started alone in 8,192-row pieces is cut into 512-row pieces once another lane decodes or reads a short prompt (re-cut), and its next piece waits until every decoding lane has made 32 steps, for at most 2 s (the decode quota).

The same lead and six workers (--parallel 16 --ctx 262144 --thinking on, a 79,952-token lead, October 1):

workers' replies	build or setting	workers' first token	workers' decode while the lead prefills	workers done	lead's first token
69–319 tokens	lane pipeline and short-first only	53.2 s	0.09 tok/s each	176.8 s (median)	175.8 s
69–319 tokens	v0.2.6 defaults	12.6 s	20.1 tok/s each (median)	18.6 s (median; the last at 29.5 s)	169.8 s
1,200 tokens	lane pipeline and short-first only	34.2 s	0.09	202.7 s	175.9 s
1,200 tokens	v0.2.6 defaults	not recorded	11.6	116.1 s	214.2 s
1,200 tokens	IE_Q35MOE_DECODE_QUOTA=0	12.7 s	0.51	187.5 s	162.4 s

The short-reply row for the defaults was measured with those values set by environment, on the build before they became the defaults. The quota is a trade: with short, tool-call-sized replies it costs the lead nothing (169.8 s against 175.8 s); with long worker replies the lead's prefill takes 214 s, 22 % more than before and 32 % more than with the quota off, and all the work is done at 215 s instead of 190 s.

Tool calls and reasoning in the OpenAI fields

• XML tool calls come back as structured tool_calls. Models whose GGUF chat template teaches the <function= XML tool form (Qwen3.8-27B, the 35B-A3B class) get that template's own tools block and history form, and their calls are parsed into tool_calls with parameters typed by the request's schema. A captured Dream explorer request went from 0 of 8 alone and 0 of 8 at once to 7 of 8 and 8 of 8 structured (35B-A3B class, --parallel 16, September 30). IE_QWEN_TOOLS_JSON=1 restores the Qwen3 JSON preamble.
• Calls are taken only from the answer. With thinking on, a call block the model quotes while it reasons is never a call; a function name the request does not list is rejected; True, 02 and Python-quoted lists are typed by the schema. A reply that ends with a complete call without ever closing its reasoning is accepted as that call.
• Thinking goes to reasoning_content on the Qwen thinking templates (Qwen3.8-Flash, Qwen3.8-27B, the 35B-A3B class), streamed and non-streamed alike; content holds the answer only. A reply cut off before the model closes its reasoning is all reasoning_content with an empty content, as on DeepSeek, MiMo and GLM. With thinking off the bytes are unchanged.
• Checked on October 1 on Qwen3.8-Flash (1 and 4 lanes), Qwen3.8-27B, the 35B-A3B class (16 lanes), MiMo-V2.6-Flash and DeepSeek-V4.1-Flash: one-call requests 4 of 4 structured on each, and requests that paste a tool-call snippet as text gave 0 calls.

Faster kernels

In the one-token Q8_0 GEMV, the engine's portable integer-dot helper compiled to 28 instructions per dot on the B70, 24 of them byte moves; the compiler's native dot takes 4. v0.2.6 moves the Q8_0 kernels of the 35B-A3B class, Qwen3.8-27B and Qwen3.8-Flash to the native dot, each in a new kernel beside the old one, with the same per-lane sums in the same order: the replies checked on all three models, and perplexity on the 35B-A3B class, are byte-identical with the switches on or off.

kernel	measured	switch (=0 = the previous kernel)
Q8_0 MoE prefill, down and gate+up (35B-A3B class)	per launch at 2,048 rows: down 17.56 → 3.15 ms, gate+up 10.57 → 6.26 ms. One request's prefill, A-B-A: 2,038 tokens 2.20 → 1.41 s, 8,065 tokens 9.04 → 5.96 s, 16,369 tokens 22.72 → 16.44 s, 32,161 tokens 60.86 → 48.38 s	IE_Q8_MOE_DOWN_V2, IE_Q8_MOE_GATEUP_V2
Q8_0 MoE decode and lane rows (35B-A3B class)	16 lanes 390.6–391.3 → 414.1 tok/s together (+5.9 %, 25.9 per lane), 8 lanes 315.2–316.7 → 331.9, one request 80.8–81.1 → 81.9–82.4; A-B-A, --ctx 32768 --parallel 16	IE_Q8_MOE_DECODE_V2
Q8_0 GEMVs, lane rows and one token (Qwen3.8-27B, 35B-A3B class, Qwen3.8-Flash)	Qwen3.8-27B 16 lanes 113.7–113.8 → 153.5 tok/s together (+34.9 %), 8 lanes 92.3 → 111.3; Qwen3.8-Flash 16 lanes 124.6–124.8 → 128.0 (+2.6 %); one request +0.4–0.6 %; A-B-A, --ctx 8192 --parallel 16	IE_Q8_SOA_BATCHED_V2, IE_Q8_SOA_GEMV_ND, IE_Q8_SOA_GEMV_G_ND
XMX decode attention at head size 256, from 4,096 tokens of context (27B and 35B-A3B splits)	decode at a 32,979-token prompt: 35B-A3B class 42.01 → 57.95 tok/s, Qwen3.8-27B Q8_0 13.75 → 14.56, Q6_K 16.68 → 17.89. Not bit-identical:...

Mach X v0.2.0

Second tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.0 puts several models behind one endpoint, decodes up to 16 requests at once on MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.6-35B-A3B-class models and Qwen3.8-27B, moves to a newer Intel driver stack, and brings MiMo-V2.6-Flash — released the day after v0.1.0 and running here the same day — to vision, conversations kept in host memory, and per-token vital signs. Requests that leave sampling fields out now get each model's recommended values for thinking or instruct mode.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM), with host RAM holding the experts that do not fit.

Highlights

Several models, one endpoint

• ie supervise --config <layout.json> starts each server of a layout as its own ie serve child on its own cards and serves one OpenAI-compatible endpoint in front of them, routing each request by its "model". Streams pass through byte for byte; /v1/models, /health and /props?model=<name> report every server; a crashed child is reported while the others keep serving, and "restart": "on-failure" restarts it (at most 3 times, never after a lost device). Shutdown asks each child to stop and never kills one.
• ie serve --config <layout.json>: a JSON layout holds the model, cards, port, context, every launch flag and environment variables, validated by the command line's own parser; command-line flags win.
• --cards and ie cards: pin a process to chosen GPUs, list them without loading anything, and at most one engine process per card (a per-card lock). Pinned to card 1, a server put 2,292–2,363 MiB on card 1 while card 0 stayed at 28 MiB; two servers, one per card, answered concurrent requests.
• Per-command help: ie --help, ie help <command>, ie <command> --help list every flag, default, architecture restriction and relevant environment variable, without loading a model.

Several requests at once (request lanes)

ie serve --parallel N takes up to 16 requests at once on MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.6-35B-A3B-class models and Qwen3.8-27B, behind a per-model check at load that the lanes fit in memory (a count that does not is refused with the numbers). New for the 35B-A3B class, Qwen3.8-Flash and Qwen3.8-27B is row batching: the decoding lanes' one-token steps run as one grouped card step that reads the weights once, and every reply is byte-identical to the same request run alone. Here and below, "byte-identical" and "the previous server byte for byte" compare requests with the same sampling settings (see Recommended sampling per model).

model	1 request	4 at once	8 at once	16 at once
Qwen3.6-35B-A3B class	60.8 tok/s	178.8	249.2	310.2	×5.11 at 16
Qwen3.8-Flash	28.0	74.7	97.7	118.4	×4.24 at 16; long counting-task replies
Qwen3.8-27B	17.1	60.0	88.4	108.0	previous joint-step path: 49.3 / 60.0 / 63.4 at 4 / 8 / 16
MiMo-V2.6-Flash	~20	22.8	–	24.7	flat (drafter on, the default)
DeepSeek-V4.1-Flash	~13	22.6	–	25.7	×1.92 at 16

Tokens per second summed over the requests; release-candidate build, new driver stack, September 27–28, 2026; --ctx 32768 with 32K lanes (the 27B --ctx 16384 --slot-ctx 4096), greedy, short prompts, long replies, one A-B-A run per N. The 35B-A3B-class row was measured on a community fine-tune of that model; V4.1 on a long tutorial task, because it declines the counting task. MiMo and DeepSeek-V4.1 scale less because their experts live partly in host RAM: experts not on the card come over PCIe or run on the CPU expert leg, and from two requests up that host side, not the cards, sets the pace, so lanes share out a fixed budget of expert work; on MiMo the drafter already fills a step's 8 rows for one request.

What each agent sees: tokens per second per request, approximate, assuming an even share (the total above ÷ N):

model	N=4, each	N=8, each	N=16, each	alone
Qwen3.6-35B-A3B class	44.7	31.2	19.4	60.8
Qwen3.8-Flash	18.7	12.2	7.4	28.0
Qwen3.8-27B	15.0	11.1	6.75	17.1

Choosing N: use few lanes when each agent needs to be fast, and many lanes for bulk throughput.

• Shared-prefix cache (on by default for the 35B-A3B class and Qwen3.8-Flash; opt-in on Qwen3.8-27B): a system prompt plus tools that several conversations share (1,024 tokens or more) is cached once and restored into any lane. Four agents arriving at once got their first tokens in 2.9–3.1 s instead of 60.5 s on the 35B-A3B class (13.1K-token prefix, 16 tools), and in 4.2–4.5 s instead of 65.6 s on Qwen3.8-Flash (8.7K), byte-identical to a cold prefill. IE_SHARED_PREFIX=0 turns it off. On the 27B, IE_QWEN35_SHARED_PREFIX=1 turns it on; it is off by default there because it splits the 27B's prefill at the shared prefix, which changes that model's single-request output for long system prompts.
• Context per lane: 16 lanes of 64K context fit on the 35B-A3B class and on Qwen3.8-Flash. On Qwen3.8-Flash lane context comes out of the expert cache, so 64K lanes cost about 10 % against 8K lanes.
• Switches: IE_Q35MOE_ROWS=0, IE_Q4E_ROWS=0 and IE_QWEN35_ROWS=0 turn row batching off; IE_QWEN35_LANES=0 returns the 27B to its joint-step path. Opt-in: IE_LANES_PREFILL_FIFO=1 (with IE_LANES_HANDOVER_MS) hands the cards back to decoding lanes between new prompts' prefills (in a burst of four ~13K-token prompts, the longest decode pause 11.4 s instead of 27.8 s, first tokens 45–91 s instead of 52 s); IE_MIMO26_DFLASH_MAX_LANES=5 stops MiMo's drafting while five or more lanes decode (off by default: slower at 8 lanes).
• MiMo lanes identity fix: at --parallel > 1 MiMo no longer reuses a cached prefix shorter than 1,024 tokens, so a reply no longer depends on the lane or the arrival order; short multi-turn chats re-read up to 1,023 tokens a turn. --parallel 1 is unchanged.

The first lanes were MiMo's; the per-model notes below date from September 26–27 and were measured with two to four lanes unless they say otherwise. On MiMo-V2.6-Flash each lane holds its own attention caches, sliding-window rings and drafter context, carved from the static expert tier. With two lanes the two cards work on different lanes' steps at the same time; with four, two lanes' rows share one forward, and every shared operation is bit-identical row by row.

lanes	aggregate decode	against one request
2	28.1–28.2 tok/s	×1.66–1.69
4	28.9–30.0 tok/s	×1.85–1.90

Plain decoding with the CPU expert path on, as served, measured A-B-A with the engine's lanes test on held-out prompts. With the CPU expert path off every lane's output is byte-identical to its solo run, and one lane (the default) is the previous engine. MiMo's single-request default adds the DFlash drafter (23.6 tok/s), so the served gain over one drafted request is smaller.

• ie serve --parallel N for MiMo-V2.6-Flash (N up to 16): lane 0 keeps --ctx, the other lanes get --slot-ctx positions (default 32,768), and --parallel 1, the default, is the previous server byte for byte, streamed responses included (ids and timings aside). The lanes draft with DFlash within a budget of 8 rows per step: up to 7 drafts for a request decoding alone or for each of two, up to 3 each for three or four (IE_MIMO26_DRAFT_BUDGET=0 drafts only for a lone request). On held-out Dream transcripts 2 requests decode at 28.38 tok/s together, 14.2 each, and 4 at 28.46, 7.1 each: ×1.25 and ×1.27 against one drafted request on the same prompts (one run each, on a busy machine). A 1.6K-token prompt arriving while three requests decode got its first token in 4.4 s (measured before the draft budget). With the CPU expert path and the drafter off, every reply equals its solo run and a cancelled request leaves the others unchanged; over 20 requests host memory grew 17 MiB after the first wave, VRAM and threads flat. Images are served at --parallel 1 only and prompt lookup is off above 1; with the drafter on, a greedy reply can change at a near-tie when other requests decode beside it; and when only a smaller lane is free, a reply can be cut at that lane's size.
• ie serve --parallel N for DeepSeek-V4.1-Flash (N up to 16, on both cards): the same lane layout (--slot-ctx default 32,768, out of the static expert tier), and --parallel 1, the default, is the previous server byte for byte. Prompt lookup works per request, as with one lane. 2 requests decode at 24.92 tok/s together, 12.46 each, against 14.81 / 13.99 for one request (×1.73); more lanes add room and little speed (22.6 tok/s at 4, 25.7 at 16). A 1.7K-token prompt arriving while two requests decode got its first token in 7.9 s, against 7.1 s alone; over 20 requests host memory grew 10 MiB after the first wave, VRAM 10 MiB per card, threads flat at 119. With the CPU expert path off, every reply equals its solo run, prompt lookup on or off, and a cancelled request leaves the others unchanged. Images work with several lanes: a prompt with images prefills in its own turn while the other requests wait. With more than one lane the load refuses DSpark (IE_DS41_SPEC=1), IE_DS41_PROFILE_OUT, IE_DS41_DUMP_ROUTING, expert parallel (IE_DS41_EP) and a single card; the lanes were measured with --ctx 32768. The same lanes run in the ie-ds41-lanes-test tool too: two lanes ×1.33–1.58 against one request with plain decoding.
• GLM-5.3-Flash: request lanes and a two-card pipe, in a test tool (ie-glm-lanes-test), off by default: two lanes decode at 22.26 tok/s together against 13.28 for one request (×1.676, in the configuration ie serve uses), byte-identical to solo with the CPU expert path off. ie serve runs GLM ...

Mach X v0.1.0

First tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM), with host RAM holding the experts that do not fit.

The five models it is tuned for

model	prefill	decode
DeepSeek-V4.1-Flash (475 GB)	304 tok/s at 2K, 319 at 32K–223K	12.8 chat, 14.0 in agent loops
DeepSeek-V4-Flash (155 GB)	571 tok/s at 4K	26.1 at 4K, 32.7 short
GLM-5.3-Flash	156 tok/s at 16K	14.5 at 16K
Qwen3.8-Flash (104 GB)	467 tok/s pipelined	35.0 chat, 41.8 code
Qwen3.8-27B (Q8_0)	945 tok/s at 2K	24.5 tensor-parallel + speculative

Conditions and methods are in the README's Benchmarks section. These come from one machine; your hardware will differ.

What is in it

• One OpenAI-compatible server (ie serve) with native tool calls, and image input on checkpoints that carry a vision tower.
• A prefix cache that answers a follow-up turn in 1–2 s instead of re-reading the whole conversation.
• Tensor-parallel multi-GPU, running models larger than one card.
• Hand-written SYCL kernels — XMX matrix engines, int-dot quantized GEMV, tiled FlashAttention.
• Dense, MoE and hybrid architectures: GLM-5.3, DeepSeek-V4 and V4.1, Qwen3.8-Flash-Next, Qwen3.6, Qwen3 / Coder / Tongyi, Qwen3-Next, Gemma-4, gpt-oss, and Llama-compatible dense models.

Requirements

Intel oneAPI 2026.x and an Intel Arc GPU — most of the engine only runs on one. A container path is in QUICKSTART.md if you want to try it without building.

Getting started

See the From source section of the README, or QUICKSTART.md for the container route. The Dream Agent Harness drives this engine as a local agent, including tool calls.

Known limitations

• Verified on Arc Pro B70. Other Arc cards are untested by the maintainer.
• Large-model configurations assume substantial host RAM — the 475 GB checkpoint wants around 256 GB.
• GLM-5.2 and Tencent Hy4-preview have experimental standalone runners rather than the main path.
• This is a 0.x release: interfaces may change.