Skip to content

Mach X v0.2.0

Choose a tag to compare

View all tags
@Red-Weasel Red-Weasel released this 02 Oct 07:14
· 12 commits to main since this release
v0.2.0
0f80922

Second tagged release of Mach X — a C++/SYCL LLM inference engine for Intel Arc GPUs, written from the metal up. No llama.cpp fork, no PyTorch, no vendor runtime.

v0.2.0 puts several models behind one endpoint, decodes up to 16 requests at once on MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.6-35B-A3B-class models and Qwen3.8-27B, moves to a newer Intel driver stack, and brings MiMo-V2.6-Flash — released the day after v0.1.0 and running here the same day — to vision, conversations kept in host memory, and per-token vital signs. Requests that leave sampling fields out now get each model's recommended values for thinking or instruct mode.

Built and tuned on two Arc Pro B70 cards (64 GB VRAM), with host RAM holding the experts that do not fit.

Highlights

Several models, one endpoint

  • ie supervise --config <layout.json> starts each server of a layout as its own ie serve child on its own cards and serves one OpenAI-compatible endpoint in front of them, routing each request by its "model". Streams pass through byte for byte; /v1/models, /health and /props?model=<name> report every server; a crashed child is reported while the others keep serving, and "restart": "on-failure" restarts it (at most 3 times, never after a lost device). Shutdown asks each child to stop and never kills one.
  • ie serve --config <layout.json>: a JSON layout holds the model, cards, port, context, every launch flag and environment variables, validated by the command line's own parser; command-line flags win.
  • --cards and ie cards: pin a process to chosen GPUs, list them without loading anything, and at most one engine process per card (a per-card lock). Pinned to card 1, a server put 2,292–2,363 MiB on card 1 while card 0 stayed at 28 MiB; two servers, one per card, answered concurrent requests.
  • Per-command help: ie --help, ie help <command>, ie <command> --help list every flag, default, architecture restriction and relevant environment variable, without loading a model.

Several requests at once (request lanes)

ie serve --parallel N takes up to 16 requests at once on MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.6-35B-A3B-class models and Qwen3.8-27B, behind a per-model check at load that the lanes fit in memory (a count that does not is refused with the numbers). New for the 35B-A3B class, Qwen3.8-Flash and Qwen3.8-27B is row batching: the decoding lanes' one-token steps run as one grouped card step that reads the weights once, and every reply is byte-identical to the same request run alone. Here and below, "byte-identical" and "the previous server byte for byte" compare requests with the same sampling settings (see Recommended sampling per model).

model 1 request 4 at once 8 at once 16 at once
Qwen3.6-35B-A3B class 60.8 tok/s 178.8 249.2 310.2 ×5.11 at 16
Qwen3.8-Flash 28.0 74.7 97.7 118.4 ×4.24 at 16; long counting-task replies
Qwen3.8-27B 17.1 60.0 88.4 108.0 previous joint-step path: 49.3 / 60.0 / 63.4 at 4 / 8 / 16
MiMo-V2.6-Flash ~20 22.8 – 24.7 flat (drafter on, the default)
DeepSeek-V4.1-Flash ~13 22.6 – 25.7 ×1.92 at 16

Tokens per second summed over the requests; release-candidate build, new driver stack, September 27–28, 2026; --ctx 32768 with 32K lanes (the 27B --ctx 16384 --slot-ctx 4096), greedy, short prompts, long replies, one A-B-A run per N. The 35B-A3B-class row was measured on a community fine-tune of that model; V4.1 on a long tutorial task, because it declines the counting task. MiMo and DeepSeek-V4.1 scale less because their experts live partly in host RAM: experts not on the card come over PCIe or run on the CPU expert leg, and from two requests up that host side, not the cards, sets the pace, so lanes share out a fixed budget of expert work; on MiMo the drafter already fills a step's 8 rows for one request.

What each agent sees: tokens per second per request, approximate, assuming an even share (the total above ÷ N):

model N=4, each N=8, each N=16, each alone
Qwen3.6-35B-A3B class 44.7 31.2 19.4 60.8
Qwen3.8-Flash 18.7 12.2 7.4 28.0
Qwen3.8-27B 15.0 11.1 6.75 17.1

Choosing N: use few lanes when each agent needs to be fast, and many lanes for bulk throughput.

  • Shared-prefix cache (on by default for the 35B-A3B class and Qwen3.8-Flash; opt-in on Qwen3.8-27B): a system prompt plus tools that several conversations share (1,024 tokens or more) is cached once and restored into any lane. Four agents arriving at once got their first tokens in 2.9–3.1 s instead of 60.5 s on the 35B-A3B class (13.1K-token prefix, 16 tools), and in 4.2–4.5 s instead of 65.6 s on Qwen3.8-Flash (8.7K), byte-identical to a cold prefill. IE_SHARED_PREFIX=0 turns it off. On the 27B, IE_QWEN35_SHARED_PREFIX=1 turns it on; it is off by default there because it splits the 27B's prefill at the shared prefix, which changes that model's single-request output for long system prompts.
  • Context per lane: 16 lanes of 64K context fit on the 35B-A3B class and on Qwen3.8-Flash. On Qwen3.8-Flash lane context comes out of the expert cache, so 64K lanes cost about 10 % against 8K lanes.
  • Switches: IE_Q35MOE_ROWS=0, IE_Q4E_ROWS=0 and IE_QWEN35_ROWS=0 turn row batching off; IE_QWEN35_LANES=0 returns the 27B to its joint-step path. Opt-in: IE_LANES_PREFILL_FIFO=1 (with IE_LANES_HANDOVER_MS) hands the cards back to decoding lanes between new prompts' prefills (in a burst of four ~13K-token prompts, the longest decode pause 11.4 s instead of 27.8 s, first tokens 45–91 s instead of 52 s); IE_MIMO26_DFLASH_MAX_LANES=5 stops MiMo's drafting while five or more lanes decode (off by default: slower at 8 lanes).
  • MiMo lanes identity fix: at --parallel > 1 MiMo no longer reuses a cached prefix shorter than 1,024 tokens, so a reply no longer depends on the lane or the arrival order; short multi-turn chats re-read up to 1,023 tokens a turn. --parallel 1 is unchanged.

The first lanes were MiMo's; the per-model notes below date from September 26–27 and were measured with two to four lanes unless they say otherwise. On MiMo-V2.6-Flash each lane holds its own attention caches, sliding-window rings and drafter context, carved from the static expert tier. With two lanes the two cards work on different lanes' steps at the same time; with four, two lanes' rows share one forward, and every shared operation is bit-identical row by row.

lanes aggregate decode against one request
2 28.1–28.2 tok/s ×1.66–1.69
4 28.9–30.0 tok/s ×1.85–1.90

Plain decoding with the CPU expert path on, as served, measured A-B-A with the engine's lanes test on held-out prompts. With the CPU expert path off every lane's output is byte-identical to its solo run, and one lane (the default) is the previous engine. MiMo's single-request default adds the DFlash drafter (23.6 tok/s), so the served gain over one drafted request is smaller.

  • ie serve --parallel N for MiMo-V2.6-Flash (N up to 16): lane 0 keeps --ctx, the other lanes get --slot-ctx positions (default 32,768), and --parallel 1, the default, is the previous server byte for byte, streamed responses included (ids and timings aside). The lanes draft with DFlash within a budget of 8 rows per step: up to 7 drafts for a request decoding alone or for each of two, up to 3 each for three or four (IE_MIMO26_DRAFT_BUDGET=0 drafts only for a lone request). On held-out Dream transcripts 2 requests decode at 28.38 tok/s together, 14.2 each, and 4 at 28.46, 7.1 each: ×1.25 and ×1.27 against one drafted request on the same prompts (one run each, on a busy machine). A 1.6K-token prompt arriving while three requests decode got its first token in 4.4 s (measured before the draft budget). With the CPU expert path and the drafter off, every reply equals its solo run and a cancelled request leaves the others unchanged; over 20 requests host memory grew 17 MiB after the first wave, VRAM and threads flat. Images are served at --parallel 1 only and prompt lookup is off above 1; with the drafter on, a greedy reply can change at a near-tie when other requests decode beside it; and when only a smaller lane is free, a reply can be cut at that lane's size.
  • ie serve --parallel N for DeepSeek-V4.1-Flash (N up to 16, on both cards): the same lane layout (--slot-ctx default 32,768, out of the static expert tier), and --parallel 1, the default, is the previous server byte for byte. Prompt lookup works per request, as with one lane. 2 requests decode at 24.92 tok/s together, 12.46 each, against 14.81 / 13.99 for one request (×1.73); more lanes add room and little speed (22.6 tok/s at 4, 25.7 at 16). A 1.7K-token prompt arriving while two requests decode got its first token in 7.9 s, against 7.1 s alone; over 20 requests host memory grew 10 MiB after the first wave, VRAM 10 MiB per card, threads flat at 119. With the CPU expert path off, every reply equals its solo run, prompt lookup on or off, and a cancelled request leaves the others unchanged. Images work with several lanes: a prompt with images prefills in its own turn while the other requests wait. With more than one lane the load refuses DSpark (IE_DS41_SPEC=1), IE_DS41_PROFILE_OUT, IE_DS41_DUMP_ROUTING, expert parallel (IE_DS41_EP) and a single card; the lanes were measured with --ctx 32768. The same lanes run in the ie-ds41-lanes-test tool too: two lanes ×1.33–1.58 against one request with plain decoding.
  • GLM-5.3-Flash: request lanes and a two-card pipe, in a test tool (ie-glm-lanes-test), off by default: two lanes decode at 22.26 tok/s together against 13.28 for one request (×1.676, in the configuration ie serve uses), byte-identical to solo with the CPU expert path off. ie serve runs GLM one request at a time in this release.
  • ie serve --parallel N for Qwen3.8-Flash (N up to 16, on both cards), the first model on a shared request-lanes module: each lane holds its own attention, indexer and DeltaNet state, taken out of the expert cache (about 0.55 GiB per card per extra lane at 32K), and --parallel 1, the default, is the previous server byte for byte. At --parallel 2 every request, sampled ones included, streamed the same bytes as at --parallel 1. 2 requests decode at 59.0–62.6 tok/s together against 29.0–31.9 for one (×1.94–2.06, two runs, base model, short prompts), and with row batching 16 at 118.4 (table above); over 16 requests after the first four, host memory grew 1 MiB, VRAM and threads flat. Images are served at --parallel 1 only, and while a long prompt prefills beside them the other requests slow sharply or pause. IE_Q4E_LANES=0 restores the old time-sliced path.
  • ie serve --parallel N for Qwen3.6-35B-A3B-class models (qwen35moe, split over both cards, N up to 16) on the same module: 0.34 GiB per card per extra lane at 32K, and --parallel 1 is the previous server byte for byte. At --parallel 2, sampled, tool-call, thinking, follow-up and 8K-token requests all equal --parallel 1. Before row batching, 2 requests decoded at 117.09 tok/s together against about 61 for one (×1.91), 3 at 117.24 (×1.92); with it, 16 decode at 310.2 (table above). A card holds about 2.8 GiB more VRAM at --parallel 2 before any request (the lane plus a prefill workspace sized up front). Measured on a Q8_0 fine-tune of Qwen3.6-35B-A3B; this split has no vision path. IE_Q35MOE_LANES=0 restores the one-at-a-time path.
  • Qwen3.8-27B now serves --parallel N (up to 16) on the shared lanes module with row batching: 16 requests at 108.0 tok/s together against 17.1 alone, where the previous joint-step path reached 63.4, every reply byte-identical to the same request alone. IE_QWEN35_LANES=0 restores the joint-step path, which on the new driver stack made 32.28 tok/s with 2 requests against 17.32 for one (×1.86, September 26); on it a long prompt held up the other request while it prefilled.
  • Images at --parallel > 1: where the lanes cannot take images (MiMo-V2.6-Flash, Qwen3.8-Flash, and DeepSeek-V4-Flash with its vision sidecar), /props says vision is not ready and an image request gets HTTP 400 vision_not_ready instead of failing later. The DeepSeek-V4-Flash case was verified with host tests and a code probe, not with the model loaded.

MiMo-V2.6-Flash (new since v0.1.0)

  • Running on its release day from the safetensors directory in ie serve and ie-mimo26-run: XMX prefill and split-K decode kernels for its 192/128 head sizes with per-head sinks, FP8 dense weights kept FP8 on the card, VRAM / pinned host RAM / NVMe expert tiers with a CPU expert path, the chat template and XML tool calls in C++. Prefill 429–472 tok/s on 4–6K prompts, 434–438 at 32K, 322 at 120K; wikitext-2 perplexity 3.4749 against llama.cpp's 3.4794 on the same checkpoint.
  • The bundled DFlash drafter, on by default: 23.6 tok/s on held-out Dream prompts against 16.2 plain (1.46×); IE_MIMO26_DFLASH=0 turns it off.
  • Native vision from the checkpoint's own tower on the GPU, streamed through a 766 MiB block that is borrowed per encode; within 5.4e-3 relative L2 of an fp32 reference of the module; about 1.3 % of decode time to keep the block free. Images as OpenAI image_url parts, /props vision readiness, HTTP 400 invalid_image for a bad image.
  • Conversations kept in host memory: a 123.7K-token conversation resumed after ten side requests in 1.7 s instead of 415.3 s.
  • A prompt-end snapshot: after a cut or discarded reply, 3.9 s instead of 44.8 s to the first token at 16K tokens, bit-identical to a fresh read.
  • Tool calls repaired at temperature 1 (unterminated parameters, stray quotes around enum values), each call parsed on its own and every repair logged; the host sampler sorts only the top-p nucleus: 12 ms to 1.8 ms per sampled row, same picks.
  • Vital signs, opt-in ("ie_vitals": true on a streamed request): entropy, margin and drafter hits every 16 tokens and a summary at the end; without the field the stream is byte for byte unchanged; about 0.1 ms per sampled row when on.
  • The fp16 SwiGLU overflow in the expert tier is rescaled, and a non-finite sampled logit fails the request instead of producing garbage.

DeepSeek-V4.1-Flash

  • The disk prompt cache survives rebuilds: entries are keyed by a manifest of the numerics sources (including the vision path), checked by a unit test, not by the executable. The first turn after a rebuild restored in 1.4 s where the cold prefill took 79 s.
  • The CPU expert path serves prompt chunks too, so continuation prefill is faster; engram tables are read with random-access advice again.
  • Per-request memory growth cut in ie serve (glibc's mmap threshold pinned) and a per-reply expert-cache event leak fixed.

Recommended sampling per model

  • Requests that omit sampling fields now use each model's published recommendation for thinking or instruct mode, so sampled replies differ from v0.1; pass the fields, or set --temp/--top-k/... or IE_SERVE_*, to keep the old 0.7/40/0.95. The fields are temperature, top_p, top_k, min_p, presence_penalty and the repeat penalty; the order is the request's value, then a command-line flag or IE_SERVE_*, then the model card's value for the request's mode, then the library default. Greedy requests (a temperature of 0 from the request or the server) and requests that set all six fields are unchanged. max_tokens and frequency_penalty are not part of it. A command-line flag or IE_SERVE_* value applies in both modes.
model thinking instruct
Qwen3.8-27B, Qwen3.8-Flash temperature 1.0, top_p 0.95, top_k 20 0.7, 0.80, top_k 20, presence 1.5
the Qwen3.8-35B-A3B distill 0.6, 0.95, top_k 20 0.7, 0.80, top_k 20, presence 1.5
MiMo-V2.6-Flash 1.0, 0.95 the same
GLM-5.3-Flash 1.0, 0.95 no official instruct mode
DeepSeek-V4.1-Flash 1.0, 0.95 the same (its "Chat" mode)

Fields a card leaves out are switched off (top_k 0, min_p 0, presence 0, repeat 1.0). Models without a card here keep the library defaults.

  • chat_template_kwargs: enable_thinking and reasoning_effort are accepted there, in the form the Qwen cards use, as aliases of the top-level fields; the top-level field wins where both are given, and a wrong type is an HTTP 400.
  • ie capabilities reports a recommended block: per mode the values, a when-to-use note and the maximum output, plus the effort levels, the context, a note and the card link ("recommended": null for a model without a card). Its defaults now report the values a request that omits a field gets in the default mode, instead of fixed library values.
  • DeepSeek-V4.1 and MiMo-V2.6 penalties: their host sampler has no presence or frequency penalty, so a nonzero value is served without it and the server logs a WARNING (the repeat penalty works).
  • scripts/hermes-engine no longer wipes its own prompt-cache caps at --parallel > 1, and no longer pins the Qwen sampling (the engine now carries the per-mode values).

The server

  • A lost device ends the process with exit code 75, so a supervisor sees it; IE_EXIT_ON_DEVICE_LOST=0 keeps the old behaviour (stay up, answer 503).
  • An idle-spin watchdog names the busy threads in the log and in /health if the process burns CPU with nothing in flight; 20 of 20 client disconnects mid-generation left the server idle.
  • Teardown drains every in-flight copy before freeing memory, and no longer throws when a device was lost mid-drain.

A newer Intel driver stack

  • Released on Intel compute-runtime 26.35, IGC 2.41.5 and GuC firmware 70.65 (from compute-runtime 26.22, IGC 2.33 and GuC 70.54). A SYCL context that holds both cards no longer costs host RAM for device memory: allocating 8 GiB on one card through such a context now leaves available host RAM unchanged, where it used to drop by 8.03 GiB. The engine already worked around that mirror with one context per card.
  • Output quality is unchanged: DeepSeek-V4.1 perplexity 1.88767 against 1.88776 on the old stack over 16,383 tokens (0.04 standard errors apart), MiMo-V2.6 wikitext-2 3.4734 against 3.4749. Greedy replies can differ from the old stack's at near-ties (new code generation). No GPU faults over the trial; the card firmware (IFWI) was checked, not changed.
  • Qwen3.8-27B on the new stack: the same binary gives byte-identical text on both stacks; perplexity 5.3419; ie serve decodes at 17.0–17.2 tok/s plain and 24.5–27.8 with --spec. --spec passes its 5-prompt byte-identity certification (twice, deterministically) but is not byte-identical to plain greedy decoding on every prompt: on one of two longer served prompts it picked a different near-tie token.

Measured and closed

  • A restricted draft vocabulary for MiMo's drafter: the drafter's LM head is 0.74 % of decode time and a 64K-token subset would leave 2.6 % of generated tokens impossible to draft, so it is left out of this release.

The models it is tuned for

model prefill decode
MiMo-V2.6-Flash (178 GB) 438–483 tok/s at 4–6K; 322 at 120K (September 22) 23.1–23.9 chat, 16.7–16.9 without the drafter; 28.38 / 28.46 together with ie serve --parallel 2 / 4; ×1.66–1.69 with 2 lanes, ×1.85–1.90 with 4 (plain); the lanes figures September 26
DeepSeek-V4.1-Flash (475 GB) 304 tok/s at 2K, 319 at 32K–223K (September 16–17) 12.8 chat, 14.0 in agent loops (September 17–18); 25.1–25.8 together with ie serve --parallel 2 (×1.68–1.69)
DeepSeek-V4-Flash (155 GB) 571 tok/s at 4K 26.1 at 4K, 32.7 short (measured on the previous driver stack, September 11, 2026)
GLM-5.3-Flash 156.4 tok/s at 16K 14.0–14.4 at 16K; ×1.68 with 2 lanes in the test tool
Qwen3.8-Flash (104 GB) 495–502 tok/s pipelined, once warm 37.1–38.2 chat, 44.4–44.6 code; 59.0–62.6 together with ie serve --parallel 2 (×1.94–2.06)
Qwen3.8-27B (Q8_0) 945 tok/s at 2K (August 15) 24.5 tensor-parallel + speculative (August 26); 32.28 together with ie serve --parallel 2

Figures without a date were measured on the driver stack this release ships on (September 26–27, 2026); dated ones are from the previous driver stack. Conditions and methods are in the README's Benchmarks section. These come from one machine; your hardware will differ.

Upgrading from v0.1.0

  • Exit on device loss. ie serve now exits (code 75) when a device is lost instead of staying up unhealthy. Set IE_EXIT_ON_DEVICE_LOST=0 to keep the old behaviour.
  • DeepSeek-V4.1 disk prompt cache. Entries written by an older build are ignored once, because the key changed, and are rebuilt on first use.
  • Served names. A layout that names a server reports that name as the model id, with "root" set to the model file's id; ie serve <model> without a layout reports the file's id as before.
  • Card locks. --cards (or a layout's cards) and ie supervise take per-card locks under /tmp (IE_CARD_LOCK_DIR moves them). ie serve without --cards takes none. scripts/ie-run-guarded --cards LIST locks only those cards.
  • MiMo-V2.6-Flash defaults. The DFlash drafter is on and prompt lookup is off while it is (IE_MIMO26_DFLASH=0 restores lookup).
  • --parallel N on MiMo-V2.6-Flash and DeepSeek-V4.1-Flash now loads N request lanes instead of being held at 1: every lane after the first takes its caches out of the static expert tier (on MiMo about 1.34 GB across both cards per extra lane at 32K), and a load whose lanes do not fit is refused with the numbers. On V4.1, more than one lane needs both cards and is refused together with DSpark (IE_DS41_SPEC=1), IE_DS41_PROFILE_OUT, IE_DS41_DUMP_ROUTING or expert parallel (IE_DS41_EP). --parallel 1, the default, is unchanged. On GLM-5.3-Flash requests still run one after another.
  • --parallel N on Qwen3.8-Flash and Qwen3.6-35B-A3B-class models on two cards now loads N request lanes (their memory comes out of the expert cache on Qwen3.8-Flash, and out of free VRAM on the 35B-A3B class) instead of time-slicing or queueing whole requests. IE_Q4E_LANES=0 and IE_Q35MOE_LANES=0 restore the old paths; IE_P2P with Qwen3.8-Flash lanes is refused at load.
  • --parallel N goes up to 16 (was 4), and on Qwen3.8-Flash, the 35B-A3B class and Qwen3.8-27B the decoding lanes are row-batched by default (IE_Q4E_ROWS=0, IE_Q35MOE_ROWS=0, IE_QWEN35_ROWS=0 turn it off). The Qwen3.8-27B split now serves --parallel N on request lanes instead of its joint-step path (IE_QWEN35_LANES=0 restores it), and the 35B-A3B class and Qwen3.8-Flash cache a shared system prompt and tools once (IE_SHARED_PREFIX=0 turns it off; on the 27B it is opt-in, IE_QWEN35_SHARED_PREFIX=1). At --parallel > 1 MiMo-V2.6-Flash no longer reuses a cached prefix under 1,024 tokens. With the same sampling settings, --parallel 1 output is unchanged on all five models (checked with the CPU expert path off on MiMo and V4.1): that holds for greedy requests and for requests that set every sampling field, while a sampled request that omits fields now gets the model's recommended values (above).
  • Sampling defaults follow the model card. Requests that omit sampling fields now use each model's published recommendation for thinking or instruct mode, so sampled replies differ from v0.1; pass the fields, or set --temp/--top-k/... or IE_SERVE_*, to keep the old 0.7/40/0.95. Greedy requests and requests that set every field are unchanged (see Recommended sampling per model, above).
  • Images at --parallel > 1 on MiMo-V2.6-Flash, Qwen3.8-Flash and DeepSeek-V4-Flash get HTTP 400 vision_not_ready before admission, and /props reports vision not ready with the reason.
  • Intel compute-runtime 26.35 serves small device-to-host copies into pageable host memory through CPU reads of the card's memory, which made MiMo-V2.6 and DeepSeek-V4.1 decode with the CPU expert path on 11–20 % slower. This release stages those copies in pinned host memory, with byte-identical output.
  • ie serve --help describes --parallel (1 to 16) and --slot-ctx as the MiMo-V2.6, DeepSeek-V4.1, Qwen3.8-Flash, 35B-A3B-class and Qwen3.8-27B lanes use them, with the switches that turn each off.

Requirements

Intel oneAPI 2026.x and an Intel Arc GPU — most of the engine only runs on one. The largest configurations want about 256 GB of system RAM. Measured on Intel compute-runtime 26.35.39758.10, IGC 2.41.5 and GuC firmware 70.65.0 with the kernel's xe driver; figures marked as from the previous stack used compute-runtime 26.22.

Getting started

See the From source section of the README; ie <command> --help lists every option. QUICKSTART.md has the container route; the prebuilt image dates from July 2026 and predates this release's models and features, so build from source for them. The Dream Agent Harness v0.2.0 drives this engine as a local agent, including ie supervise layouts and a model per role.

Known limitations

  • Verified on Arc Pro B70. Other Arc cards are untested by the maintainer.
  • Large-model configurations assume substantial host RAM — the 475 GB checkpoint wants around 256 GB.
  • Request lanes in ie serve (--parallel N, up to 16) cover MiMo-V2.6-Flash, DeepSeek-V4.1-Flash, Qwen3.8-Flash, Qwen3.6-35B-A3B-class models and Qwen3.8-27B. A long new prompt's prefill pauses the decoding lanes until it ends (IE_LANES_PREFILL_FIFO=1 shortens the pauses at the cost of later first tokens). With the drafter on (the default), MiMo can give a different reply beside other requests than alone on some sampled or thinking prompts: 1 of 16 requests at 16 lanes, and 2 of the release gate's prompts at 4 lanes; with the drafter off (IE_MIMO26_DFLASH=0) every check passes. On DeepSeek-V4.1 at --parallel > 1 a follow-up turn reuses a lane's cached prefix and can differ from the same turn read cold at --parallel 1 (pre-existing: the previous build gives the same bytes; batch equals solo within one server). MiMo and DeepSeek-V4.1 scale little past two lanes: the CPU expert leg, not the cards, bounds them. MiMo and Qwen3.8-Flash serve images at --parallel 1 only; on V4.1 a prompt with images prefills in its own turn while the other requests wait. The 16-lane figures come from one run per N on short prompts; the 35B-A3B-class lanes were measured on a community fine-tune. GLM-5.3-Flash has request lanes in a test tool; ie serve runs it one request at a time.
  • Qwen3.8-27B --spec passes its byte-identity certification but can pick a different near-tie token than plain greedy decoding on some prompts.
  • A card on this test machine has, more than once, fallen into a copy-engine reset loop that only a reboot clears. It is a known Xe driver/firmware issue class, reported upstream (intel/compute-runtime #948, drm/xe #8390), with no driver release known to fix it, including the 26.35 stack this release uses; ie serve now exits on the lost device instead of hanging.
  • GLM-5.2 and Tencent Hy4-preview have experimental standalone runners rather than the main path.
  • This is a 0.x release: interfaces may change.

Full changelog: v0.1.0...v0.2.0

Assets 2
Loading