Unofficial Intel XPU Optimization Lab

Community setup guides, benchmark recipes, troubleshooting notes, and patches for local AI work on Intel XPUs. This is not a single-model repo. It is a working lab notebook and reproducibility collection for multiple model lanes that we revisit as new runtime, compiler, and kernel ideas appear.

This is experimental research, not a supported product. Commands, patches, and benchmark observations are provided under the repository LICENSE and the risks described in DISCLAIMER.md; review and use them at your own risk.

• Local AI users who want reproducible Intel Arc/B-series commands, patches and benchmark recipes.
• Anyone deciding whether Intel Arc/B-series hardware is worth it for local inference, and wanting to see the state of support.
• Optimization agents and contributors who need a map of current work, archived lessons, and validity rules.
• Upstream vLLM, llama.cpp, oneAPI, SYCL, and Intel/XPU developers looking for concrete repros and failure signatures.

The repo is organized around model lanes, not one-off leaderboard rows. A serious lane should leave behind:

• results/<model>-<quant>-<hardware>/: promoted or closed-out result packet, validity gates, best commands, invalid fast lanes, and lessons.
• repro/<model>-.../: classified reproduction artifacts, ranging from portable candidates to originating-host lab replays and archived records; the catalog, not the directory name, states portability.
• packages/<model>-.../: user-facing, machine-readable front doors over exact in-repository guides, commands, dependencies, and evidence.
• experiments/<model>-.../: active research lanes that are not production recipes yet.
• notes/: chronological lab notebook entries, including negative results and postmortems.
• patches/: patch snapshots and source/config deltas, including failed experiments worth preserving.
• data/: compact structured benchmark records, payloads, responses, and logs.
• scripts/: reusable harnesses, analyzers, launchers, and submission helpers.
• model-intake/: revision-pinned candidate catalog, storage budget, and the boundary between discovery, download, validation, and promotion.
• community/<contributor>-...: runnable work contributed from outside the reference lab, with explicit evidence status and validation history.
• community/field-reports/: unverified measurements and observations from community systems, kept separate from recipes, patches, validation assets, and promoted results. See community/README.md.

The point is to make model switching cheap. Gemma, Qwen, MiniMax, and future lanes should all share validation discipline, result-packet shape, and reusable kernel/runtime lessons without dragging stale generated builds or huge artifacts forward.

The official 27B FP8 quickstart now selects one current runtime and a one-user MTP1 setup with 32K input capacity. Its helper provides persistent start, status, and owned stop commands. A clean public-source replay passed 12/12 reference outputs, six practical requests with exact repeats, and clean stop. Historical higher-draft speed records keep their original labels.

Official 27B FP8 reading speed now includes a 2,048-token prompt measurement and a final bounded profiling review. Results and evidence cover two B70s, one user and fixed MTP1; no new optimization was promoted.

A bounded AMD-to-B70 transfer test adds 512/2K/16K reading measurements for the recommended 27B FP8 setup. The exact projection-dispatch candidate was neutral. The newer-runtime/V2/ DFlash2 startup froze the host before inference; the user rebooted it and the original qualified service was restored. No new speedup is claimed. The MTP-only source follow-up excludes DFlash and ranks the remaining exactness-preserving ideas. The implementation follow-up built metadata and exact communication candidates. The communication attempt failed exactness and recorded GPU faults; no speedup or new default was qualified.

CURRENT.md alone owns the live service and active research state. These are evidence-backed examples; the broader expected-performance view is the performance index.

Additional 512-token prefill measurements now cover 4B/9B on two B70s and 27B INT4 on one B70. Results and optimization findings keep reading speed separate from decode and HTTP first-token wait.

This table is generated from the package manifests. It is a synchronization view, not a ranking: workloads, context lengths, card counts, quantization, and speculative-decoding modes differ. Open the detail page for the exact measurement scope and evidence.

The optional R308 single-request boundary repair is available for Qwen3.5 4B and 9B. Both models passed fresh-server boundary and strict identity gates; the qualification note defines its one-card, fixed-depth scope.

These are entry points, not the whole repo:

For the full queue and archive, use docs/model-effort-index.md.

Last audited 2026-08-30. Qwen3.8 retains Qwen3.6's exact 64-layer, three-GDN-to-one-full-attention tensor geometry, so the accepted exact-shape TP2 stack transfers mechanically. Every new weight set is still independently gated. Every row states whether it is target-only or speculative; promoted measurements are cache-zero. Other B70 hosts and agents should start with the multi-host handoff and do-not-repeat index.

Community-reported alternatives are kept outside the promoted rows above:

Measured 2026-08-22 on one Arc Pro B70 with the accepted llama.cpp SYCL TP1 lane build, Q4_K_M target-only, flash-attn on, llama-bench pp2048 + tg128, 5 reps per depth. These are raw-engine rates for the decode/prefill shape and the KV-dtype delta - not the promoted 27.83 tok/s class-balanced realistic-suite metric (depth-0 tg128 here is 24.81; different harness, both real). Raw data and chart: sweep JSON, chart SVG, finding.

KV8-vs-KV16 finding: the q8_0-KV decode penalty grows with context - ~2% at 0 ctx to -51% at 32K - because the per-token KV dequant scales with cached length. f16-KV decode stays nearly flat (24.81 -> 21.77, -12% to 32K). Prefill is KV-dtype-independent (<1.5% at every depth). Practical rule: keep KV at f16 for speed on this lane; use q8_0 KV only to fit longer context in 32 GiB, accepting a large long-context decode hit. (The Reddit vLLM report ran fp8 KV by default on the newer 0.27.1 XPU path. Our own vLLM TP1 is now unblocked on the XPU nightly image - see the matrix below; there, fp8_e4m3 KV measured decode-neutral at short context but output-divergent, and fp8_e5m2 is refused outright by FlashAttention on this device.)

Same lane, same config, f16 KV, three weight files. Decode tg128 (tok/s):

Counter-intuitive result: decode speed is inverse to file size here - the smallest file (UD-Q4_K_XL, 16.35 GiB) is the slowest decoder, and the largest (plain Q4_K_M) is the fastest at every depth. Decode is therefore not purely bandwidth-bound on this build: it is Q4_K-tuned (Q4_K reorder + MMVQ), so uniform Q4_K_M hits the fast path while the unsloth UD mixed-precision dynamic quants do not. This ordering is a property of the tuned lane build, not a universal ranking of the formats. Takeaway: on this lane, prefer Q4_K_M for speed (it is also the quality-validated promoted lane); pick a UD quant only for its quality/size, accepting slower decode here. data, chart, finding.

Measured 2026-08-22/23 on one B70 (GPU0), same 25-prompt realistic suite and conventional metric as the promoted lanes, cache-zero gated, prefix caching off, --max-num-seqs 1. Image: vllm-openai-xpu nightly e9d1398d9 (dated 11 calendar days after v0.27.1 but not its descendant, torch 2.13+xpu). The pinned 0.20.2rc1 TP1 crashes do not reproduce here; container bring-up needs CCL_ZE_IPC_EXCHANGE=sockets + a read-only /dev/dri/by-path mount. The full quality battery (code canary 14, repeats, 8K needle) passed on both certified configs: MTP-off f16 graph-off AND graph-on. Those battery runs did not supply --baseline-json, so their baseline_match_all=true compatibility field is vacuous and is not used as evidence of oracle parity.

TP scaling on the same container (2026-08-23, same suite/metric, characterization-only): eager multi-GPU decode is FLAT (~17 tok/s at both TP2 and TP4, below single-card - per-step container collective tax), while XPU graph restores scaling: 30.2 / 48.8 / 71.7 tok/s at TP1 / TP2 / TP4. The 71.7 TP4 row is the fastest target-only Qwen3.8 result for this AutoRound/nightly identity, not the lab-wide target-only record. All three graph configs passed the objective battery, but multi-GPU XPU Graph is explicitly unsupported/experimental in this vLLM build. The only TP4 MTP2 probe is infrastructure-invalid: concurrent ranks lost shared Triton-cache artifacts, and the later shm_broadcast starvation was a downstream symptom. It does not establish a TP>1 speculative-decode deadlock. Prefill scales even in eager (281 -> ~500 -> ~860 tok/s); TTFT drops to 0.09 s at TP4. TP3 is architecturally impossible (16 GDN K heads % 3). TP4 needs gpu-memory-utilization <= 0.6 (XPU single-allocation cap). TP-scale data, TP-scale finding.

Key findings: XPU graph (VLLM_XPU_ENABLE_XPU_GRAPH=1, default off on the nightly) is worth +25% MTP-off and is output-faithful. MTP at TP1 works and is output-faithful with excellent acceptance, but the verify step costs ~190-200 ms per extra verify token (~4-5x a whole MTP-off step), so net decode collapses ~5x and deeper drafts cannot amortize it - opposite to the community GPTQ result, pointing at the INC/AutoRound W4A16 small-batch verify path or GDN serial verify. Graph + MTP corrupts outputs (0% acceptance and 0/25 oracle match) - quarantined. The lane is deterministic within a boot but not across boots (autotuned kernel selection), so it makes no cross-boot token-exactness claim yet. data, finding, driver.

Plain-language explainers for these results live in the Learn library: Quantization vs decode speed, KV cache precision, Context length, The MTP ladder.

Last audited 2026-08-15. These rows share a model family, not a quality, runtime, or benchmark class. “Target only” means no speculative draft; MTP and DFlash rows retain the declared target as verifier. The first number is the repository's historical published 100-event/99-interval rate where that harness was used; the parenthesized value is conventional 99-interval accounting from the same timestamps. Relative A/B gains are unchanged.

The official BF16 target was not downloaded in the current campaign. At 55.586 GB of weights, two 608 GB/s B70s have a target-only one-weight-read roofline of about 21.9 tok/s before collective and runtime overhead, so a plain lossless TP2 goal above 30 tok/s is not physically credible. The Q8 target-only TP2 lane remains active toward the 40 tok/s stretch goal; speculative rows stay separate.

Diagnostic runs are allowed and useful, but headline records require the model-lane gate. For Gemma/Qwen-style fresh-response records, that means:

• fixed realistic prompt suite;
• each prompt run once as a cold response;
• cached_tokens=0 for every request;
• no prompt/KV cache reuse, context checkpoints, response reuse, warmed repeated prompts, or n-gram/history acceleration;
• target model and quantization unchanged;
• speculative decoding/MTP allowed only when accepted tokens are verified by the declared target model;
• primary metric is the median conventional rate across the 99 inter-token intervals between timestamps 1 and 100 after TTFT, with p10, mean, TTFT, wall-clock full-output throughput, full-output after-TTFT throughput, hashes, runtime identity, env vars, flags, and logs. Historical 100-event/99-interval compatibility fields must be labeled as such.

Synthetic, repeated, warmed, cached, or history-assisted rows stay diagnostic unless revalidated by the lane's promotion gate. This matters because several very fast historical rows were useful optimization clues but not real-world fresh-response claims.

The measuring host has four Intel Arc Pro B70 32 GB cards and about 125 GiB system RAM. A second host has two ASRock Arc Pro B70 32 GB cards (64 GB aggregate VRAM), full-size ReBAR, PCIe 5.0 x16 links, and only about 15 GiB system RAM; it is currently restricted to source/build/op-level work. Hardware, count, and host identity remain part of every result. B70 is the platform on which maintainers can independently reproduce and verify submitted patches, but this is an Intel Arc/XPU project, not a B70-only repository.

Results, fixes, and portability reports from Intel Arc Pro B50, B60, B65, and B70 owners are welcome, as are useful observations from other Intel Arc and XPU systems. A B70 rerun verifies what a patch does on B70; it does not certify a contributor's score on hardware the maintainers do not possess. Hardware and verification status therefore stay explicit in every promoted result.

The four-card host enables TP4 or independent one-/two-GPU screens. The two-card host has enough VRAM for TP2 but not enough system RAM for the current Qwen3.8 AutoRound server, so GPU count alone is not a safety signal.

Intel specifies B65 and B70 with the same 32-GiB VRAM capacity and 608 GB/s memory bandwidth, but B65 has 20 versus 32 Xe cores and 160 versus 256 XMX engines. A community report relayed from boyter claims roughly 42 tok/s with MTP, 22 tok/s without MTP, and 19 tok/s at a 150-W limit on one B65. The command, prompt, metric, and exact power/MTP identity were not captured, so these are purchasing-orientation anecdotes rather than verified comparisons. Also note that this lab's 49.717503 tok/s Qwen3.8 Q4_K_M result uses two B70s, not one. See the B65 field report.

Steve Seguin maintains this repo and posts ongoing build notes at https://x.com/xyster.

This remains primarily a working optimization lab, but careful outside contributions are welcome. Start with CONTRIBUTING.md and the manual verification policy. No result needs to be a record to be useful.

Useful contributions include:

• reproducing a result on another Intel/XPU stack and sharing exact versions;
• testing a failed lane after a driver, PyTorch XPU, vLLM, llama.cpp, or oneAPI update;
• turning local patches into clean upstream issues or PRs;
• adding quality canaries for new model families;
• sharing high-signal failure logs with model, quantization, graph mode, and hardware identity intact;
• providing temporary access to larger Intel hardware for models that do not fit cleanly on 32 GB cards.

When opening an issue or discussion, include GPU, OS, model, quantization, runtime, exact command, benchmark shape, quality gate, result JSON/log paths, and what changed from the closest known-good run.

The durable value of this lab is not only its fastest rows. Start with the cross-model pattern catalog for evidence-linked strategies that transfer across model lanes, and the model optimization guide for the identity, quality, A/B, variance, and negative-result process used to test them. Model-specific packets retain the commands and caveats behind each lesson.

The former top-level chronology was preserved verbatim in notes/2026-07-11-readme-historical-b70-archive.md. It remains useful for chronology and link recovery, but maintained navigation now lives in the result packets, model-effort index, reproduction recipes, and optimization guides.

Use the historical archive for the original long-form findings. Current model-specific starting points are the results index, including the MiniMax M2.7 packet, and the cross-model effort index. Underlying notes, data, patches, experiments, and reproduction paths were intentionally left in place.

Use the docs index for maintained navigation and the repository organization above for placement rules. The former file-by-file inventory is retained in the historical archive; it is not duplicated here because detailed inventories become stale quickly.