Squeezing Every Token/s from the Intel Arc Pro B70

[JohnTDI-cpu]

Testing llama.cpp settings on an Intel Arc Pro B70

A log of configuration variants tried on one B70, using stock llama.cpp at
master 63b64a50a — no patches or forks, only build flags, command line flags
and environment variables.

Much of the ground here is already covered by others; links are in the last
section. This adds a dense 27B Q8_0 model with built-in multi-token prediction,
plus a few variants that did not seem to have numbers published yet.

What mattered most, in order:

build with -DGGML_SYCL_F16=ON	3.72x prefill — it is OFF by default in CMake
--spec-type draft-mtp	2.1x–3.2x generation, depending on the answer
-ub 2048	+35% prefill
-ctk q4_0 -ctv q4_0	full 131K context on one 32 GB card
-fa 1	+6% prefill
never set GGML_SYCL_ENABLE_OPT=0	it costs 68% of generation

What the numbers mean

If you have not benchmarked llama.cpp before:

• Prefill (also prompt processing, pp) is reading your prompt. pp2048
  means a 2048-token prompt. This is what you wait for before the first word
  appears.
• Generation (also decode, tg) is writing the answer, one token at a
  time. tg128 means 128 generated tokens. This is the speed you watch while it
  types.
• t/s is tokens per second — higher is better in both cases.
• Prefill is limited by arithmetic, generation by memory bandwidth, which is why
  a setting can help one and do nothing for the other.
• MTP (multi-token prediction) lets the model guess several tokens ahead and
  check them in one pass. Draft acceptance is the fraction of those guesses
  that turn out right; the higher it is, the more time the trick saves.

Setup

GPU	Intel Arc Pro B70, 32 GB, Battlemage (Xe2), reports as BMG G31
Driver	xe, kernel 7.0.0-28-generic, Level Zero 1.15.39122
Toolchain	oneAPI 2026.1, icx/icpx
Model	Qwen3.8-27B-Q8_0, 27.04 GiB, 27.32 B parameters, arch qwen35
	65 blocks — 48 gated-delta-net, 16 attention (GQA 24/4, head dim 256)
	nextn_predict_layers = 1: MTP is part of the model, Q8 like the rest of it
Baseline	-ngl 99 -b 4096 -ub 2048 -fa 1, 2–3 repetitions, stddev under 0.5%

Build flags

build	pp2048	pp8192	tg128
-DGGML_SYCL_F16=OFF (CMake default)	389.2	385.3	15.80
-DGGML_SYCL_F16=ON	1446.8	1388.2	15.79

3.72x on prefill from one flag that defaults to OFF, with generation
untouched — which is why it is easy to miss if you only watch decode. The
official Intel container sets it; source builds do not.

cmake -B build -DGGML_SYCL=ON -DGGML_SYCL_TARGET=INTEL \
      -DGGML_SYCL_F16=ON -DGGML_SYCL_DEVICE_ARCH=bmg_g21 \
      -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx

-DGGML_SYCL_DEVICE_ARCH=bmg_g21 compiles ahead of time and removes the JIT
stall on first inference. The card reports itself as BMG G31; g21 still works.

Speculative decoding (MTP)

The model carries its multi-token-prediction layer in the file, so
--spec-type draft-mtp needs no extra download and no separate draft model.

Throughput is governed by draft acceptance, and acceptance depends on how
predictable the answer is. Both columns are natural generation, greedy,
-c 4096:

--spec-draft-n-max	short answer (22 tokens)	acceptance	long answer (512 tokens)	acceptance
off	15.73	—	15.67	—
1	26.69	0.909	25.99	0.868
2	33.52	0.812	30.62	0.701
3 (default)	41.35	0.833	33.53	0.615
4	46.11	0.800	33.23	0.507
6	50.92	0.708	32.50	0.395
8	server failed to start at -c 4096

Short answers reach 50.9 t/s, 3.2x over plain generation. Long ones peak at
33.5, 2.1x — and they peak at a different draft length.

The optimum is not one value. A short reply keeps improving to n-max 6,
because acceptance stays high enough that the extra drafted tokens pay for
themselves. A long technical passage gets steadily harder to guess — acceptance
falls from 0.87 to 0.40 across the sweep — and peaks at n-max 3.

Serving chat, use 6. Generating long documents, 3 is enough.

--spec-draft-device, --spec-draft-ngl and --backend-sampling made no
measurable difference, so the MTP layer reaches the GPU without being told.

Micro-batch

-b	-ub	pp2048	pp8192
2048	512	1074.8	1032.3
2048	1024	1304.7	1253.6
2048	2048	1448.7	1390.1
4096	512	1066.1	1030.6
4096	1024	1302.4	1258.2
4096	2048	1444.8	1388.5
4096	4096	1445.1	1416.0

512 → 2048 is +34.8%; -b makes almost no difference as long as it is at
least -ub. For prompts of 8K and up, -ub 4096 adds another 2%. Below about
1024 tokens of prompt the flag does nothing, because there is nothing to fill
the micro-batch with.

A published figure for a dense model on the B60 is +8%; this card and this model
give considerably more.

Flash attention

	pp2048	pp8192	tg128
-fa 0	1361.6	1270.3	15.81
-fa 1	1446.8	1388.2	15.79

Worth +6% at a 2048-token prompt and +9% at 8192 — the longer the prompt, the
more it helps. Generation is unaffected. The default is auto; make it explicit.

KV cache and context length

KV type	largest context tested	prefill there
f16	49152	1115.7
q8_0	65536	1032.6
q4_0	131072 (the model's full context)	795.3

These are the sizes that were run, not measured ceilings — f16 was not pushed
past 49152, and q8_0 not past 65536.

Cost at short context is small — tg128 is 15.79 / 15.72 / 15.69 for
f16 / q8_0 / q4_0, prefill unchanged. A 27B Q8 model does reach its full 128K
context on a single 32 GB card with q4_0 KV.

Others report q4_0 KV costing roughly half of generation speed past 16K
context. Deep-context generation was not measured here, so treat the cheapness
above as applying to short contexts only.

Environment variables

Against the same baseline (pp2048 1438.3, tg128 15.79). Repeated runs of the
unchanged configuration land between 1434 and 1450, so treat anything within
about ±1% on prefill as noise — several rows below sit above the baseline
purely for that reason:

variable	pp2048	tg128	note
GGML_SYCL_ENABLE_OPT=0	1442.4	5.07	−68% generation
GGML_SYCL_PRIORITIZE_DMMV=1	1448.7	13.86	−12% generation
GGML_SYCL_FA_ONEDNN=0	1372.1	16.34	−5% prefill, +3% generation
SYCL_UR_USE_LEVEL_ZERO_V2=0	1450.2	16.01	+1.4% generation, free
both of the above	1372.2	16.59	they compose
GGML_SYCL_ENABLE_FUSION=0	1401.0	15.52	worse both ways
GGML_SYCL_USM_SYSTEM=1	1434.1	15.82	−0.3%
GGML_SYCL_ENABLE_DNN=0	1437.9	15.79	noise
GGML_SYCL_ENABLE_VMM=0	1443.9	15.80	noise
GGML_SYCL_ENABLE_ESIMD=0	1447.7	15.79	no effect
GGML_SYCL_ENABLE_MKL_FA=0	1449.3	15.79	no effect
GGML_SYCL_ENABLE_GRAPH=1	1450.0	15.79	no effect
GGML_SYCL_USE_ASYNC_MEM_OP=0	1448.1	15.79	no effect
GGML_SYCL_NO_PINNED=1	1449.1	15.79	no effect

Two seem worth flagging.

GGML_SYCL_ENABLE_OPT=0 costs 68% of generation speed. That switch is the
documented workaround for issue #21893, where the reorder optimisation produces
corrupted output on this exact card. On this setup — master 63b64a50a, Level
Zero 1.15.39122, kernel 7.0.0-28 — the corruption does not reproduce: perplexity
is 6.4750 with the optimisation on and 6.4747 with the workaround. Anyone still
carrying that workaround may want to retest whether they need it.

SYCL_UR_USE_LEVEL_ZERO_V2=0 — the older Level Zero adapter is slightly
faster at generation and costs nothing at prefill.

Settings we ended up using

Long prompts:

llama-server -m model.gguf -ngl 99 -b 4096 -ub 4096 -fa 1 -ctk q4_0 -ctv q4_0 -c 131072

Chat and long answers:

GGML_SYCL_FA_ONEDNN=0 SYCL_UR_USE_LEVEL_ZERO_V2=0 \
llama-server -m model.gguf -ngl 99 -b 4096 -ub 2048 -fa 1 \
             --spec-type draft-mtp --spec-draft-n-max 6

Built with -DGGML_SYCL_F16=ON -DGGML_SYCL_DEVICE_ARCH=bmg_g21.

The two environment variables in the chat profile were measured on plain
generation, so they were rechecked with MTP on — they still help there: 51.30
against 50.69 t/s on a short answer, 32.98 against 31.95 on a long one.

Host notes

Battlemage needs the xe driver, not i915 — lsmod | grep '^xe '. Your user
needs to be in the render group.

For GPU performance counters with unitrace or VTune:

echo 'dev.xe.observation_paranoid = 0' | sudo tee /etc/sysctl.d/60-xe-perf.conf
sudo sysctl --system

Without it the profiler fails with Failed to create metric query pool (status = 0x78000004); ZET_ENABLE_METRICS=1 alone does not help. It does not
affect inference speed, and it does not survive a reboot unless written to a file.

A note on Vulkan

Vulkan was measured as an alternative and set aside: prompt processing is 2.2x
slower, because it multiplies the Q8_0 weights directly in cooperative-matrix
shaders at about 46 TFLOPS while SYCL dequantizes them to F16 and hands the GEMM
to oneDNN on the XMX units at about 174, and under speculative decoding roughly a
third of requests come back as a single end-of-sequence token where SYCL answers
all of them. Its generation speed also swings by a factor of three between driver
releases — 6.1 t/s on Mesa 26.0.8 against 17.9 on 26.1.7, with the shader
compiler version making under 0.5% either way — so any Vulkan figure is only as
good as the driver it was taken on.

Prior work covering the same ground

• PMZFX/intel-arc-pro-b70-benchmarks — B70 across many models and quantizations, single and dual card, including GGML_SYCL_F16 and GGML_SYCL_PRIORITIZE_DMMV
• Discussion #23313 — SYCL performance across Intel GPUs, GGML_SYCL_F16, flash attention, xe vs i915
• Jonathan Mann, B70 weekend testing — B70, backend comparison, ubatch, KV quantization, MTP
• Tuning batch sizes on Arc Pro B60 — -b/-ub on the B60
• Issue #21517 — Q8_0 reaching only 21–24% of memory bandwidth on B70, open
• Issue #21893 — reorder corruption on B70, the source of the GGML_SYCL_DISABLE_OPT=1 workaround

[arthw]

@JohnTDI-cpu
Thank you for your test and sharing!
It's very useful to Intel GPU user who is using llama.cpp SYCL backend.

I supply some info:

1. fp32 vs fp16.
   fp16 has better performance than fp32 in PP stage.
   But it has a little worse performance in TG stage.
   fp16 has low accuracy issue, that means fp32 has better answer.

Here is our suggestion:
User needs to test fp32/fp16 to choose the better solution by balance the performance (PP vs TG) and accuracy.
In general, fp16 could match all requirements.

We keep fp32 as default build argument to keep compatible for CI/test script.

2. ENV variables:
   All env vars' default values are best solution for common LLMs.
   That means user needn't to change them for common LLMs to get the best performance.
   But user can change them to tune for special LLM to be better.
   And developer can use them to debug the issues.

Thank you!