Performance of llama.cpp on Intel GPU with SYCL backend #23313
Replies: 28 comments 68 replies
|
compiled with cmake -B build-sycl -DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DGGML_SYCL_TARGET=INTEL -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_FLAGS="-march=znver4" -DCMAKE_CXX_FLAGS="-march=znver4" -DCMAKE_BUILD_TYPE=Release && cmake --build build-sycl --config Release -j 16 single b70 dual b70: |
|
If instead compiling and using with f16=off: cmake -B build-sycl -DGGML_SYCL=ON -DGGML_SYCL_F16=OFF -DGGML_SYCL_TARGET=INTEL -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_FLAGS="-march=znver4" -DCMAKE_CXX_FLAGS="-march=znver4" -DCMAKE_BUILD_TYPE=Release && cmake --build build-sycl --config Release -j 16 Single B70: Dual B70: |
|
And with a much more interesting model, namely Qwen 3.6 27B: q4 and q8: |
|
@FCLC Thank you! |
|
Ooft. A770 16GB, i5 14600k, current cachyOS. fp16: fp32: Very weird, compared with the one in the table - much better prefill, half the decode performance. |
|
From the change of performance, I guess the Flash-attention is enabled in second case. 34 token/s is very similar with my old test result, which is impacted by driver. Could you run: |
|
Interesting. Just tried it with the FA switch: So...I do get a bit of a boost from FA on! I don't know what's happened with the performance from recompiling it, it's just fallen off a cliff. The Could it be the i915 driver that's the problem? |
|
Yes, this is the root cause. FA's impact is still less than I expected. Please remove one of them. It's risk to remove the GPU driver. |
|
Aha...winning! Mildly frustrating that I can't reproduce that nice PP result now, though. |
|
It's known issue. We are checking it. It's great to see better performance in your test. Thank you! |
|
B580, AMD Ryzen 7 5700X3D, Ubuntu 25.10 built with fp16 in |
|
Update as your feedback! |
|
Intel Arc Pro B50, Intel i7-8700 32GB RAM build: c0c7e14 (9298)
build: 2f6c815 (9397)
Command line arguments: Build options: cmake .. -B build -DGGML_VULKAN=1 -DGGML_RPC=ON nothing else changed between these runs, I tested my old version, ran "git pull", built it and retested |
|
@thordarsen Because your test result differs vastly from the common results as we known, I suggest not adding them in the table. how do you think? Thank you! |
|
It's an Arc Pro B50 and actually looking at the relative specs on Intel Ark I'm kinda inline with the B570 and B580 from the above data for FP16
( I purchased the B50 for the 16GB VRAM and 70W draw ) |
|
I haven't B50. |
B50 owner here as well. The B50 doesnt get the recognition it deserves, 16GB VRAM at max 70W makes it an out of this world addition to something like my unraid server. Ive got it doing Plex transcodes, Ollama, Llama.cpp, A-eye (image cataloguing), Memories (nextcloud). For a 24x7 machine its tough to beat. It isnt a Ferrari, and it isnt meant to be. Its a Camry. It gets you to where youre going, comfortably, and cheaply. |
|
@shansen008 Thank you for your sharing the usage of B50! |
|
I hope it helps. 255H, ARC 140T, 32GB RAM
build: d4c8e2c (9442) |
|
@jlionhan Thank you! |
|
~/llama.cpp$ cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON :~/llama.cpp$ GGML_OPENVINO_STATEFUL_EXECUTION=1
specs https://www.asrockind.com/en-gb/NUC%20BOX-358H |
|
📊 Intel Panther Lake Xe3 iGPU (12 EU) Benchmark Matrix: OpenVINO vs. Vulkan vs. SYCL Benchmarking sweep across all three major acceleration backends available in Environment
Models Tested
📈 Performance Summary Matrix
🛠️ Deep-Dive Analysis
📋 Raw Build & Execution Logs 1. Qwen3 80B MoE — OpenVINO Crash Log |
|
@bobguns Thank for your sharing! |
Okay i understand, made some new benchmarks with 512 input tokens Hardware & Environment
📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: d5ab083 (9476) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: d5ab083 (9476) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: d5ab083 (9476) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: d5ab083 (9476) |
|
16 bit Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: a468b89 (9477) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: a468b89 (9477) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: a468b89 (9477) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: a468b89 (9477) 📊 Running SYCL Benchmark on Panther Lake Xe3 iGPU
build: a468b89 (9477) |
|
@bobguns Thank you for your sharing! |
|
HW:ryzen5 5600X, DDR4-3600 128GB, ARC B570 FP32 FP16 |
|
@yqYo1 |
|
HW:ryzen5 5700X, DDR4-3600 64GB, ARC B580 + ARC PRO B60 (24gb) llama build b60+b580
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -fa 1
build: 65ef50a (9501) on arc b60 ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -fa 0 -dev sycl0
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev sycl0 -fa 1
build: 65ef50a (9501) on arc b580 ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev sycl1 -fa 0
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev sycl1 -fa 1
build: 65ef50a (9501) VULKAN After promt processing, the GPU frequency is reset to minimum and TG is low ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -fa 0
build: 65ef50a (9501) I set the minimum frequency on the GPU to b60 2300 and b580 2683 ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -fa 0
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -fa 1
build: 65ef50a (9501) on B60 ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev vulkan1 -fa 0
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev vulkan1 -fa 1
build: 65ef50a (9501) on b580 ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev vulkan0 -fa 0
build: 65ef50a (9501) ./llama-bench -m ~/llama-2-7b.Q4_0.gguf -dev vulkan0 -fa 1
build: 65ef50a (9501) |
|
@Andr010-lang Thank you for your sharing! |
Arc A380I know, I know... I got this it for its AV1 encoding/decoding, not LLMs, but here we are...
F16
F32
Got a couple of warnings during execution:
Other info: Build config (F16/F32 variations):
Found 1 SYCL devices:
SYCL Optimization Feature:
build: 7c158fb (b9518) |
|
@GerardoNevarez If you don't care the performance and running time, you could run a 2-7B LLM by llama-server on existed Intel GPU (like A380) to help to create a report or abstract based on your raw texts. Improve your work and life without cost. Thank you for your sharing! |
|
Updated Arc A380 bench runs with default build on e82beaa (9534) - you could use these for the table above. Forgot to mention, this rig uses PCIe 3.0 only, but has ReBar enabled (SYCL refuses to use the xe driver otherwise) F16
F32
"Static" build e82beaa (9534) with F16
F32
|
|
@GerardoNevarez But your case shows it is increased. Thank you! |
|
@arthw I ran it twice before posting, I was surprised than the "official" build is slower, I'm running natively on Ubuntu, not in Docker. I just pulled updates from GH and ran them again, results below (without FA) - they are a bit worse than the last time, I can't explain that either. "Official" build "Static" build My CPU is an old Xeon, but it has 45MB L3 cache, on a single die. Maybe communication between CPU/GPU gets improved by that - less long call jumps to the libraries? Does not explain your results on the A770 though. |
|
@GerardoNevarez We will check it later. |
|
Hi,
./llama-bench -fa 0,1 -m ../../models/llama-2-7b.Q4_0.gguf
build: 6471e3c (9607) sycl-ls Thanks for your great work! |
A380 - DockerApologies for the testing in docker, my local env is messed up in all sort of ways, im unable to test on bare metal however Ill share the docker compose and commands to reproduce if anyone is interested. Ill also attach other benchmarks to compare the current state as of this commit e95dae1 All results are the third results printed SYCL F16
SYCL F32
I ran a similar docker image a few days ago and I remember my results being far far better, not sure what has happened with the pp. Vulkanggml_vulkan: Found 1 Vulkan devices:
OpenvinoOpenVINO: using device GPU
Docker composeservices:
bench-openvino:
build:
context: .
dockerfile: .devops/openvino.Dockerfile
target: full
image: llama.cpp:full-openvino-local
devices:
- /dev/dri:/dev/dri
volumes:
- llama-cache:/models
environment:
- LD_LIBRARY_PATH=/app
- GGML_OPENVINO_DEVICE=${GGML_OPENVINO_DEVICE:-GPU}
- GGML_OPENVINO_STATEFUL_EXECUTION=1
- LLAMA_CACHE=/models
entrypoint: /app/llama-bench
command:
- -hf
- ${HF_REPO:-TheBloke/Llama-2-7B-GGUF:Q4_0}
- -fa
- "1"
- -ngl
- "99"
bench-sycl-f16:
build:
context: .
dockerfile: .devops/intel.Dockerfile
target: full
args:
GGML_SYCL_F16: "ON"
image: llama.cpp:full-sycl-f16-local
devices:
- /dev/dri:/dev/dri
volumes:
- llama-cache:/models
environment:
- LLAMA_CACHE=/models
- ONEAPI_DEVICE_SELECTOR=level_zero:0
- ZES_ENABLE_SYSMAN=1
entrypoint: /app/llama-bench
command:
- -hf
- ${HF_REPO:-TheBloke/Llama-2-7B-GGUF:Q4_0}
- -fa
- "1,0"
- -ngl
- "99"
bench-sycl-f32:
build:
context: .
dockerfile: .devops/intel.Dockerfile
target: full
args:
GGML_SYCL_F16: "OFF"
image: llama.cpp:full-sycl-f32-local
devices:
- /dev/dri:/dev/dri
volumes:
- llama-cache:/models
environment:
- LLAMA_CACHE=/models
- ONEAPI_DEVICE_SELECTOR=level_zero:0
- ZES_ENABLE_SYSMAN=1
entrypoint: /app/llama-bench
command:
- -hf
- ${HF_REPO:-TheBloke/Llama-2-7B-GGUF:Q4_0}
- -fa
- "1,0"
- -ngl
- "99"
bench-vulkan:
build:
context: .
dockerfile: .devops/vulkan.Dockerfile
target: full
image: llama.cpp:full-vulkan-local
devices:
- /dev/dri:/dev/dri
volumes:
- llama-cache:/models
environment:
- LLAMA_CACHE=/models
entrypoint: /app/llama-bench
command:
- -hf
- ${HF_REPO:-TheBloke/Llama-2-7B-GGUF:Q4_0}
- -fa
- "1,0"
- -ngl
- "99"
volumes:
llama-cache:
name: llama-cacheCommandsdocker compose run --build --rm bench-openvino # OpenVINO
docker compose run --build --rm bench-sycl-f16 # SYCL F16
docker compose run --build --rm bench-sycl-f32 # SYCL F32
docker compose run --build --rm bench-vulkan # VulkanFor repeated runs remove the |
|
@arthw dpkg -l | grep libze-intel-gpu1 SYCL F16
SYCL F32
Openvino
Vulkan
Performance for SYCL F16 improved after driver update, vulkan still low somehow. Openvino PP is insane however model selection is still catching up. |
|
@toomanybyt3s Could you check the driver by following cmds? |
Ah missing piece drivers, let me update them and ill rerun tests |
|
Uname shows this as my kernal, i might also update my system to 25.10 |
|
@toomanybyt3s Could you disable i915 to enable Xe driver on A380? Thank you! |
|
@toomanybyt3s try adjusting which driver is used, modifying the
This forces enabling the Xe and disabling i915 drivers for the card, value must match the PCIe Device ID for the card ( A380 = |
|
@GerardoNevarez , I did try to play around with XE drivers but since my server is mostly for jellyfin I couldnt get av1 decoding to work with the XE driver, im planning to get a intel arc b60 and continue with sycl backend and forcing the XE driver, since I read somewhere the newer XE driver is able to decode av1 with the b60 but not the a380 |
|
Hardware: Intel Core Ultra 5 250K Plus, DDR5-6400 16GBx1, Intel Arc B580 LE with minimum core clock set to 2850 MHz since it seems to drop during inference despite having plenty thermal headroom
Gemma4 E4B has 4B parameters active, so I have calculated the effective memory bandwidth utilization as 45%. There seems to be some overhead with MoE models in general. |
|
@egeoz Thank you! |
|
Intel Arc Pro B70 32GB — Llama 3.1 8B & 70B Build: b9739 (8452824) | OS: Ubuntu 26.04 | oneAPI 2026.0.0 | fp32 | FA=1
* 70B Q4_K_M exceeds 32GB VRAM — run at ngl=60 (hybrid CPU+GPU). All other models fit fully on-GPU. Q8_0 tg128 unchanged across b9672 and b9739 — generation appears memory-bandwidth-limited on this SKU. PR #21527 did not affect this result. Full results + raw JSONL: https://github.com/Wesley-Jakob-Gilbert/Intel-Arc-GPU-Benchmarks/blob/main/results/b70-solo/llama-cpp-inference.md |
|
@Wesley-Jakob-Gilbert Thank you! |
|
I recently ran new benchmarks and realized a mistake in my previous table. Flash attention was enabled in each of the above benchmarks. I've edited my original post. @arthw , please edit the first table for each of my results to set FA = 1. Additional BenchmarksAdding 15 more models (dense + MoE) benchmarked on the same Arc Pro B70 32GB / i9-14900K setup. Most run fully on-GPU (
* Hybrid CPU+GPU offload — model exceeds 32GB VRAM. Full results + raw llama-bench output: https://github.com/Wesley-Jakob-Gilbert/Intel-Arc-GPU-Benchmarks |
|
@Wesley-Jakob-Gilbert Thank you! |
|
Intel Arc A770 16GB — Llama 2 7B Q4_0 build: 81ff7ab (9928) | OS: Ubuntu 26.04 | oneAPI 2026.1.0 | fp32 | FA=0,1 ZES_ENABLE_SYSMAN=1 ./build/bin/llama-bench -m /path/to/llama-2-7b.Q4_0.gguf -fa 0,1
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | pp512 | 1624.44 ± 10.67 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | tg128 | 47.63 ± 0.06 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | pp512 | 1079.70 ± 0.51 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | tg128 | 55.31 ± 0.63 |
build: 81ff7abe5 (9928)Have modify the system env and drivers, but still could not reach the best performance above. lspci -nnk | grep -i vga -A3
03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A770] [8086:56a0] (rev 08)
Subsystem: Acer Incorporated [ALI] Device [1025:3888]
Kernel driver in use: xe
Kernel modules: i915, xe
icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2026.1.0 (2026.1.0.20260617)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2026.1/bin/compiler
Configuration file: /opt/intel/oneapi/compiler/2026.1/bin/compiler/../icpx.cfg
dpkg -l | grep libze-intel-gpu1
ii libze-intel-gpu1 26.05.37020.3-1 amd64 Intel oneAPI L0 support implementation for Intel GPUs -- shared library
|
|
I find your driver is old. |
|
After upgrade the driver, I get better score: dpkg -l | grep libze-intel-gpu1
ii libze-intel-gpu1 26.22.38646.4-0 amd64 Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
./build/bin/llama-bench -fa 0,1 -m /path/to/llama-2-7b.Q4_0.gguf
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | pp512 | 1618.98 ± 12.05 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | tg128 | 51.39 ± 0.04 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | pp512 | 1089.10 ± 1.74 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | tg128 | 60.98 ± 0.15 |
build: 683f0c72e (9941)And I found a env SYCL_UR_USE_LEVEL_ZERO_V2=1 ./build/bin/llama-bench -fa 0,1 -m /path/to/llama-2-7b.Q4_0.gguf
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | pp512 | 1712.78 ± 12.60 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 0 | tg128 | 58.68 ± 0.11 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | pp512 | 1126.50 ± 0.56 |
| llama 7B Q4_0 | 3.56 GiB | 6.74 B | SYCL | -1 | 1 | tg128 | 68.18 ± 0.61 |
build: 683f0c72e (9941) |
|
Yes. It's great! Thank you for your sharing! |
|
Hardware:
Software:
Model:
Command: Results:
Notes:
If useful, I can also share longer-context measurements for this same model (Qwen3.6-27B Q5_K_M), including FA on/off behavior at deeper KV/cache depths. |
|
@mikellantalde Thank you! |
|
Your benchmarks stand out, as well as your older commit reported. I just got my b70 and am struggling to hit 30t/s gen. I managed to go from <200t/s pp to 450-500 t/s pp by changing my batch and ubatch. But I am still only hitting about 20-25 t/s gen. I managed to get close to 40 with MTP. But still, your benchmarks seem a lot better overall. Do you have any insight to that? Is it the older commit ? the i9 ? the older models? (I'm compiling the old commit now for testing). |
|
Hi, there was a big difference between compiling with -DGGML_SYCL_F16=ON and without it. With F32, performance is much worse in PP—50% worse on average. Did you perhaps compile using F32? |
|
@NeoZhangJianyu , even if I do separate builds in separate folders ? I have several different builds in different folders. I am not running the script, I am calling ./llama-bench directly from the build folder that I am attempting to benchmark. |
|
I hope I'm not spamming this discussion, and I am grateful for everyone's input ! 🙏 I don't think the
But, some of the reports in this discussion show great results for MoE, so I have a couple of ideas, questions :
One new observation: In extended usage with hermes, prompt processing performance degrades over time. |
|
The perf data in the first table comes from the following comments. You could contact to the reporter for detailed parameter/hardware info. No, we didn't add the MoE and MTP columns.
|
|
@mknap try setting from here, I'm using this If after compilation llama-server crash just remove -DGGML_SYCL_DEVICE_ARCH=xe2 PP is like 800-900 tps, gen depends of output python code is more than 32 |
|
hello @mknap Sorry for the late reply, but you can see the details of my setup in my original post and at https://github.com/Wesley-Jakob-Gilbert/Intel-Arc-GPU-Benchmarks. Looking at your numbers closely, your dense model results align well with mine (tg ~21-23 vs my 23.47 and your F16 pp of 768 is even higher than my own dense pp which ran F16=OFF) My best guess is that for some reason, your MoE models may be spilling into hybrid CPU/GPU processing. I see you didn't pass -ngl explicitly and the table shows -1, so it may be landing on some auto-fit heuristic that doesn't guarantee full GPU residency for a large MoE model. That alone could account for the gap. Are you 100% sure in your benchmarks that you confirmed no CPU offload flags? Can you try again with an explicit -ngl 999 flag? I explicitly added the -ngl 999 flag in my results. Here's an example I ran: ./build/bin/llama-bench -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 999 -p 512 -n 128 But I agree that the discrepancy is strange. MoE models performed significantly better for me than dense models of the same size. |
|
Oh, that's interesting; I'll give it a try. cmake --build build-sycl --config Release -j |
GGML_SYCL_ENABLE_MKL_FA=1It's opened as default. In code: commit 11924d4 (tag: b10223, origin/master, master) It will get performance increase in more LLMs event with fp32 building on B60: Test Script
Environment Configurations
Per Metric
|
GGML_SYCL_FA_ONEDNN=1 on PTL (Intel Arc B390)I was curious about the performance impact of the oneDNN SDPA path introduced by PR #25222, so I benchmarked three paths: oneDNN / MKL / FA-off. I know oneDNN is currently only enabled on BMG (Battlemage) hardware, but I had an LLM run a test against the reproduction scheme of uxlfoundation/oneDNN#5510, and at least on PTL + Level Zero 1.15.39122 there were no computation errors. In code: commit 0713275 (build: b10270) To enable the oneDNN path on PTL, the BMG-only hardware check in Test Script
Per Metricllama-2-7b.Q4_0.gguf (llama 7B Q4_0)
gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf (gemma4 26B.A4B Q4_0)
Hardware
Software
Notes
|
|
I test it on Arc770 for PR: #25222 Arc770
As the comments of the code: We need to check if the above issue (wrong result) is fixed on target GPU. |
|
Wow, what a difference. I have a benchmark from a week ago to compare with. Spectacular: Qwen3.6-27B-Q5_K_M Performance ComparisonAll benchmarks were run with identical llama.cpp runtime parameters. The only compilation changes were: -DGGML_SYCL_DEVICE_ARCH=xe2 -DGGML_SYCL_DNN=ONThe second run also used a larger context size, but this does not affect these benchmark results. Runtime Parameters (identical in both runs)
Speculative Decoding--spec-type draft-mtp --spec-draft-n-max 3Baseline BuildBuild ConfigurationBackend: SYCL
Precision: FP16Optimized BuildBuild ConfigurationBackend: SYCL
Precision: FP16
-DGGML_SYCL_DEVICE_ARCH=xe2 -DGGML_SYCL_DNN=ONPrompt Processing Performance
|
|
In code: commit f8e3026 (tag: b10322, origin/master, master) Test Script
This is a community fine-tuned version of standard gemma4 12b model - yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF. Other models shared the same command options. Build Configurations
My setup
Metric
Interestingly, error "failed to create context" was raised when fa=0. Besides, I didn't see any notable speed differences between MKL_FA and DNN_FA, only on this gemma4 and granite and not on llama2 though... |
2. Thank you for your sharing! |
|
I have swept a few parameters trying to improve the decode speed of some modern useful models (Muse Glimmer and Qwen 3.8 27B) on my double-GPU setup:
I built llama.cpp from source, commit 4df29be using the following command: Over the sweep, F16 made the biggest impact on the performance, with AOT helping squeeze a few more percent. On Muse Glimmer (bartowski/Muse-Glimmer-30B-Q4_K_S.gguf), this translated into the following results on the i915 driver:
GGML_SYCL_DEV2DEV_MEMCPY had no impact. Xe DriverI wanted to improve FA performance further, as well as get better tensor split decode results. The first step was to move to the Xe driver, which yielded:
Note both PP and TG improvements, especially with Flash Attention. These figures are on par with Vulkan+i915 on decode (~17tps) while being much better on prefill. GGML_SYCL_DEV2DEV_MEMCPY still had no effect. IOMMU PTAs I wanted to improve the tensor split, I set IOMMU to the passthrough mode. The same benchmark command gave the following results:
Note that the layer split is not affected, while in the tensor split mode the prefill is 466->580 - 24% faster, outperforming the layer split mode! The model is coherent. No luck on decode yet :( Counterintuitively (?), GGML_SYCL_DEV2DEV_MEMCPY=1 actually decreases the performance:
Setting attention cache to q8_0 (GGML_SYCL_DEV2DEV_MEMCPY=0) makes both prefill and decode a bit slower:
Bonus: Qwen3.8-27BWith lmstudio-community/Qwen3.8-27B-Q4_K_M.gguf, the best result is:
Takeaways
Hope this is useful (and also hope decode can be made faster). |
|
It's good result. Thank your sharing! |
|
Are these suggestions applicable to Windows? |
Environment
Build optionscmake -B build/sycl -G Ninja \
-DGGML_SYCL=ON \
-DGGML_SYCL_TARGET=INTEL \
-DGGML_SYCL_DNN=ON \
-DGGML_SYCL_F16=ON \
-DGGML_SYCL_DEVICE_ARCH=bmg-g31 \
-DGGML_NATIVE=ON \
-DMKL_DIR=/opt/intel/oneapi/mkl/latest/lib/cmake/mkl \
-DCMAKE_C_COMPILER=icx \
-DCMAKE_CXX_COMPILER=icpx \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_RPATH="\$ORIGIN/../lib" \
-DCMAKE_CXX_FLAGS="-ffast-math -O3" \
-DCMAKE_C_FLAGS="-ffast-math -O3"Benchmark options./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
-ctk q8_0 -ctv q8_0 -fa on -ngl 99 \
-p 512,1024 -n 512
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
-ctk q8_0 -ctv q8_0 -fa on -ngl 99 \
-p 512,1024 -n 512
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_M \
-ctk q8_0 -ctv q8_0 -fa on -ngl 99 \
-p 512,1024 -n 512Qwen 3.8 27B (Q4, Q5, Q6) - Q5 has the best performance.
build: cd26896 (10553)
build: cd26896 (10553)
build: cd26896 (10553) |
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Purpose
It's used to share the performance data on Intel GPU with SYCL backend.
The performance data is only used as reference, since we don't double check the data.
It can not be used as any commercial purpose.
Rule
Encourage to test with default setting (environment variables).
If you want to update the data with special building or running setting, please create a new table.
Create/update the tables directly following the format.
Insert new record, instead of update it for same keys; Sort the records by col1, col2, col3.
Add your comments in the latest for more discussion.
Don't add table to compare with other hardware, framework or backend.
Please run 1+ times and update with the stable data.
Tips and tricks
Xe driver gives a huge boost over i915.
Combined with IOMMU and Xe,
--split-mode tensorgives decent number on prefill2 x Arc A770 can give usable speeds, at least with benchmark 512+128 regimes.
Refer to: Performance of llama.cpp on Intel GPU with SYCL backend #23313 (comment)
User can choose them according to the test result.
Quick build scripts:
Performance data on Intel GPU
Default setting
Build:
Run:
Data:
FP16
t/s
t/s
DDR5-6400 16GB
DDR5-6400 16GB
- Medium
32GB
- Medium
32GB
DDR5-6400 16GB
DDR5-6400 16GB
64GB
24.04.4
64GB
24.04.4
5600X
DDR4-3600
128GB
24.04.4
6.17.0-29-generic
5600X
DDR4-3600
128GB
24.04.4
6.17.0-29-generic
5600X
DDR4-3600
128GB
24.04.4
6.17.0-29-generic
5600X
DDR4-3600
128GB
24.04.4
6.17.0-29-generic
DDR5-6400 16GB
DDR5-6400 16GB
5700X3D
25.10
5700X3D
25.10
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
64GB DDR5
26.04
64GB DDR5
26.04
More PP/TG Types:
FP16
t/s
t/s
t/s
t/s
(12 EUs)
Panther Lake
26.04
(12 EUs)
Panther Lake
26.04
All reactions