Performance of llama.cpp on Arm CPUs

[manunicholasjacob]

There are already benchmark threads for CUDA (#15013) and Apple (#4167), but I couldn't find a similar thread for Arm CPUs, so I thought it would be useful to start one here. For consistency, it would be great if people could post one hardware line, the /proc/cpuinfo Features line, the llama-bench table, the build: line, and whether KleidiAI was enabled or disabled. The command I used was:

llama-bench -m <model> -t 1,2,...,N -p 512 -n 128 -r 5

If a board has enough memory, using Llama 2 7B Q4_0 would make the results easier to compare with the existing benchmark threads; I used Qwen2.5-0.5B Q4_0 on the Pi 5 because the board only has 2 GB and the 7B model needs around 4 GB. On my Pi 5, the best prompt-processing result was 289.92 t/s at 4 threads, while the best decode result was 33.84 t/s at 2 threads. It would be useful to get rows from Graviton, Ampere Altra, Snapdragon, RK3588, other Raspberry Pis, and any other Arm CPUs people have available.

Raspberry Pi 5 Model B Rev 1.0 (BCM2712, 4x Cortex-A76 @ 2.4 GHz), 2 GB LPDDR4X, Debian 12 (bookworm), kernel 6.12.93+rpt-rpi-2712, governor ondemand, throttled=0x0 before and after

Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

cmake -B build -DCMAKE_BUILD_TYPE=Release (gcc 12.2.0), CPU backend, KleidiAI off

./build/bin/llama-bench -m qwen2.5-0.5b-instruct-q4_0.gguf -t 1,2,3,4 -p 512 -n 128 -r 5

model	size	params	backend	threads	test	t/s
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	1	pp512	93.49 ± 0.20
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	1	tg128	32.42 ± 0.14
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	2	pp512	173.68 ± 0.08
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	2	tg128	33.84 ± 0.05
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	3	pp512	233.22 ± 0.94
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	3	tg128	29.37 ± 0.05
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	4	pp512	289.92 ± 0.11
qwen2 1B Q4_0	403.20 MiB	630.17 M	CPU	4	tg128	27.28 ± 0.02

build: 836d571 (1)

[tudou139]

X2 extreme？

[manunicholasjacob]

I don’t have any X2 Elite Extreme numbers yet since I don’t have access to one. If you have one, it would be useful to post the hardware details, CPU Features line (or equivalent), llama-bench results, build/commit, and whether KleidiAI was enabled.

There’s also the existing Snapdragon X Elite discussion in #8273 with first-generation results for comparison.