I2_S inference is broken on aarch64. The model loads and runs, but output is nonsense — it does not crash, so it looks like a bad model rather than a broken kernel.
On an Orange Pi 5 Plus (RK3588, Debian 12, GCC 12.2) with microsoft/BitNet-b1.58-2B-4T's official ggml-model-i2_s.gguf:
$ llama-cli -m ggml-model-i2_s.gguf -p "The capital of France is" -n 12 --temp 0
> The capital of France is ????????????????
[ Prompt: 0.7 t/s | Generation: 0.7 t/s ]
The same file on x86-64 gives The capital of France is Paris. at ~40 t/s. Model md5 verified identical on both machines.
Three separate bugs, all in 3rdparty/llama.cpp code paths that x86 never compiles:
QK_I2_S is 128 under AVX2 but 64 under __ARM_NEON, in both quants.c and ggml-cpu-i2s.c. It is the on-disk block size, so it must match the file format on every architecture.- The scalar
vec_dot fallback decodes the block-interleaved weight layout sequentially.
ggml_gemm_i2_i8_s's ACT_PARALLEL branch inverts ggml_vec_dot_i2_i8_s's nrc semantics, corrupting prefill.
There is also no NEON path for I2_S at all — aarch64 unpacks one 2-bit weight at a time.
Fixes in isHuangXin/llama.cpp#2, against the pinned 3rdparty/llama.cpp submodule. After them, perplexity on aarch64 matches x86 to 0.161% (74.0952 vs 73.9758, same model and corpus), and kernel output is bit-identical for both GEMV and GEMM.
This may be the same root cause as #55.
I2_S inference is broken on aarch64. The model loads and runs, but output is nonsense — it does not crash, so it looks like a bad model rather than a broken kernel.
On an Orange Pi 5 Plus (RK3588, Debian 12, GCC 12.2) with
microsoft/BitNet-b1.58-2B-4T's officialggml-model-i2_s.gguf:The same file on x86-64 gives
The capital of France is Paris.at ~40 t/s. Model md5 verified identical on both machines.Three separate bugs, all in
3rdparty/llama.cppcode paths that x86 never compiles:QK_I2_Sis 128 under AVX2 but 64 under__ARM_NEON, in bothquants.candggml-cpu-i2s.c. It is the on-disk block size, so it must match the file format on every architecture.vec_dotfallback decodes the block-interleaved weight layout sequentially.ggml_gemm_i2_i8_s's ACT_PARALLEL branch invertsggml_vec_dot_i2_i8_s'snrcsemantics, corrupting prefill.There is also no NEON path for I2_S at all — aarch64 unpacks one 2-bit weight at a time.
Fixes in isHuangXin/llama.cpp#2, against the pinned
3rdparty/llama.cppsubmodule. After them, perplexity on aarch64 matches x86 to 0.161% (74.0952 vs 73.9758, same model and corpus), and kernel output is bit-identical for both GEMV and GEMM.This may be the same root cause as #55.