Skip to content

ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 - #28479

Draft
alanhc wants to merge 1 commit into
ggml-org:masterfrom
alanhc:repost-q8-ime1-v2
Draft

ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60#28479
alanhc wants to merge 1 commit into
ggml-org:masterfrom
alanhc:repost-q8-ime1-v2

Conversation

@alanhc

@alanhc alanhc commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Overview

I have a Milk-V Jupiter board and wanted Q8_0 to run properly on the SpacemiT X60. The X60 already
had IME acceleration, but only for Q4_0/Q4_1/Q4_K. Q8_0 had no IME1 kernel at all, and because the
SpacemiT build sets GGML_CPU_REPACK=OFF, there was no repack path compiled in either — so Q8_0
had no accelerated path whatsoever and ran at about a tenth of Q4_0's prefill speed on the same
board. This PR adds the missing Q8_0 IME1 kernel.

This replaces #28025, which I closed myself. That one hit the 1-open-PR limit for new contributors
while #27961 was still under review, and the description was flagged as AI-assisted. #27961 has
since merged, so the limit no longer applies, and I have written this description by hand.

The kernel is also further along than in #28025. That PR was M1-only and reached 39.97 t/s prefill;
this one adds the 4-row path, which roughly doubles it again.

Additional information

Changes:

  • make_block_q8_0x16 and the repack entry in repack.cpp: interleave Q8_0 weights into the
    16-column layout the IME1 vmadot sequence expects.
  • ime1::gemm_kernel_i8i8 in ime1_kernels.cpp: the int8 x int8 IME1 kernel, with both a single-row
    and a 4-row A path. The 4-row path loads each B panel once and reuses it across 4 rows of A, which
    is where most of the gain over the M1-only version comes from.
  • quantize_a_4row_i8: the 4-row activation quantization the M4 path needs. In ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 #28025 this was
    nullptr and gemm_m >= 4 was forced back through the single-row path.
  • wiring in forward_mul_mat and the repack factory for block_q8_0.
  • docs: mark Q8_0 as supported on X60.

Testing

Milk-V Jupiter (SpacemiT X60, 8 cores / 4 preferred, RVV1.0, VLEN=256), Bianbu 2.1.1, gcc 14.2.
Qwen2.5-0.5B-Instruct, llama-bench, -t 4 under taskset -c 0-3, 5 repetitions, board otherwise
idle.

Correctness: I compared the kernel against a quant-exact reference (dequantizing the same blocks and
doing the multiply in integer arithmetic, rather than against a float reference) for K = 32 up to
4096. Max relative error is ~1e-6, which is the expected float accumulation difference. I also
checked that generation stays coherent across several prompts.

One note in case it saves someone else the time: test-backend-ops is not a useful gate here. On a
CPU-only build it prints "Skipping CPU backend" and then reports OK without having tested anything,
so a green run there means nothing for this kernel.

Performance, pp128 prefill, before and after this change on the same build settings (both with IME1
enabled, so the only variable is the kernel):

model before after speedup
Qwen2.5-0.5B Q8_0 10.70 t/s 95.27 t/s 8.9x

For reference, Q4_0 on the same board measures 107.81 t/s with this change applied, against 106.40 t/s
before it, i.e. unchanged within run-to-run variation, which is what I would expect since this PR
does not touch that path.

I am only claiming prefill here. Token generation does improve, but Q4_0 token generation moved by
about 1.19x between the two builds despite that path being untouched, so I do not trust the
token-generation numbers enough to quote them as a result of this change.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: the kernel code was written with the help of an AI coding agent. This PR
    description and the commit messages are my own.

On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K.
Q8_0 had no IME1 kernel, and since the SpacemiT build sets
GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0
had no accelerated path at all and ran roughly ten times slower than
Q4_0 for prefill on the same board.

- add make_block_q8_0x16 and the Q8_0 repack entry: interleave the
  weights into the 16-column layout the IME1 vmadot sequence expects
- add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a
  single-row and a 4-row A path; the 4-row path loads each B panel once
  and reuses it across 4 rows of A
- add quantize_a_4row_i8 for the 4-row activation quantization
- wire both into forward_mul_mat and the repack factory for Q8_0
- docs: mark Q8_0 as supported on X60

Correctness was checked against a quant-exact integer reference for
K = 32 up to 4096, with a max relative error of about 1e-6, and by
checking that generation stays coherent across several prompts.

Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with
Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5
repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0
is unchanged at 106.40 -> 107.51 t/s, as expected since this does not
touch that path.
@github-actions github-actions Bot added documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning labels Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant