ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 - #28479
Draft
alanhc wants to merge 1 commit into
Draft
Conversation
On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K. Q8_0 had no IME1 kernel, and since the SpacemiT build sets GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0 had no accelerated path at all and ran roughly ten times slower than Q4_0 for prefill on the same board. - add make_block_q8_0x16 and the Q8_0 repack entry: interleave the weights into the 16-column layout the IME1 vmadot sequence expects - add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a single-row and a 4-row A path; the 4-row path loads each B panel once and reuses it across 4 rows of A - add quantize_a_4row_i8 for the 4-row activation quantization - wire both into forward_mul_mat and the repack factory for Q8_0 - docs: mark Q8_0 as supported on X60 Correctness was checked against a quant-exact integer reference for K = 32 up to 4096, with a max relative error of about 1e-6, and by checking that generation stays coherent across several prompts. Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5 repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0 is unchanged at 106.40 -> 107.51 t/s, as expected since this does not touch that path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
I have a Milk-V Jupiter board and wanted Q8_0 to run properly on the SpacemiT X60. The X60 already
had IME acceleration, but only for Q4_0/Q4_1/Q4_K. Q8_0 had no IME1 kernel at all, and because the
SpacemiT build sets
GGML_CPU_REPACK=OFF, there was no repack path compiled in either — so Q8_0had no accelerated path whatsoever and ran at about a tenth of Q4_0's prefill speed on the same
board. This PR adds the missing Q8_0 IME1 kernel.
This replaces #28025, which I closed myself. That one hit the 1-open-PR limit for new contributors
while #27961 was still under review, and the description was flagged as AI-assisted. #27961 has
since merged, so the limit no longer applies, and I have written this description by hand.
The kernel is also further along than in #28025. That PR was M1-only and reached 39.97 t/s prefill;
this one adds the 4-row path, which roughly doubles it again.
Additional information
Changes:
make_block_q8_0x16and the repack entry inrepack.cpp: interleave Q8_0 weights into the16-column layout the IME1
vmadotsequence expects.ime1::gemm_kernel_i8i8inime1_kernels.cpp: the int8 x int8 IME1 kernel, with both a single-rowand a 4-row A path. The 4-row path loads each B panel once and reuses it across 4 rows of A, which
is where most of the gain over the M1-only version comes from.
quantize_a_4row_i8: the 4-row activation quantization the M4 path needs. In ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 #28025 this wasnullptrandgemm_m >= 4was forced back through the single-row path.forward_mul_matand the repack factory forblock_q8_0.Testing
Milk-V Jupiter (SpacemiT X60, 8 cores / 4 preferred, RVV1.0, VLEN=256), Bianbu 2.1.1, gcc 14.2.
Qwen2.5-0.5B-Instruct,
llama-bench,-t 4undertaskset -c 0-3, 5 repetitions, board otherwiseidle.
Correctness: I compared the kernel against a quant-exact reference (dequantizing the same blocks and
doing the multiply in integer arithmetic, rather than against a float reference) for K = 32 up to
4096. Max relative error is ~1e-6, which is the expected float accumulation difference. I also
checked that generation stays coherent across several prompts.
One note in case it saves someone else the time:
test-backend-opsis not a useful gate here. On aCPU-only build it prints "Skipping CPU backend" and then reports OK without having tested anything,
so a green run there means nothing for this kernel.
Performance, pp128 prefill, before and after this change on the same build settings (both with IME1
enabled, so the only variable is the kernel):
For reference, Q4_0 on the same board measures 107.81 t/s with this change applied, against 106.40 t/s
before it, i.e. unchanged within run-to-run variation, which is what I would expect since this PR
does not touch that path.
I am only claiming prefill here. Token generation does improve, but Q4_0 token generation moved by
about 1.19x between the two builds despite that path being untouched, so I do not trust the
token-generation numbers enough to quote them as a result of this change.
Requirements
description and the commit messages are my own.