ggml-cpu: add repack GEMM and GEMV for floating-point

[taimur-10x]

Summary

This PR adds repacking and GEMM/GEMV kernels for floating-point (FP16 and FP32) for RVV (with the zvfh extension).

Key Changes

• Added RVV kernels for GEMM with tiling:
  • 7 x {16, 32, 64, 128} (selected based on VLEN)
• Added RVV kernels for GEMV with tiling:
  • 1 x {16, 32, 64, 128} (selected based on VLEN)
• Added scalar functions for repacking. They support arbitrary tile sizes.
• Generic scalar fallbacks for GEMM/GEMV operations.
• ggml_quantize_mat_t is refactored to ggml_repack_mat_t to allow for a common interface for both quantization and floating-point repacking.
• Additional template parameter NB_ROWS added to select the number of rows to interleave for repacking. Previously, this was fixed at 4.

Tile Sizes

The repack operation interleaves N rows of activations with an interleave size of K, and M columns of weights with an interleave size of K.

NxK is fixed at 7x1. This introduces 7 accumulators with LMUL=4 (7 x 4 = 28 registers), each accumulating M results.

M is varied based on the available VLEN:

VLEN	Tile Size (N x M x K)
128	7 x 16 x 1
256	7 x 32 x 1
512	7 x 64 x 1
1024	7 x 128 x 1

M is the maximum number of values that can be loaded in (LMUL=2 for F16, LMUL=4 for F32).

Testing

Kernels were functionally tested on QEMU for VLENs (128-bit, 256-bit, 512-bit and 1024-bit) for a range of input sizes.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256)

Prefill / Prompt Processing (GEMM)

Tokens / Second

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
Tinyllama F16 1.1B	28	24.72	8.31
Tinyllama F16 1.1B	32	16.72	8.42
Tinyllama F16 1.1B	64	22.55	8.57
Tinyllama F16 1.1B	128	22.78	8.78
Tinyllama F16 1.1B	256	21.82	8.57
Tinyllama F16 1.1B	512	21.81	8.68

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
Tinyllama F32 1.1B	28	11.45	3.72
Tinyllama F32 1.1B	32	7.13	3.75
Tinyllama F32 1.1B	64	10.76	3.74
Tinyllama F32 1.1B	128	10.86	3.73
Tinyllama F32 1.1B	256	10.94	3.68
Tinyllama F32 1.1B	512	11.12	3.79

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
BERT Large Uncased F16	28	82.25	32.13
BERT Large Uncased F16	32	63.16	27.33
BERT Large Uncased F16	64	79.16	30.01
BERT Large Uncased F16	128	76.20	30.80
BERT Large Uncased F16	256	66.09	28.01
BERT Large Uncased F16	512	50.57	24.26

Model	Prompt Size	Repack GEMM (7x32)	Vec Dot
BERT Large Uncased F32	28	43.29	11.16
BERT Large Uncased F32	32	29.17	11.08
BERT Large Uncased F32	64	37.31	11.35
BERT Large Uncased F32	128	39.70	11.16
BERT Large Uncased F32	256	35.62	11.95
BERT Large Uncased F32	512	31.60	11.18

Result: ~2x-3x speedup over vec_dot

Decode (GEMV)

Tokens / Second

Model	Decode Size (Prompt=32)	Repack GEMV (1x32)	Vec Dot
Tinyllama F16 1.1B	10	3.37	3.11
Tinyllama F16 1.1B	16	3.29	3.45
Tinyllama F16 1.1B	32	3.12	3.25
Tinyllama F16 1.1B	64	3.23	3.27
Tinyllama F16 1.1B	100	3.04	3.15
Tinyllama F16 1.1B	128	3.09	3.2
Tinyllama F16 1.1B	256	3.15	3.19

Model	Decode Size (Prompt=32)	Repack GEMV (1x32)	Vec Dot
Tinyllama F32 1.1B	10	1.66	1.74
Tinyllama F32 1.1B	16	1.73	1.63
Tinyllama F32 1.1B	32	1.81	1.68
Tinyllama F32 1.1B	64	1.61	1.69
Tinyllama F32 1.1B	100	1.72	1.75
Tinyllama F32 1.1B	128	1.76	1.72
Tinyllama F32 1.1B	256	1.75	1.69

Result: No noticeable improvement, as decode remains memory-bound.

Additional Notes

• Current fallback model requires every architecture to have a scalar fallback for each implementation. This creates a clutter in arch-fallback.h as 7xMx1 is very RVV-specific tiling, and should not be used by other architectures.
• GEMM reaches peak performance when the prompt is a multiple of 7 (for example, prompt=28). To handle leftovers, it defaults to GEMV, which impacts performance. Ideally, there should be leftover Nx32 kernels which handle each leftover case from 2-6 leftover tokens.

Future Work

Subsequent PRs plan to add RVV kernels for quantization types, as well as extend existing quantization support to other VLENs.

[luhenry]

@xctan it would be lovely to have your review on that as it will go to upstream next.

[xctan]

Changing ncols_interleaved from a template parameter to a regular argument would offer a couple of advantages: ability of vectorized handling of leftover elements and, more importantly, it would significantly reduce the number of specialized kernels in arch-fallback.h (a file created due to MachO limitations). The logic for selecting the vector length could then be moved into the inner kernels to avoid changing the unified function signatures. Furthermore, the NB_COL template parameter in the outer wrapper structs might need a new special value to represent indefinite vector length, impacting related index calculations. Ultimately, this approach would lead to a cleaner implementation and provide flexibility for other variable vector length instructions like SVE.

Since my suggestion effectively refactors the repack logic to accommodate variable vector length architectures, I think it's perfectly acceptable to implement this in a follow-up PR after discussing it with other upstream maintainers.

[taimur-10x]

Since my suggestion effectively refactors the repack logic to accommodate variable vector length architectures, I think it's perfectly acceptable to implement this in a follow-up PR after discussing it with other upstream maintainers.

@xctan, I'll move this to upstream then to further discuss the structural changes around the code.