ggml : Implementations for Q4_0_8_8 quantization based functions - RISC-V vector version

[xctan]

• This PR supersedes Implementations for Q4_0_8_8 quantization based functions - RISC-V vector version #9953
• The PR contains RISC-V Vector version of ggml_gemm_q4_0_8x8_q8_0 used for Q4_0_8_8 quantized models

I've tested Mistral 7B in qemu, and it just worked. I'm still choosing suitable 3B models for my dev board with only 4GB RAM (Banana Pi BPI-F3), so I can't give any performance evaluation as of now and any help is welcome! BTW Mistral 7B could run on BPI-F3 4GB with another 4GB swap memory enabled, but it was much slower than even qemu.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[xctan]

Model: https://huggingface.co/CobraMamba/mamba-gpt-3b-v4
Compiler: GCC 13.2.0

model	size	params	backend	threads	test	t/s	speedup	commit
llama 3B Q4_0_8_8	1.84 GiB	3.43 B	CPU	8	pp512	1.82 ± 0.00	271%	78c78e2
llama 3B Q4_0	1.84 GiB	3.43 B	CPU	8	pp512	1.04 ± 0.00	112%	66c2c93
llama 3B Q4_0_8_8	1.84 GiB	3.43 B	CPU	8	pp512	0.49 ± 0.00		66c2c93
llama 3B Q4_0_8_8	1.84 GiB	3.43 B	CPU	8	tg128	2.25 ± 0.10	350%	78c78e2
llama 3B Q4_0	1.84 GiB	3.43 B	CPU	8	tg128	1.27 ± 0.03	154%	66c2c93
llama 3B Q4_0_8_8	1.84 GiB	3.43 B	CPU	8	tg128	0.50 ± 0.01		66c2c93

[xctan]

Is there anything wrong with this PR? Should I provide more test results like llama-perplexity? @ggerganov
I'm trying ./llama-perplexity -m <model_name> -f wikitext-2-raw/wiki.test.raw for both Q4_0_8_8 GEMM implementations with and without this PR, but it takes ~400 hours in total to finish.

[ixgbe]

Model: https://huggingface.co/CobraMamba/mamba-gpt-3b-v4 Compiler: GCC 13.2.0

model size params backend threads test t/s speedup commit
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 pp512 1.82 ± 0.00 271% 78c78e2
llama 3B Q4_0 1.84 GiB 3.43 B CPU 8 pp512 1.04 ± 0.00 112% 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 pp512 0.49 ± 0.00 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 tg128 2.25 ± 0.10 350% 78c78e2
llama 3B Q4_0 1.84 GiB 3.43 B CPU 8 tg128 1.27 ± 0.03 154% 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 tg128 0.50 ± 0.01 66c2c93

Questions about Testing and Model Configuration

Hello! I have two questions regarding your project:

1. Functional Testing Coverage

I notice that only performance tests are visible in the documentation/repository. How do you ensure functional correctness without explicit functional tests? Could you provide more details about your testing strategy for functional validation?

2. Model Variants Clarification

You reference the model: https://huggingface.co/CobraMamba/mamba-gpt-3b-v4

However, in your test results, I see references to:

• llama 3B Q4_0_8_8
• llama 3B Q4_0

These variants don't appear to be directly available in the linked model repository. Could you clarify:

• How these specific quantized versions were obtained?
• Are these custom quantizations you performed?
• If so, could you share the quantization process or provide links to these model variants?

Thank you for your time and assistance!

[xctan]

Model: https://huggingface.co/CobraMamba/mamba-gpt-3b-v4 Compiler: GCC 13.2.0
model size params backend threads test t/s speedup commit
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 pp512 1.82 ± 0.00 271% 78c78e2
llama 3B Q4_0 1.84 GiB 3.43 B CPU 8 pp512 1.04 ± 0.00 112% 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 pp512 0.49 ± 0.00 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 tg128 2.25 ± 0.10 350% 78c78e2
llama 3B Q4_0 1.84 GiB 3.43 B CPU 8 tg128 1.27 ± 0.03 154% 66c2c93
llama 3B Q4_0_8_8 1.84 GiB 3.43 B CPU 8 tg128 0.50 ± 0.01 66c2c93

Questions about Testing and Model Configuration

Hello! I have two questions regarding your project:

1. Functional Testing Coverage

I notice that only performance tests are visible in the documentation/repository. How do you ensure functional correctness without explicit functional tests? Could you provide more details about your testing strategy for functional validation?

My apologies for the confusion, my test was a bit coarse. The functional correctness is validated by examining the output of model evaluations intuitively (whether the model is coherent) and statistically (by checking the perplexity metric before and after optimization). I remember that the perplexity results were very close, but the original test data was lost.

p.s. The reference perplexity metric can be calculated on other performant devices, so you don't need to wait another painful 400 hours.

2. Model Variants Clarification

You reference the model: https://huggingface.co/CobraMamba/mamba-gpt-3b-v4

However, in your test results, I see references to:

• llama 3B Q4_0_8_8
• llama 3B Q4_0

These variants don't appear to be directly available in the linked model repository. Could you clarify:

• How these specific quantized versions were obtained?
• Are these custom quantizations you performed?
• If so, could you share the quantization process or provide links to these model variants?

Thank you for your time and assistance!

These quantized versions were obtained using the quantize tool in this llama.cpp project. Steps include converting the original format to GGUF format and quantizing the model to the desired formats. You might need to checkout an older version of llama.cpp because the manual Q4_0 8x8 repack format was deprecated afterward, and the repacking process was made automatic as a new backend. I haven't checked the Q4_0 format after the automatic repacking patch was merged, and have focused on K-quant types instead.