Question About use_adreno_kernels Threshold for Q4 MatMul on Adreno 750

[forforever73]

@lhez Sorry for taking your time, I’m running a new model on an Adreno 750 GPU and noticed for Q4 weights, using the optimized kernel CL_mul_mat_Ab_Bi_8x4 seems to require that use_adreno_kernels() returns true. However, in my model there are several matmul shapes like:
A: [256, 1280, 1, 1]
B: [256, 512, 1, 1]
→ Output: [1280, 512, 1, 1]
So the kernel falls back to kernel_mul_mat_q4_0_f32_1d_8x_flat. This fallback kernel is about 10× slower on Adreno 750. I experimented by modifying the internal threshold
int64_t threshold_ne0 = 256;
After lowering the threshold, the Adreno kernels are used, performance improves dramatically, and the model’s PPL shows no meaningful change.
So what was the original reasoning behind the use_adreno_kernels() threshold? If I reduce the threshold to 256, is there any potential risk I should be aware of?

[lhez]

@forforever73 Apologies for the delay. kernel_mul_mat_Ab_Bi_8x4 should be general enough to handle smaller sizes. When the kernel was initially written, our main target was 7B models. In addition, most models we work with don't have such dimensions so we put a threshold here (is it possible for you to share which model this is, or point us to a similar open source model?) This kernel requires both weights and activation matrices be transposed. This kernel have caused some issues for us in the past (e.g., initially it wasn't able to properly handle 0.5B models). Feel free to use this kernel for smaller ne00 values and let us know if you see any issue.

[forforever73]

Thank you very much for the detailed reply!

Our team is working on a new model that uses PLE (Per-Layer Embeddings) , which introduces weight shapes like [256, 1280, 1, 1]. So far, after lowering the threshold to threshold_ne0 = 256, we haven’t observed any issues — both accuracy and performance.

We’ll continue testing , we’ll definitely report back if encounter any problems.
Thanks again for your help!