CUDA: use fastdiv + ggml_cuda_mad for mmvf

[am17an]

I see speedups in my 3090, but not so much on a 4090. I suspect it due to better integer division hardware on newer cards, but I did not find any documentation to confirm.

on 3090:

Model	Test	t/s master	t/s cuda_mmvf_fastdiv	Speedup
lfm2moe 8B.A1B BF16	tg32	117.54	128.09	1.09
lfm2moe 8B.A1B BF16	tg64	120.75	127.56	1.06
lfm2moe 8B.A1B BF16	tg128	121.43	128.15	1.06

on 4090:

Model	Test	t/s master	t/s cuda_mmvf_fastdiv	Speedup
lfm2moe 8B.A1B F16	tg32	139.51	140.07	1.00
lfm2moe 8B.A1B F16	tg64	139.41	139.74	1.00
lfm2moe 8B.A1B F16	tg128	139.35	139.57	1.00

[JohannesGaessler]

I can confirm a speedup, though a smaller one. Presumably it will depend on the model.

GPU	Model	Test	t/s 477a66b	t/s 8898040	Speedup
MI50	llama 1B BF16	tg128	150.16	150.15	1.00
MI50	llama 1B F16	tg128	149.20	150.19	1.01
MI50	llama 1B all F32	tg128	93.51	93.58	1.00
P40	llama 1B BF16	tg128	109.53	110.07	1.00
P40	llama 1B F16	tg128	109.01	109.70	1.01
P40	llama 1B all F32	tg128	59.18	59.29	1.00
RTX 3090	llama 1B BF16	tg128	269.84	271.69	1.01
RTX 3090	llama 1B F16	tg128	270.05	272.03	1.01
RTX 3090	llama 1B all F32	tg128	152.44	153.25	1.01
RTX 4090	llama 1B BF16	tg128	316.85	317.81	1.00
RTX 4090	llama 1B F16	tg128	316.88	317.98	1.00
RTX 4090	llama 1B all F32	tg128	174.09	174.31	1.00
RX 6800	llama 1B BF16	tg128	94.65	96.46	1.02
RX 6800	llama 1B F16	tg128	94.41	96.61	1.02
RX 6800	llama 1B all F32	tg128	80.23	80.64	1.00
RX 9060 XT	llama 1B BF16	tg128	98.26	99.33	1.01
RX 9060 XT	llama 1B F16	tg128	99.47	99.96	1.00
RX 9060 XT	llama 1B all F32	tg128	57.23	57.74	1.01

[JohannesGaessler]

I think mul_mat_vec_f should always pass float2, half2, or nv_bfloat162 to ggml_cuda_mad and then let that function decide how to do the calculation. For example, on I think Hopper and Blackwell there are mixed-precision instructions that can be used (possibly in a future PR) and there definitely are such instructions on AMD GPUs (which are already supported).

[am17an]

Sorry I am not able to fix the HIP builds

[JohannesGaessler]

For now keep the problematic code in mmvf.cu as-is for HIP with a comment briefly explaining the problem.

[IMbackK]

Sorry I am not able to fix the HIP builds

ill take a look

[JohannesGaessler]

ill take a look

Would be appreciated, otherwise I would have tried to fix this myself. My preferred approach would be to merge this PR as-is and to fix the HIP issues in a follow-up PR. Is that fine with both of you?

[IMbackK]

sure, yes