More ggml cuda kernels

[LaurentMazare]

These kernels are experimental for now, they can be tried out in the quantized example via the --fast-cuda flag.

As a benchmark,

cargo run --example quantized --profile=release-with-debug --features cuda -- --prompt "Building a website can be done in 10 simple steps:\nStep 1:" -n 100 --which 7b-mistral --fast-cuda

This goes from 33 token/s to 63 token/s with the new kernels, whereas llama.cpp is at ~55 token/s using the same model (mistral-7b-v0.1.Q4_K_S.gguf from TheBloke).

[EricLBuehler]

@LaurentMazare , this is a funny coincidence, I was just about to open a PR for this. Thanks for the work!

I observed a 1.9x speed decrease when forcing llama.cpp to also use dmmv, and it was slower than mistral.rs then. I think after using mmvq QMatMul will be much faster.

[EricLBuehler]

Could this maybe be an environment variable or ideally influenced by the compute cap? The __dp4a intrinsic is only supported on CC>610. I'm not sure what the current minimum compute cap for Candle is, but it seems like it would be better to avoid increasing it?

[EricLBuehler]

Looks like __nv_bfloat16 is CC>800, so this is not a problem.

[LaurentMazare]

I observed a 1.9x speed decrease when forcing llama.cpp to also use dmmv, and it was slower than mistral.rs then. I think after using mmvq QMatMul will be much faster.

Seems pretty much in line with what I'm seeing. In my early testing I get a speedup from 35 token/s to 60.8 token/s vs 55.0 token/s for llama.cpp, not sure why it actually outperforms it but that seems pretty promising.

[EricLBuehler]

not sure why it actually outperforms it but that seems pretty promising.

It seems plausible that the fact that dmmv needs to call a dequantize method many times per matmul may be the cause of this. Also, the __dp4a intrinsic usage in mmvq may be playing a role.

[LaurentMazare]

Well it's pretty clear why it outperforms the dmmv version, q8_1 is much more efficient than using a f32, but I'm more wondering how it could outperform the llama.cpp implementation overall.

[EricLBuehler]

Thank you for the excellent work!