Update llama.cpp for new CUDA support

[amrrs]

Thanks very much for the project, any plans to enable the recently merged CUDA support?

ggml-org/llama.cpp#1827

[NeoAnthropocene]

Thanks very much for the project, any plans to enable the recently merged CUDA support?

ggerganov/llama.cpp#1827

I am also looking for that.
The speed boost is promising, I would like to try it asap :)

[gaby]

@abetlen Is this supported in 0.1.63 ?

[abetlen]

@gaby @NeoAnthropocene yes should've been included in 0.1.63. I also just published 0.1.64 with some additional fixes from llama.cpp, let me know if that works!

[gaby]

@gaby @NeoAnthropocene yes should've been included in 0.1.63. I also just published 0.1.64 with some additional fixes from llama.cpp, let me know if that works!

@amrrs Thanks, i didnt get the email notification because its a git tag, not a Github release 😂

[matib99]

Should it work now? I installed v0.1.64 with cuBLAS, and set n_gpu_layers parameter and it still runs only on CPU. I'm on Ubuntu with rtx3080, torch.cuda.is_available() returns true.

[berkut1]

On the RTX 2080 I got +~10% performance (13b models) after the upgrade but can only load 50-60% of the GPU layers due to VRAM. (I have a large context min. 400 tokens)

[NeoAnthropocene]

@gaby @NeoAnthropocene yes should've been included in 0.1.63. I also just published 0.1.64 with some additional fixes from llama.cpp, let me know if that works!

@abetlen Thx for the feedback.
I uninstalled my previous llama-cpp-python==0.1.63 model and now I can't install it back with BLAS=1

I've read everything on the internet but I somehow it builds without CUDA support. That is strange because I couldn't remember about any change that I've done before with the CUDA toolkit installation on my system.

[bryanhpchiang]

was able to get CUDA accl working with this

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose