Using GGUF models efficiently

[N1h1lv5]

Hello all,

So today finally we have GGUF support ! Quite exciting and many thanks to @PromtEngineer !

At the moment I run the default model llama 7b with --device_type cuda, and I can see some GPU memory being used but the processing at the moment goes only to the CPU. Does anyone have experience with GGUF's + GPU ?

[N1h1lv5]

This was the solution for me, I am running windows with a conda env :

1. set the environment variable properly :

$Env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
$Env:FORCE_CMAKE="1"

2. check that it works

echo $Env:CMAKE_ARGS

3. uninstall previous version of llama-cpp-python

pip uninstall llama-cpp-python

4. install the proper version :

pip install llama-cpp-python==0.1.83 --no-cache-dir

Now GPU + CPU works toether. Thanks @PromtEngineer

[nekiee13]

What I don't understand is - does LocalGPT automatically determine a number of layers that can be offloaded to GPU?
Or am I supposed to set a number of offloaded layers somewhere?

[HWiese1980]

Is this answer still valid?