Description:
According to the documentation, setting n_gpu_layers determines the number of layers offloaded to the GPU (-ngl). If -1, all layers should be offloaded. However, even after setting n_gpu_layers=-1, the program runs entirely on the CPU.
Steps to Reproduce
- Install
llama-cpp-python with GPU support using:
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
- Load the model using
llama_cpp.Llama with n_gpu_layers=-1.
- Run
create_chat_completion on an image input.
- Observe that the execution is still CPU-bound.
Expected Behavior
The model should offload computation to the GPU when n_gpu_layers is set to -1.
Actual Behavior
The model runs entirely on the CPU despite setting n_gpu_layers=-1.
Code Snippet
from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava16ChatHandler
import base64
def image_to_base64_data_uri(file_path):
with open(file_path, "rb") as img_file:
base64_data = base64.b64encode(img_file.read()).decode("utf-8")
return f"data:image/png;base64,{base64_data}"
file_path = "./images/image (1).jpg"
data_uri = image_to_base64_data_uri(file_path)
chat_handler = Llava16ChatHandler(
clip_model_path="C:/Users/eyob9/Desktop/myModels/llava/mmproj-model-f16.gguf"
)
llm = Llama(
model_path="C:/Users/eyob9/Desktop/myModels/llava/llava.gguf",
chat_handler=chat_handler,
n_ctx=4096,
n_gpu_layers=-1, # Expected to offload all layers to GPU
)
llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are an assistant who perfectly describes images.",
},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": data_uri}},
{"type": "text", "text": "Describe this image in detail please."},
],
},
]
)Environment Details
- OS: Windows 11
- Python Version: Python 3.12.6
- llama-cpp Version: llama_cpp_python 0.3.7
- GPU Model: NVIDIA RTX A500 Laptop GPU (4GB VRAM)
- CUDA Installed: CUDA 12.7
Any insights into why GPU offloading isn't working would be greatly appreciated!
Description:
According to the documentation, setting
n_gpu_layersdetermines the number of layers offloaded to the GPU (-ngl). If-1, all layers should be offloaded. However, even after settingn_gpu_layers=-1, the program runs entirely on the CPU.Steps to Reproduce
llama-cpp-pythonwith GPU support using:CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-pythonllama_cpp.Llamawithn_gpu_layers=-1.create_chat_completionon an image input.Expected Behavior
The model should offload computation to the GPU when
n_gpu_layersis set to-1.Actual Behavior
The model runs entirely on the CPU despite setting
n_gpu_layers=-1.Code Snippet
Environment Details
Any insights into why GPU offloading isn't working would be greatly appreciated!