Where can I download wheel for Cuda 12.8? Trying to install llama.cpp to use with ComfyUI custom nodes.

[overallbit]

Trying to install this for comfyui, it install with pip install llama-cpp-python and works but it is loading the gguf models to System RAM, I'm trying to make it load on GPU VRAM.

Asked chatgpt for help and it want me to download prebuilt wheel for my Python version.... and run this on comfyui venv: pip install path\to\llama_cpp_python-cu128.whl

I read from some user that to install llama-cpp-python with CUDA support need to use: CMAKE_ARGS="-DGGML_CUDA=ON -DLLAMA_LLAVA=OFF" pip install llama-cpp-python

but it gives me error: 'CMAKE_ARGS' is not recognized as an internal or external command,
operable program or batch file.

RTX 2060
Cuda Toolkit 12.8
Python 3.12

[sergey21000]

but it gives me error: 'CMAKE_ARGS' is not recognized as an internal or external command, operable program or batch file.

Correct command for PowerShell

$env:CMAKE_ARGS="-DGGML_CUDA=on -DLLAMA_LLAVA=OFF"
pip install llama-cpp-python

For CMD (run each command separately)

set CMAKE_ARGS=-DGGML_CUDA=on -DLLAMA_LLAVA=OFF
pip install llama-cpp-python

To verify the installation

python -c "import llama_cpp; print(llama_cpp.llama_print_system_info())"

# Example output for CPU support:
# CPU : SSE3 = 1 | SSSE3 = 1 | ...

# Example output for CUDA support:
# ggml_cuda_init: found 1 CUDA devices: ...
# CUDA : ARCHS = 700,750,800,860,890,900 | FORCE_MMQ = 1 ...

[overallbit]

but it gives me error: 'CMAKE_ARGS' is not recognized as an internal or external command, operable program or batch file.

Correct command for PowerShell

$env:CMAKE_ARGS="-DGGML_CUDA=on -DLLAMA_LLAVA=OFF"
pip install llama-cpp-python
For CMD (run each command separately)

set CMAKE_ARGS=-DGGML_CUDA=on -DLLAMA_LLAVA=OFF
pip install llama-cpp-python
To verify the installation

python -c "import llama_cpp; print(llama_cpp.llama_print_system_info())"

Example output for CPU support:

CPU : SSE3 = 1 | SSSE3 = 1 | ...

Example output for CUDA support:

ggml_cuda_init: found 1 CUDA devices: ...

CUDA : ARCHS = 700,750,800,860,890,900 | FORCE_MMQ = 1 ...

Thanks but I solved it by using another llama-cpp fork.