Unable to Use GPU with llama-cpp-python on Jetson Orin

[watchstep]

I'm trying to install the llama-cpp-python package to run code on NVIDIA Jetson AGX Orin (CUDA version: 12.2) using the GPU, but it's running on the CPU instead.

I attempted the following commands to enable CUDA support:

CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install --no-cache-dir llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122

However, I am still unable to get it to use the GPU.

[watchstep]

I fixed the issue by the running this command:

CUDACXX=/usr/local/cuda-12.2/bin/nvcc CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose

[Vreyka]

very thank you