Need help with "no kernel image is available for execution on the device"

[joshuachris2001]

by using CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip3 install llama-cpp-python --force-reinstall --upgrade --no-cache-dir to compile llama-cpp-python to use CUDA acceleration. without exporting any layers, it seems fine, but when I export 2 layers (I only have 2GB of VRAM, but that 2 layers can make a difference in speed) it just spits out: CUDA error 209 at /tmp/pip-install-atpt4j3d/llama-cpp-python_af3c026c37204b7c88c5a61a1e35af57/vendor/llama.cpp/ggml-cuda.cu:2221: no kernel image is available for execution on the device

using the llama.cpp main program seems to work fine with no apparent issues.

[joshuachris2001]

I solved it by running (TORCH_CUDA_ARCH_LIST="<Minimum compatibility on my gpu ['5.0']>" pip3 install torch --force-reinstall --upgrade --no-cache-dir --verbose)
although due to the download size of torch you might want to remove --no-cache-dir if you have to experiment or re-download again and again.