CUDA error 102

[Lookforworld]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ Yes] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [Yes ] I carefully followed the README.md.
• [Yes ] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [Yes ] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Operating Environment:
wsl2(Ubuntu 22.04.2 LTS)
python:3.10.13
llama-cpp-python: 0.2.15
NVIDIA-SMI 535.43.02
Driver Version: 535.98
CUDA Version: 11.7

The installation process is fine, running with the CPU without any errors, but it often happens once it is running on the GPU

CUDA error 102 at /tmp/pip-install-7_e6fm5e/llama-cpp-python_bce7693afa4f423faad647793d7b8412/vendor/llama.cpp/ggml-cuda.cu:7100: device doesn't have valid Grid license
current device: 0

It's not 100% that there is such an error, sometimes the first few conversations are fine, and suddenly the error is reported! What is certain is that if you keep talking, this error will happen, and then the process will be terminated. And I've tested multiple different models, and the problem is the same, how can I fix it?

[abetlen]

@Lookforworld does this issue appear if you just run llama.cpp standalone?

[Lookforworld]

@Lookforworld does this issue appear if you just run llama.cpp standalone?

@abetlen Do you mean let me run the llama .cpp ? I just tested it with llama .cpp and it didn't have this problem (a total of 50 rounds of conversation, usually llama-cpp-python dialogue will appear after a few rounds). By the way， I'm still having issues running with docker.

[abetlen]

@Lookforworld yeah that's what I meant, could you post the cmake build logs for both? For llama-cpp-python use the --verbose flag when pip installing to get the full log, this may shed some light on what if anything is different during the build step.

[Lookforworld]

@Lookforworld yeah that's what I meant, could you post the cmake build logs for both? For llama-cpp-python use the --verbose flag when pip installing to get the full log, this may shed some light on what if anything is different during the build step.
@abetlen
Okey!
llama.cpp_log.txt
llama-cpp-python_log.txt