CUDA error: the provided PTX was compiled with an unsupported toolchain.

[anamariaUIC]

Hi,

I installed privateGPT following these steps:

git clone https://github.com/imartinez/privateGPT.git 

cd privateGPT

conda create -n privategpt python=3.11

conda activate privategpt

#loading modules

module load  Anaconda3

module load CUDA/12.2.2

#install poetry 

pip install poetry

# Install dependencies
poetry install --with ui,local

# Download Embedding and LLM models. all customization can be done in setup

#used this model: mistral-7b-instruct-v0.2.Q6_K.gguf

poetry run python scripts/setup

CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

PGPT_PROFILES=local make run

I run this on: NVIDIA A100-SXM4-80GB

so my gradio UI starts, I upload my document to query on it and when I ask my question I am getting this error message:

CUDA error: the provided PTX was compiled with an unsupported toolchain.
current device: 0, in function ggml_cuda_op_flatten at /tmp/pip-install-vxbb3kr9/llama-cpp-python_384550df678f489e99c439d8354165a3/vendor/llama.cpp/ggml-cuda.cu:9136

In attach you can see my version of CUDA diver and my toolkit. Can you please let me know what is the issue here and how to resolve it?

Thanks

[SuperSonnix71]

Some other guy had the same issue. Which was solved

https://discord.com/channels/1164200432894234644/1164200433779212400/1202511250760798318

[anamariaUIC]

I don’t see that post. Can you pleaser copy the solution here? On Feb 19, 2024, at 2:14 PM, Sonny ***@***.******@***.***>> wrote: Some other guy had the same issue. Which was solved https://discord.com/channels/1164200432894234644/1164200433779212400/1202511250760798318 — Reply to this email directly, view it on GitHub<#1615 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUHBH5FHZO3JSZSUMLBSB63YUOXBBAVCNFSM6AAAAABDM6AT5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGEYDMNJQGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[SuperSonnix71]

The CUDA error: unsuported toolchain is suggesting that the PTX Parallel Thread Execution code used by CUDA for kernels was compiled with a version of the toolchain (compiler, linkers, etc ') that is not compatible with the CUDA version you are using. so either your CUDA toolkit on your system is older or newer that the one used to compile the PTX code.

**solution to your problem is to Update NVIDIA Driver so that your CUDA Toolkit version matches more closely with the CUDA runtime version **

[TonyIvanova]

I'm having the same issue.

~/privateGPT$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Tue_Feb_27_16:19:38_PST_2024 Cuda compilation tools, release 12.4, V12.4.99 Build cuda_12.4.r12.4/compiler.33961263_0

[TonyIvanova]

OK, you literally need to update your drivers.
If you're using WSL like I do - just go to (in Windows) https://www.nvidia.com/en-us/geforce/drivers/, download automatic driver update etc.
Don't forget to restart after update!