CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!

[15201308415]

C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C'), WindowsPath('/ProgramData/Anaconda3/envs/novelai/lib')}
warn(msg)
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: C:\ProgramData\Anaconda3\envs\novelai did not contain libcudart.so as expected! Searching further paths...
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
warn(msg)
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
warn(msg)
CUDA SETUP: Loading binary C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: Loading binary C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...

[kyuuube]

I encountered the same problem

[collant]

Windows is not supported yet.

[collant]

This made it work in Windows for me: oobabooga/text-generation-webui#147 (comment)

[xNul]

@15201308415 I had this problem and fixed it. I wrote some installation instructions over here for my Discord bot if you want to try them.

[Keith-Hon]

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

[Nacho48]

Uninstall Bitsandbytes and installed the one u provided.

Same: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues

Trying with the 13b model in windows 10 any insides?

[Keith-Hon]

Hi, reading this thread made me more confused. Where can I find the latest documentation? I use python server.py --model llama-7b-4bit --wbits 4 --pre_layer 20 from "llama-7b-4bit-128g" i got

        CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
        If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
        https://github.com/TimDettmers/bitsandbytes/issues

but CUDA is definitely installed

nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 516.94       Driver Version: 516.94       CUDA Version: 11.7     |

i tried oobabooga/text-generation-webui#147 (comment) but I only have 1 computer, so breaking pyenv and my OS with global conda installs is not an option. So I tried pip install bitsandbytes-windows but i get

Loading llama-7b-4bit...
CUDA extension not installed.
Loading model ...

size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([1, 11008]).

and then it just ends. i downloaded the dll and put it in the text-generation-webui folder but there seems to be no mention of what it is beyond a dll for bitsandbytes.

could you try creating a virtual python env first before installing the dependencies.

[maximus-sallam]

I have fixed it by including the .dll and fixed the file path. It now can work on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

This worked for me too.

[AIGC-newbee]

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

Thank you very much, this really helped!

[Dygitz]

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

This worked for me. I was initially trying pip install bitsandbytes-windows, but that wasn't working.

[IMJONEZZ]

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

@Keith-Hon: any chance of using the same version numbers as the non-windows package?

[Keith-Hon]

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

@Keith-Hon: any chance of using the same version numbers as the non-windows package?

Yes, will merge and update later