CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! #175

15201308415 · 2023-03-03T03:33:07Z

C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C'), WindowsPath('/ProgramData/Anaconda3/envs/novelai/lib')}
warn(msg)
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: C:\ProgramData\Anaconda3\envs\novelai did not contain libcudart.so as expected! Searching further paths...
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
warn(msg)
C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
warn(msg)
CUDA SETUP: Loading binary C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: Loading binary C:\ProgramData\Anaconda3\envs\novelai\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...

kyuuube · 2023-03-16T12:48:15Z

I encountered the same problem

collant · 2023-03-16T22:19:27Z

Windows is not supported yet.

collant · 2023-03-16T22:43:01Z

This made it work in Windows for me: oobabooga/text-generation-webui#147 (comment)

xNul · 2023-03-19T06:13:39Z

@15201308415 I had this problem and fixed it. I wrote some installation instructions over here for my Discord bot if you want to try them.

Keith-Hon · 2023-03-29T06:13:11Z

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

Nacho48 · 2023-03-30T10:34:19Z

Uninstall Bitsandbytes and installed the one u provided.

Same: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues

Trying with the 13b model in windows 10 any insides?

Keith-Hon · 2023-04-10T02:58:49Z

Hi, reading this thread made me more confused. Where can I find the latest documentation? I use python server.py --model llama-7b-4bit --wbits 4 --pre_layer 20 from "llama-7b-4bit-128g" i got
        CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
        If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
        https://github.com/TimDettmers/bitsandbytes/issues
but CUDA is definitely installed
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 516.94       Driver Version: 516.94       CUDA Version: 11.7     |
i tried oobabooga/text-generation-webui#147 (comment) but I only have 1 computer, so breaking pyenv and my OS with global conda installs is not an option. So I tried pip install bitsandbytes-windows but i get
Loading llama-7b-4bit...
CUDA extension not installed.
Loading model ...
size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([1, 11008]).

and then it just ends. i downloaded the dll and put it in the text-generation-webui folder but there seems to be no mention of what it is beyond a dll for bitsandbytes.

could you try creating a virtual python env first before installing the dependencies.

maximus-sallam · 2023-04-11T03:56:57Z

I have fixed it by including the .dll and fixed the file path. It now can work on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

This worked for me too.

AIGC-newbee · 2023-06-01T14:13:28Z

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

Thank you very much, this really helped!

Dygitz · 2023-06-10T16:55:04Z

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

This worked for me. I was initially trying pip install bitsandbytes-windows, but that wasn't working.

IMJONEZZ · 2023-06-30T20:35:46Z

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

@Keith-Hon: any chance of using the same version numbers as the non-windows package?

Keith-Hon · 2023-07-03T12:25:15Z

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

@Keith-Hon: any chance of using the same version numbers as the non-windows package?

Yes, will merge and update later

az-123-none · 2023-07-05T12:42:38Z

It seems ctypes.CDLL(f"{DLL_FILE_PATH}") cannot locate the dll file directly in new version of Python.
I use ct.CDLL(str(binary_path), winmode=0x8) to let Windows locate it.
So just add this line before self.lib = ct.cdll.LoadLibrary(str(binary_path)) should work.

Creative-Priyamvada · 2023-07-06T16:52:41Z

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version
@Keith-Hon: any chance of using the same version numbers as the non-windows package?

Yes, will merge and update later

For now, how do i solve this error : ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

IMJONEZZ · 2023-07-06T16:57:46Z

The way I solved it was literally going to the places they specified the bitsandbytes version and typing 0.35.0 instead, there are 3 files. Dňa št 6. 7. 2023 o 10:52 Priyamvada ***@***.***> napísala(a):

ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version @Keith-Hon <https://github.com/Keith-Hon>: any chance of using the same version numbers as the non-windows package? Yes, will merge and update later For now, how do i solve this error : ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version — Reply to this email directly, view it on GitHub <#175 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALEZCMFLKCJ2K3PYBBMWD7DXO3UOJANCNFSM6AAAAAAVODJHZE> . You are receiving this because you commented.Message ID: ***@***.***>

-- Chris Brousseau (Bru-so) ***@***.*** The terms 'ethical' and 'accurate' have been used so much by the wrong people that they've lost all meaning.

brayden1moore · 2023-07-12T04:21:42Z

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

For anyone wondering, it seems the necessary files to edit are "quantization_config.py" and "bitsandbytes.py" in transformers>utils, and "modeling_utils.py" in transformers. Thanks for the tip, @IMJONEZZ !

hyao1 · 2023-07-17T13:06:04Z

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

Great! It works for my project! Thank u

ashishpatel26 · 2023-07-19T08:08:28Z

Right now it not supporting windows

Dexter-Wang · 2023-07-30T13:26:42Z

Many thanks for all the valuable advises. After I installed bitsandbytes-windows by
pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git
on windows 10, it passed the CUDA setup problem.

But however when I set use_8bit_adam = True, it shows
File ~\anaconda3\envs\Diff_Hugg\lib\site-packages\bitsandbytes\optim\optimizer.py:455 in
update_step
if state["state1"].dtype == torch.float:
KeyError: 'state1'

By the way, I also have the problem as bellow,
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
It showed up even before installed bitsandbytes-windows.git. Is this something to do with the above problem.

Thanks for the help in advance.

Dexter-Wang · 2023-08-01T00:00:28Z

After took "from diffusers.utils import is_wandb_available" away, "No module named 'triton'" gone away.

But it shows as bellow now,
Error no kernel image is available for execution on the device at line 167 in file "D:\ai\tool\bitsandbytes\csrc\ops.cu
C:\arrow\cpp\src\arrow\filesystem\s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit"

I am doing train_text_to_image in diffusers, using GTX 1080Ti GPU. Is the GPU too old for use_8bit_adam?

Dexter-Wang · 2023-08-01T03:12:56Z

On the same issue, I also try to use 2 GPUs, so I set as below,

use_8bit_adam = False
dataloader_num_workers = 1

When doing "for step, batch in enumerate(train_dataloader)", It showed

~\Anaconda\envs\Diff_Hugg\lib\multiprocessing\reduction.py:60 in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main..preprocess_train'

please help, if someone knows the problem. Thanks!

suntea233 · 2023-08-21T07:04:47Z

i met the same question.

sprog-xhy · 2023-08-31T13:55:00Z

I compile bitsandbytes from source code and solved this problem:
first: set environment variable

export PATH=/usr/local/cuda-11.6/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.6/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
`export CUDA_HOME=/usr/local/cuda-11.6``

second: download source code and compile

git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=116 make cuda11x
python setup.py install`

finally, execute :

python -m bitsandbytes

you will see these information at the end of the output:

`++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.6', '8.6', '8.6', '8.6']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
+ library is importable
+ CUDA function is callable

WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!`

my cuda is 11.6.

twers1 · 2023-10-10T10:25:29Z

I have fixed it by including the .dll and fixed the file path. It now works on windows 10.

https://github.com/Keith-Hon/bitsandbytes-windows.git

Install the bitsandbytes library by

pip install bitsandbytes-windows.

Be noted that it may not work directly with transformers library as it references the bitsandbytes package by using 'bitsandbytes' name. <= to avoid this issue, you could directly install from the git repo

pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

I did as you said, now I have another error:

n post_init
raise ValueError(
ValueError: 4 bit quantization requires bitsandbytes>=0.39.0 - please upgrade your bitsandbytes version

swumagic · 2023-11-10T06:43:10Z

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

twers1 · 2023-11-10T08:28:11Z

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

yep, i did so

swumagic · 2023-11-12T02:24:32Z

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

twers1 · 2023-11-12T11:43:06Z

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

I did everything as you said, but when I launch my console application with AI, it doesn’t give me an answer

MuhammadShifa · 2023-12-14T18:30:19Z

Try this: it's worked for me in Window 10.

pip uninstall bitsandbytes
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.0-py3-none-win_amd64.whl

DarkAlchy · 2023-12-23T01:40:27Z

(venv) F:\eib>pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl
Collecting bitsandbytes==0.41.1
Downloading https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl (152.7 MB)
---------------------------------------- 152.7/152.7 MB 6.8 MB/s eta 0:00:00
Requirement already satisfied: scipy in f:\esd\venv\lib\site-packages (from bitsandbytes==0.41.1) (1.11.4)
Requirement already satisfied: numpy<1.28.0,>=1.21.6 in f:\esd\venv\lib\site-packages (from scipy->bitsandbytes==0.41.1) (1.26.2)
Installing collected packages: bitsandbytes
Successfully installed bitsandbytes-0.41.1

Titus-von-Koeller · 2024-03-15T18:44:47Z

Dear all,

Since the current release (last week, 8th of March) we now have official support for Windows 🎉 (which we did not have before) via

pip install bitsandbytes>=0.43.0

We're closing all old Windows issues and are asking everyone to try installing with this new version as outlined above and validate the install with python -m bitsandbytes which should spit out a bunch of stuff and then SUCCESS. Please let us know if everything worked correctly in this new umbrella / catch-all issue. Thanks 🤗

GreenGarnets mentioned this issue Mar 4, 2023

Support for LLaMA models oobabooga/text-generation-webui#147

Closed

Keith-Hon mentioned this issue Mar 31, 2023

is it work on Win platform? facebookresearch/bitsandbytes#7

Open

zcsctc mentioned this issue May 26, 2023

Support for QLoRA oobabooga/text-generation-webui#2352

Closed

afcarzero1 mentioned this issue Jul 24, 2023

Bitsandbytes does not work in Windows PromtEngineer/localGPT#265

Open

Gilnore mentioned this issue Aug 17, 2023

needing a linux file on windows, recommended fix in bug report uses linux commands somehow. #713

Closed

hzho2000 mentioned this issue Oct 15, 2023

PPO训练时不能使用QLoRA吗？ hiyouga/LLaMA-Factory#1185

Closed

hzho2000 mentioned this issue Oct 15, 2023

无法使用QLoRA来训练奖励模型以及进行PPO训练。 hiyouga/LLaMA-Factory#1189

Closed

Titus-von-Koeller closed this as completed Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! #175

CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! #175

15201308415 commented Mar 3, 2023

kyuuube commented Mar 16, 2023

collant commented Mar 16, 2023

collant commented Mar 16, 2023

xNul commented Mar 19, 2023

Keith-Hon commented Mar 29, 2023 •

edited

Loading

Nacho48 commented Mar 30, 2023

Keith-Hon commented Apr 10, 2023

maximus-sallam commented Apr 11, 2023

AIGC-newbee commented Jun 1, 2023

Dygitz commented Jun 10, 2023

IMJONEZZ commented Jun 30, 2023

Keith-Hon commented Jul 3, 2023

az-123-none commented Jul 5, 2023 •

edited

Loading

Creative-Priyamvada commented Jul 6, 2023

IMJONEZZ commented Jul 6, 2023 via email

brayden1moore commented Jul 12, 2023 •

edited

Loading

hyao1 commented Jul 17, 2023

ashishpatel26 commented Jul 19, 2023

Dexter-Wang commented Jul 30, 2023

Dexter-Wang commented Aug 1, 2023

Dexter-Wang commented Aug 1, 2023

suntea233 commented Aug 21, 2023

sprog-xhy commented Aug 31, 2023 •

edited

Loading

twers1 commented Oct 10, 2023

swumagic commented Nov 10, 2023

twers1 commented Nov 10, 2023

swumagic commented Nov 12, 2023

twers1 commented Nov 12, 2023

MuhammadShifa commented Dec 14, 2023

DarkAlchy commented Dec 23, 2023

Titus-von-Koeller commented Mar 15, 2024

CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! #175

CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! #175

Comments

15201308415 commented Mar 3, 2023

kyuuube commented Mar 16, 2023

collant commented Mar 16, 2023

collant commented Mar 16, 2023

xNul commented Mar 19, 2023

Keith-Hon commented Mar 29, 2023 • edited Loading

Nacho48 commented Mar 30, 2023

Keith-Hon commented Apr 10, 2023

maximus-sallam commented Apr 11, 2023

AIGC-newbee commented Jun 1, 2023

Dygitz commented Jun 10, 2023

IMJONEZZ commented Jun 30, 2023

Keith-Hon commented Jul 3, 2023

az-123-none commented Jul 5, 2023 • edited Loading

Creative-Priyamvada commented Jul 6, 2023

IMJONEZZ commented Jul 6, 2023 via email

brayden1moore commented Jul 12, 2023 • edited Loading

hyao1 commented Jul 17, 2023

ashishpatel26 commented Jul 19, 2023

Dexter-Wang commented Jul 30, 2023

Dexter-Wang commented Aug 1, 2023

Dexter-Wang commented Aug 1, 2023

suntea233 commented Aug 21, 2023

sprog-xhy commented Aug 31, 2023 • edited Loading

twers1 commented Oct 10, 2023

swumagic commented Nov 10, 2023

twers1 commented Nov 10, 2023

swumagic commented Nov 12, 2023

twers1 commented Nov 12, 2023

MuhammadShifa commented Dec 14, 2023

DarkAlchy commented Dec 23, 2023

Titus-von-Koeller commented Mar 15, 2024

Keith-Hon commented Mar 29, 2023 •

edited

Loading

az-123-none commented Jul 5, 2023 •

edited

Loading

brayden1moore commented Jul 12, 2023 •

edited

Loading

sprog-xhy commented Aug 31, 2023 •

edited

Loading