RuntimeError: CUDA Setup failed despite correct environment paths on Windows

[klei30]

System Info

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

Reproduction

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('C')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.9.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Exception in thread Thread-8 (run_exp):
Traceback (most recent call last):
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\utils\import_utils.py", line 1382, in get_module
return importlib.import_module("." + module_name, self.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\importlib_init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in find_and_load
File "", line 1147, in find_and_load_unlocked
File "", line 690, in load_unlocked
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\integrations\bitsandbytes.py", line 11, in
import bitsandbytes as bnb
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes_init.py", line 6, in
from . import cuda_setup, utils, research
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\research_init.py", line 1, in
from . import nn
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\optim_init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\bitsandbytes\cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\PC\Desktop\test\LLaMA-Factory\src\llmtuner\train\tuner.py", line 29, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "C:\Users\PC\Desktop\test\LLaMA-Factory\src\llmtuner\train\sft\workflow.py", line 29, in run_sft
model, tokenizer = load_model_and_tokenizer(model_args, finetuning_args, training_args.do_train)
File "C:\Users\PC\Desktop\test\LLaMA-Factory\src\llmtuner\model\loader.py", line 87, in load_model_and_tokenizer
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\models\auto\auto_factory.py", line 561, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\modeling_utils.py", line 3476, in from_pretrained
from .integrations import get_keys_to_not_convert, replace_with_bnb_linear
File "", line 1229, in _handle_fromlist
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\utils\import_utils.py", line 1372, in getattr
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\PC\anaconda3\envs\llama_factory\Lib\site-packages\transformers\utils\import_utils.py", line 1384, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Keyboard interruption in main thread... closing server.

Expected behavior

Issue Summary:

I am encountering a RuntimeError indicating that bitsandbytes cannot find the CUDA runtime library despite the CUDA path being correctly set in the environment variables on Windows.

System Information:

• Operating System: Windows 10/11 (Specify version)
• CUDA Version: 11.8
• PyTorch Version: (Run import torch; print(torch.__version__) in Python and paste the output here)
• Python Version: (Include your Python version)
• bitsandbytes Version: (Include the version of bitsandbytes you're using)

Steps to Reproduce:

1. Set the CUDA Toolkit path in the environment variables.
2. Install bitsandbytes using pip.
3. Run python -m bitsandbytes and encounter the error.

Current Behavior:

When running python -m bitsandbytes, I receive a warning that libcudart.so is not found in any environmental path, and the main CUDA runtime library is not detected. The error message suggests adding the CUDA library path to the LD_LIBRARY_PATH variable, which is not applicable on Windows as it's a Linux-specific environment variable.

Expected Behavior:

bitsandbytes should recognize the CUDA installation and not raise an error about missing runtime libraries, as they exist in the specified path.

Troubleshooting Steps Taken:

• Verified that the CUDA Toolkit binaries are present in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin.
• Restarted the system after setting the environment variables.
• Tried adding the CUDA bin directory to the os.add_dll_directory in Python before importing bitsandbytes.
• Checked administrative privileges for the command prompt and Python environment.

[Entretoize]

Any help ?

[0xC0DEACE]

Thats problem with library bitsandbytes on windows.
In my case help to me remove library and manualy install from https://jllllll.github.io/bitsandbytes-windows-webui/bitsandbytes/

In my case help version 0.41.1 with cuda 12.1

[Entretoize]

That works ! I spent several with that errors with no success. Thank you !

[criskb]

Thats problem with library bitsandbytes on windows. In my case help to me remove library and manualy install from https://jllllll.github.io/bitsandbytes-windows-webui/bitsandbytes/

In my case help version 0.41.1 with cuda 12.1

can comfirm this also helped me out, ty ty
for anyone who needs help just download the whl and drop it into your comfyui folder then cd and do pip install (name_of_file.whl)

[Chawklates]

Thats problem with library bitsandbytes on windows. In my case help to me remove library and manualy install from https://jllllll.github.io/bitsandbytes-windows-webui/bitsandbytes/
In my case help version 0.41.1 with cuda 12.1

can comfirm this also helped me out, ty ty for anyone who needs help just download the whl and drop it into your comfyui folder then cd and do pip install (name_of_file.whl)

+1 this helped, made an account to say thank you. For me, I chose "bitsandbytes-0.41.1-py3-none-win_amd64.whl"

[Titus-von-Koeller]

Dear all,

Since the current release (last week, 8th of March) we now have official support for Windows 🎉 (which we did not have before) via

pip install bitsandbytes>=0.43.0

We're closing all old Windows issues and are asking everyone to try installing with this new version as outlined above and validate the install with python -m bitsandbytes which should spit out a bunch of stuff and then SUCCESS. Please let us know if everything worked correctly in this new umbrella / catch-all issue. Thanks 🤗