Support for CUDA 12.3

[SSusantAchary]

System Info

For Windows when are you planning for release of BitsAndBytes compatibility for CUDA12.3

Reproduction

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('/Users/susan/miniconda3/envs/ml_lab/lib'), WindowsPath('C')}
The following directories listed in your path were found to be non-existent: {WindowsPath('vs/workbench/api/node/extensionHostProcess')}
The following directories listed in your path were found to be non-existent: {WindowsPath('module'), WindowsPath('//matplotlib_inline.backend_inline')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary c:\Users\susan\miniconda3\envs\ml_lab\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.so...
[WinError 193] %1 is not a valid Win32 application
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
c:\Users\susan\miniconda3\envs\ml_lab\Lib\site-packages\bitsandbytes\cuda_setup\main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run

Expected behavior

by when can i expect for CUDA support for 12.3

[younesbelkada]

Hi @SSusantAchary
Thanks for the issue - for windows support you should build bnb from source, see the instructions here: https://huggingface.co/docs/bitsandbytes/main/en/installation?OS+system=Windows I think CUDA 12.1 is already supported as we test it in our workflows:

bitsandbytes/.github/workflows/cmake.yml

Line 23 in ea77bfe

cuda-version: ['11.8', '12.1']

but I don't see why CUDA 12.3 shouldn't be supported, @SSusantAchary can you try out the instructions for installing bnb for windows?

[wkpark]

Hi @SSusantAchary Thanks for the issue - for windows support you should build bnb from source, see the instructions here: https://huggingface.co/docs/bitsandbytes/main/en/installation?OS+system=Windows I think CUDA 12.1 is already supported as we test it in our workflows:

or you can download latest windows wheel from the following: (github login required)
https://github.com/TimDettmers/bitsandbytes/actions/runs/7776527957

[matthewdouglas]

but I don't see why CUDA 12.3 shouldn't be supported

For what it's worth, I've been able to successfully build on Windows with CUDA toolkits for 12.0.1, 12.1.1, 12.2.2, and 12.3.2. Most unit tests pass. It should at least work for inference, but I'm not sure on training as I haven't tried that yet.

[rkaunismaa]

I am getting the same error with Ubuntu 22.04.3 LTS ...

nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.default.path')}
The following directories listed in your path were found to be non-existent: {PosixPath('0'), PosixPath('1')}
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.mandatory.path')}
The following directories listed in your path were found to be non-existent: {PosixPath('@/tmp/.ICE-unix/2836,unix/KAUWITB'), PosixPath('local/KAUWITB')}
The following directories listed in your path were found to be non-existent: {PosixPath('vs/workbench/api/node/extensionHostProcess')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-ubuntu')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-ubuntu')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/rob/miniforge3/envs/mylocalseer/etc/xml/catalog file'), PosixPath('file')}
The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 8.9.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Required library version not found: libbitsandbytes_cuda121.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:

1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release!
   CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113.
   CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.
   ================================================================================

[matthewdouglas]

@rkaunismaa That seems like a separate issue. I suspect you have an older version of bitsandbytes installed and would suggest upgrading to the latest release. The cu121 build was not included in the wheels until bitsandbytes>=0.38.0.

[SSusantAchary]

Thanks