[SOLVED] CUDA bitsandbytes error GPU Ubuntu

[arjunssat]

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda123.so
False
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 123
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:

1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
   CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113.
   CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.
   ================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123
python setup.py install
CUDA SETUP: Setup Failed!
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)

RuntimeError Traceback (most recent call last)
Cell In[3], line 7
5 import boto3
6 from pprint import pprint
----> 7 import bitsandbytes as bnb
8 import torch
9 import torch.nn as nn

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/init.py:6
1 # Copyright (c) Facebook, Inc. and its affiliates.
2 #
3 # This source code is licensed under the MIT license found in the
4 # LICENSE file in the root directory of this source tree.
----> 6 from . import cuda_setup, utils, research
7 from .autograd._functions import (
8 MatmulLtState,
9 bmm_cublas,
(...)
13 matmul_4bit
14 )
15 from .cextension import COMPILED_WITH_CUDA

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/research/init.py:1
----> 1 from . import nn
2 from .autograd._functions import (
3 switchback_bnb,
4 matmul_fp8_global,
5 matmul_fp8_mixed,
6 )

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/research/nn/init.py:1
----> 1 from .modules import LinearFP8Mixed, LinearFP8Global

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/research/nn/modules.py:8
5 from torch import Tensor, device, dtype, nn
7 import bitsandbytes as bnb
----> 8 from bitsandbytes.optim import GlobalOptimManager
9 from bitsandbytes.utils import OutlierTracer, find_outlier_dims
11 T = TypeVar("T", bound="torch.nn.Module")

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/init.py:6
1 # Copyright (c) Facebook, Inc. and its affiliates.
2 #
3 # This source code is licensed under the MIT license found in the
4 # LICENSE file in the root directory of this source tree.
----> 6 from bitsandbytes.cextension import COMPILED_WITH_CUDA
8 from .adagrad import Adagrad, Adagrad8bit, Adagrad32bit
9 from .adam import Adam, Adam8bit, Adam32bit, PagedAdam, PagedAdam8bit, PagedAdam32bit

File /usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py:20
18 CUDASetup.get_instance().generate_instructions()
19 CUDASetup.get_instance().print_log_stack()
---> 20 raise RuntimeError('''
21 CUDA Setup failed despite GPU being available. Please run the following command to get more information:
22
23 python -m bitsandbytes
24
25 Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
26 to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
27 and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues''')
28 lib.cadam32bit_grad_fp32 # runs on an error if the library could not be found -> COMPILED_WITH_CUDA=False
29 lib.get_context.restype = ct.c_void_p

RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
    
    
    We have installed the CUDA and CUDA toolkit both of the same version 12.3 and the error pops out 
    We have also set the path to :
    
    ENV PATH=/usr/local/cuda-12.3/bin:${PATH}
    ENV LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:${LD_LIBRARY_PATH}

Although the docker is detecting nvidia-smi and nvcc --version (12.3 each) still the error pops up.

[conorKotwasinski]

I'm having a similar issue. I've tried compiling CUDA_HOME=/usr/local/cuda-12.3 CUDA_VERSION=123 make cuda12x but I still do not have libbitsandbytes_cuda123.so. My theory is that cuda 12.3 isn't supported but any help would be appreciated

[conorKotwasinski]

run

git clone https://github.com/timdettmers/bitsandbytes.git

cd bitsandbytes

CUDA_VERSION=121 make cuda12x

pip install . 

then
python -m bitsandbytes
should work

[arjunssat]

run

git clone https://github.com/timdettmers/bitsandbytes.git

cd bitsandbytes

CUDA_VERSION=121 make cuda12x

pip install . 

then python -m bitsandbytes should work

I tried it it didnt help.

[arjunssat]

I have currently solved the issue as by setting up the following :

1. Nvidia SMI 12.2
2. Nvidia docker2
3. Nvidia toolkit 11.8
4. bits and bytes the latest version

Setting all the above in the host instance and setting the relevant paths in bashrc solved the issue.