Can pascal ever be supported? Is it a lost cause?

[Ph0rk0z]

I have a quadro built on pascal and see a bunch of people with P40 posting here. Someone has compiled the library for windows and is using it on a 10xx series geforce for training.

I see some docs and forum post from nvidia.
https://developer.nvidia.com/blog/new-pascal-gpus-accelerate-inference-in-the-data-center/
https://forums.developer.nvidia.com/t/about-cublasgemm-int8-support/48483

But the problem is that int8 multiplication function.

Is it a hard blocking issue that we should give up hope on? The most accessible high vram GPUs are in this series.

Also, since they are indeed running this for training, does this mean other 8 bit operations are currently working and it's just the one required for language models?

[TimDettmers]

This is supported in the latest release. Please update to 0.37.0 and let me know if you are still having troubles.

[henk717]

@TimDettmers Maxwell user here (Nvidia M40), when trying to import the library I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/__init__.py", line 7, in <module>
    from .autograd._functions import (
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/autograd/__init__.py", line 1, in <module>
    from ._functions import undo_layout, get_inverse_transform_indices
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 9, in <module>
    import bitsandbytes.functional as F
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/functional.py", line 17, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 13, in <module>
    setup.run_cuda_setup()
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 92, in run_cuda_setup
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 395, in evaluate_cuda_setup
    has_cublaslt = is_cublasLt_compatible(cc)
  File "/home/henk717/koboldai/runtime/envs/koboldai/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 153, in is_cublasLt_compatible
    cuda_setup.add_log_entry("WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!", is_warning=True)
NameError: name 'cuda_setup' is not defined

[Ph0rk0z]

Thank you for fixing it so quickly. Unfortunately I have the same problem as henk does above. In addition to trying kobold, I used https://github.com/oobabooga/text-generation-webui and got the same "NameError: name 'cuda_setup' is not defined" Also this additional error: https://pastebin.com/9wySYKfv

When I tried searching this was related to it not being able to find the cuda libs?

[henk717]

The same environment I use for Kobold works on a newer GPU, so I think the not finding the libs error is deceiving.

[Ph0rk0z]

I am able to bypass the error by doing this:

def is_cublasLt_compatible(cc):
    has_cublaslt = False
    if cc is not None:
        cc_major, cc_minor = cc.split('.')
        #if int(cc_major) < 7 or (int(cc_major) == 7 and int(cc_minor) < 5):
            #CUDASetup.add_log_entry("WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!", is_warning=True)
        #else:
         #   has_cublaslt = True
    return has_cublaslt

But now I get an error like this: https://pastebin.com/H0gvKSMF when I try to generate text.

[henk717]

KoboldAI isn't going to be a good example because we do things quite different from other projects and have no official 8-bit support yet. I recommend using something else first based on more basic HF functions.

[Ph0rk0z]

Any suggestions? I also had the same failure in the textgen-webui that natively supports 8 bit. From looking it up, it's some kind of divide by zero error due to torch.

I got a new error message and attempt at generation. Turned off stuff like temperature/ top P. Acts like it generated but get nothing except exclamation marks and this.

  File "/home/mint/miniconda3/envs/textgen/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/mint/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 242, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
  File "/home/mint/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
  File "/home/mint/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
    output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x2 and 3x4096)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.