An illegal memory access was encountered at line 74 in ops.cu

[00INDEX]

This error occurs when I try to put the model on CUDA:

model.layers.cuda()

Error an illegal memory access was encountered at line 74 in file /mmfs1/gscratch/zlab/timdettmers/git/bitsandbytes/csrc/ops.cu

[00INDEX]

Update: A way to reproduce has been found. torch.set_default_dtype will make Linear4bit unusable.
Example:

import torch
import bitsandbytes as bnb
    
torch.set_default_dtype(torch.float16)
model = torch.nn.ModuleList([
    bnb.nn.Linear4bit(
        4096, 4096, False, torch.float32, False, 'fp4'
    ) for _ in range(50)
])
model.cuda() # An illegal memory access was encountered

[CRyan2016]

how to deal with this promble? i found same promble.

[jphme]

I have the exact same problem, albeit the message varies.

When trying to finetune a model i got:
Error an illegal instruction was encountered at line 74 in file /home/tim/git/bitsandbytes/csrc/ops.cu

Whent copying the above code by @00INDEX i got:
Error invalid program counter at line 74 in file /home/tim/git/bitsandbytes/csrc/ops.cu.

~$ python -m bitsandbytes

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
CUDA SETUP: CUDA runtime path found: /usr/lib/x86_64-linux-gnu/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++


+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++
/home/ubuntu/.local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so
/home/ubuntu/.local/lib/python3.9/site-packages/torch/lib/libtorch_cuda_linalg.so
/home/ubuntu/.local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda120_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda120.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda115_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda115.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda116.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda112.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda116_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda112_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113_nocublaslt.so
/home/ubuntu/.local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111.so

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
+++++++++++ /usr/lib/x86_64-linux-gnu CUDA PATHS +++++++++++
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcudart.so
/usr/lib/x86_64-linux-gnu/stubs/libcuda.so

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable


WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!

[TimDettmers]

Thank you for reporting this and creating a minimally reproducible script. I found the bug and will fix it later today or tomorrow. The problem was that setting the default type to torch.float16 lead to a bug where the absolute maximum quantization constant tensor would be in torch.float16 but the CUDA code expected a torch.float32 type.

[TimDettmers]

This has been fixed in the commit above. Upgrading via pip install -U bitsandbytes should resolve this issue. I will close this issue. Please reopen if upgrading does not resolve the error.