Hugging Face peft and bitsandbytes mismatch

[jasonmcaffee]

Describe the bug

This just happened overnight. Yesterday things were working great on google colab, but this morning I ran the same notebook and ran into an error:

AttributeError: module 'bitsandbytes.nn' has no attribute 'Linear4bit'

I did some digging, and found that:
HuggingFace Peft has just added 4-bit QLoRA via bitsandbytes
huggingface/peft#476

Since the requirements use latest git peft, but are locked to bitsandbytes version 0.38.1, this error occurs due to bitsandbytes being outdated.

A temporary workaround is change requirements.txt to lock peft to a commit that was done a week ago, and does not include this change.

accelerate==0.19.0
colorama
datasets
flexgen==0.1.7
gradio_client==0.2.5
gradio==3.31.0
markdown
numpy
pandas
Pillow>=9.5.0
pyyaml
requests
rwkv==0.7.3
safetensors==0.3.1
sentencepiece
tqdm
git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa
transformers==4.29.1
bitsandbytes==0.38.1; platform_system != "Windows"
llama-cpp-python==0.1.51; platform_system != "Windows"
https://github.com/abetlen/llama-cpp-python/releases/download/v0.1.51/llama_cpp_python-0.1.51-cp310-cp310-win_amd64.whl; platform_system == "Windows"

You can simply add this echo command before installing requirements.txt:

!echo -e "accelerate==0.19.0\ncolorama\ndatasets\nflexgen==0.1.7\ngradio_client==0.2.5\ngradio==3.31.0\nmarkdown\nnumpy\npandas\nPillow>=9.5.0\npyyaml\nrequests\nrwkv==0.7.3\nsafetensors==0.3.1\nsentencepiece\ntqdm\ngit+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa\ntransformers==4.29.1\nbitsandbytes==0.38.1; platform_system != \"Windows\"\nllama-cpp-python==0.1.51; platform_system != \"Windows\"\nhttps://github.com/abetlen/llama-cpp-python/releases/download/v0.1.51/llama_cpp_python-0.1.51-cp310-cp310-win_amd64.whl; platform_system == \"Windows\"" > requirements.txt
!pip install -r requirements.txt

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

In google colab, use a GPU instance and run the following

!git clone https://github.com/oobabooga/text-generation-webui
%cd text-generation-webui/
!pip install -r requirements.txt
!python server.py

Screenshot

No response

Logs

Right after this log

TF-TRT Warning: Could not find TensorRT
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so

You will get the error:

AttributeError: module 'bitsandbytes.nn' has no attribute 'Linear4bit'

### System Info

```shell
Google colab notebook running GPU: T4

[riyazweb]

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/text-generation-webui/server.py:47 in │
│ │
│ 44 from PIL import Image │
│ 45 │
│ 46 import modules.extensions as extensions_module │
│ ❱ 47 from modules import chat, shared, training, ui, utils │
│ 48 from modules.extensions import apply_extensions │
│ 49 from modules.html_generator import chat_html_wrapper │
│ 50 from modules.LoRA import add_lora_to_model │
│ │
│ /content/text-generation-webui/modules/training.py:14 in │
│ │
│ 11 import torch │
│ 12 import transformers │
│ 13 from datasets import Dataset, load_dataset │
│ ❱ 14 from peft import (LoraConfig, get_peft_model, prepare_model_for_int8_t │
│ 15 │ │ │ │ set_peft_model_state_dict) │
│ 16 │
│ 17 from modules import shared, ui, utils │
│ │
│ /usr/local/lib/python3.10/dist-packages/peft/init.py:22 in │
│ │
│ 19 │
│ 20 version = "0.4.0.dev0" │
│ 21 │
│ ❱ 22 from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CON │
│ 23 from .peft_model import ( │
│ 24 │ PeftModel, │
│ 25 │ PeftModelForCausalLM, │
│ │
│ /usr/local/lib/python3.10/dist-packages/peft/mapping.py:16 in │
│ │
│ 13 # See the License for the specific language governing permissions and │
│ 14 # limitations under the License. │
│ 15 │
│ ❱ 16 from .peft_model import ( │
│ 17 │ PeftModel, │
│ 18 │ PeftModelForCausalLM, │
│ 19 │ PeftModelForSeq2SeqLM, │
│ │
│ /usr/local/lib/python3.10/dist-packages/peft/peft_model.py:31 in │
│ │
│ 28 from transformers.modeling_outputs import SequenceClassifierOutput, T │
│ 29 from transformers.utils import PushToHubMixin │
│ 30 │
│ ❱ 31 from .tuners import ( │
│ 32 │ AdaLoraModel, │
│ 33 │ AdaptionPromptModel, │
│ 34 │ LoraModel, │
│ │
│ /usr/local/lib/python3.10/dist-packages/peft/tuners/init.py:21 in │
│ │
│ │
│ 18 # limitations under the License. │
│ 19 │
│ 20 from .adaption_prompt import AdaptionPromptConfig, AdaptionPromptModel │
│ ❱ 21 from .lora import LoraConfig, LoraModel │
│ 22 from .adalora import AdaLoraConfig, AdaLoraModel │
│ 23 from .p_tuning import PromptEncoder, PromptEncoderConfig, PromptEncoder │
│ 24 from .prefix_tuning import PrefixEncoder, PrefixTuningConfig │
│ │
│ /usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py:735 in │
│ │
│ 732 │ │ │ │ result += output │
│ 733 │ │ │ return result │
│ 734 │ │
│ ❱ 735 │ class Linear4bit(bnb.nn.Linear4bit, LoraLayer): │
│ 736 │ │ # Lora implemented in a dense layer │
│ 737 │ │ def init( │
│ 738 │ │ │ self, │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: module 'bitsandbytes.nn' has no attribute 'Linear4bit'

[riyazweb]

please fix this error

[jllllll]

The 4bit bitsandbytes update is currently in closed beta. It will be some time before that version is publicly released.

[FeuerDrachenEinhorn]

can you tell me where I should paste this? when using it within the "start_windows.bat" I get an syntax error

You can simply add this echo command before installing requirements.txt:
!echo -e "accelerate==0.19.0\ncolorama\ndatasets\nflexgen==0.1.7\ngradio_client==0.2.5\ngradio==3.31.0\nmarkdown\nnumpy\npandas\nPillow>=9.5.0\npyyaml\nrequests\nrwkv==0.7.3\nsafetensors==0.3.1\nsentencepiece\ntqdm\ngit+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa\ntransformers==4.29.1\nbitsandbytes==0.38.1; platform_system != "Windows"\nllama-cpp-python==0.1.51; platform_system != "Windows"\nhttps://github.com/abetlen/llama-cpp-python/releases/download/v0.1.51/llama_cpp_python-0.1.51-cp310-cp310-win_amd64.whl; platform_system == "Windows"" > requirements.txt
!pip install -r requirements.txt

[jllllll]

@FeuerDrachenEinhorn You can just install normally and then afterward open cmd_windows.bat and enter this command:

python -m pip install git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa --force-reinstall --no-deps

[jasonmcaffee]

Sorry @FeuerDrachenEinhorn, but my command is tailored for a linux box on Google colab. Looks like @jllllll command should do the trick for you.

[FuhrerStein]

@jllllll 's answer works for me. Thanks!
Windows 11.

[hennypurwadi]

Thanks. This is works for me 👍
!pip install git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa --force-reinstall --no-deps

[ai-joe-git]

@FeuerDrachenEinhorn You can just install normally and then afterward open cmd_windows.bat and enter this command:

python -m pip install git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa --force-reinstall --no-deps

Worked thanks !

[jllllll]

This is now fixed: 2aa01e2

[JoeyTPChou]

python -m pip install git+https://github.com/huggingface/peft@27af2198225cbb9e049f548440f2bd0fba2204aa --force-reinstall --no-deps

Hi I am still seeing the same error. How long should we wait for the fixes to be public?

[Erthar]

This is now fixed: 2aa01e2

Thanks @jllllll! This fixed the bitsandbytes issue.

A new one came up, however.

I'm not sure if this is related or something else entirely, but I didn't get this error last time I ran my notebook. Apologies if it's a separate issue.

INFO:Loaded the model in 44.69 seconds.

INFO:server listening on 127.0.0.1:5005
Traceback (most recent call last):
  File "/content/Colab-TextGen/extensions/api/util.py", line 61, in _start_cloudflared
    public_url = _run_cloudflared(port, port + 1)
  File "/usr/local/lib/python3.10/dist-packages/flask_cloudflared.py", line 69, in _run_cloudflared
    _download_cloudflared(cloudflared_path, command)
  File "/usr/local/lib/python3.10/dist-packages/flask_cloudflared.py", line 101, in _download_cloudflared
    update_cloudflared = subprocess.Popen([(cloudflared_path+'/'+command), 'update'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/tmp/cloudflared-linux-amd64'
Traceback (most recent call last):
  File "/content/Colab-TextGen/extensions/api/util.py", line 61, in _start_cloudflared
    public_url = _run_cloudflared(port, port + 1)
  File "/usr/local/lib/python3.10/dist-packages/flask_cloudflared.py", line 69, in _run_cloudflared
    _download_cloudflared(cloudflared_path, command)
  File "/usr/local/lib/python3.10/dist-packages/flask_cloudflared.py", line 101, in _download_cloudflared
    update_cloudflared = subprocess.Popen([(cloudflared_path+'/'+command), 'update'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/tmp/cloudflared-linux-amd64'
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/Colab-TextGen/server.py:1011 in <module>                            │
│                                                                              │
│   1008 │   │   })                                                            │
│   1009 │                                                                     │
│   1010 │   # Launch the web UI                                               │
│ ❱ 1011 │   create_interface()                                                │
│   1012 │   while True:                                                       │
│   1013 │   │   time.sleep(0.5)                                               │
│   1014 │   │   if shared.need_restart:                                       │
│                                                                              │
│ /content/Colab-TextGen/server.py:922 in create_interface                     │
│                                                                              │
│    919 │   if shared.args.listen:                                            │
│    920 │   │   shared.gradio['interface'].launch(prevent_thread_lock=True, s │
│    921 │   else:                                                             │
│ ❱  922 │   │   shared.gradio['interface'].launch(prevent_thread_lock=True, s │
│    923                                                                       │
│    924                                                                       │
│    925 if __name__ == "__main__":                                            │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/gradio/blocks.py:1727 in launch      │
│                                                                              │
│   1724 │   │   if not isinstance(self.blocked_paths, list):                  │
│   1725 │   │   │   raise ValueError("`blocked_paths` must be a list of direc │
│   1726 │   │                                                                 │
│ ❱ 1727 │   │   self.validate_queue_settings()                                │
│   1728 │   │                                                                 │
│   1729 │   │   self.config = self.get_config_file()                          │
│   1730 │   │   self.max_threads = max(                                       │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/gradio/blocks.py:1575 in             │
│ validate_queue_settings                                                      │
│                                                                              │
│   1572 │   │   │   │   )                                                     │
│   1573 │   │   │   for i in dep["cancels"]:                                  │
│   1574 │   │   │   │   if not self.queue_enabled_for_fn(i):                  │
│ ❱ 1575 │   │   │   │   │   raise ValueError(                                 │
│   1576 │   │   │   │   │   │   "Queue needs to be enabled! "                 │
│   1577 │   │   │   │   │   │   "You may get this error by either 1) passing  │
│   1578 │   │   │   │   │   │   "into an interface without enabling the queue │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: Queue needs to be enabled! You may get this error by either 1) 
passing a function that uses the yield keyword into an interface without 
enabling the queue or 2) defining an event that cancels another event without 
enabling the queue. Both can be solved by calling .queue() before .launch()

[jasonmcaffee]

I confirmed this has been fixed. Next time I'll submit the PR myself. Thanks @jllllll 👍

[BuffMcBigHuge]

@Erthar I solved the same issue by reverting back to May 17th build.

git checkout f052ab9c8fed3dedc446c3847f10ab22e42bfb3A

I think something committed recently is causing the error, didn't dig into it however.

[Erthar]

I solved the same issue by reverting back to May 17th build.
git checkout f052ab9c8fed3dedc446c3847f10ab22e42bfb3A
I think something committed recently is causing the error, didn't dig into it however.

This worked for me. Thanks @BuffMcBigHuge!

[asad-ullah321]

This is now fixed: 2aa01e2

I got this type of error and am not able to solve it can you help me with this, here is the link to my issue #4264

[Rubiel1]

The 4bit bitsandbytes update is currently in closed beta. It will be some time before that version is publicly released.

Hello, is the 4bit bitsandbytes still in closed beta?