Support GPTQ file training LoRA

[ssbg2]

Description

First of all, thank you very much for this tool to help me get started quickly.
Can you support GPTQ models in the training LORA files in subsequent versions?

Additional Context
https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GPTQ
This is the model I use, which can load and chat(with --wbits 4 --groupsize 128), but cannot train to generate LORA files, when I start LoRA Training,command says: LoRA training has only currently been validated for LLaMA,OPT,GPT-J,and GPT-NeoX models.(Found model type:LlamaGPTQForCausalLM)

[ssbg2]

Thank you very much!
So I can just use scripts instead of WEBUI operations?
Can you explain in detail how to load model with "monkeypatch"?I searched the repo documentation and there is no mention of how to do this...

[GamingDaveUk]

You just use --monkeypatch as a parameter. Everything just got changed so I'll see how well all that works when I integrate it for myself.

when you try training you get :

from monkeypatch.peft_tuners_lora_monkey_patch import \
ModuleNotFoundError: No module named 'monkeypatch'

Been trying to work out how to train a lora or qlora on a gptq 4bit model for well over a couple of weeks now, everything i read says its do-able, everything i try fails.

We really need a step by step guide to training Lora/Qlora in textgen webui for these very popular and used models

[GamingDaveUk]

You have to download and install the repo and compile it's kernel from: https://github.com/sterlind/GPTQ-for-LLaMa/tree/lora_4bit

qlora should just work with new bits and bytes and latest transformers.

Do you have a link to a guide that waorks for training gptq 4bit models inside textgen webui?

I'll take a look at the link when i get home, though no idea how to compile something with out an IDE.

I wish making Lora's for LLM's was as well documented as it is form aking Lora's for image generation models lol

[NicolasMejiaPetit]

You have to download and install the repo and compile it's kernel from: https://github.com/sterlind/GPTQ-for-LLaMa/tree/lora_4bit
qlora should just work with new bits and bytes and latest transformers.

Do you have a link to a guide that waorks for training gptq 4bit models inside textgen webui?

I'll take a look at the link when i get home, though no idea how to compile something with out an IDE.

I wish making Lora's for LLM's was as well documented as it is form aking Lora's for image generation models lol

I'm currently training Wizard 7b uncensored on my computer, with out monkey patch and just using autogptq to load_in_4bit. For some reason this works with no monkey patch, I've followed the monkey patch guide but it just dosen't work for me. Also in the Docs folder of this repo there is a readme for training with 4bit lora's and it has the instructions for using monkey patch.

[GamingDaveUk]

EDIT:
I renamed GPTQ-for-LLaMa folder inside repositories
I loaded cmd_linux.sh so i would be inside textgen webui environement
I then started to follow the instructions from the repo you linked to cloning it into the repositories folder so textgen webui would use that. (obviously not creating a new environment for it as i need textgen webui to use it)
when i did pip install -r requirements.txt it hit a snag, it needed safetensors==0.3.1 but safetensors==0.3.0 was on the text file, i edited the file and re-ran that installed
I could not do the setup.py part as that file doesnt exist, oddly i see it on the github but not in the cloned repo the instructions tell you to do.
Textgen webui would then not load:
ImportError: accelerate>=0.20.3 is required for a normal functioning of this module, but found accelerate==0.17.1.
so i updated acceletate to 0.20.3

Text gen loaded.
However trying to train a lora on TheBloke_chronos-wizardlm-uc-scot-st-13B-GPTQ resulted in this error:

Traceback (most recent call last): File “/home/mrgreaper/oobabooga_linux/text-generation-webui/modules/training.py”, line 374, in do_train lora_model = get_peft_model(shared.model, config) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/mapping.py”, line 120, in get_peft_model return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/peft_model.py”, line 691, in init super().init(model, peft_config, adapter_name) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/peft_model.py”, line 100, in init self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type]( File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/tuners/lora.py”, line 174, in init self.add_adapter(adapter_name, self.peft_config[adapter_name]) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/tuners/lora.py”, line 181, in add_adapter self._find_and_replace(adapter_name) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/tuners/lora.py”, line 315, in _find_and_replace new_module = self._create_new_module(lora_config, adapter_name, target) File “/home/mrgreaper/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/peft/tuners/lora.py”, line 285, in _create_new_module raise ValueError( ValueError: Target module QuantLinear() is not supported. Currently, only torch.nn.Linear and Conv1D are supported.

[GamingDaveUk]

You have to download and install the repo and compile it's kernel from: https://github.com/sterlind/GPTQ-for-LLaMa/tree/lora_4bit
qlora should just work with new bits and bytes and latest transformers.

Do you have a link to a guide that waorks for training gptq 4bit models inside textgen webui?
I'll take a look at the link when i get home, though no idea how to compile something with out an IDE.
I wish making Lora's for LLM's was as well documented as it is form aking Lora's for image generation models lol

I'm currently training Wizard 7b uncensored on my computer, with out monkey patch and just using autogptq to load_in_4bit. For some reason this works with no monkey patch, I've followed the monkey patch guide but it just dosen't work for me. Also in the Docs folder of this repo there is a readme for training with 4bit lora's and it has the instructions for using monkey patch.

Interesting, going to do a fresh download and install and give it a try with out attempting to do anything else. Good to see someone successful able to train a lora for a gptq 4bit model, i was really starting to think it was impossible

Nope same error as above

[GamingDaveUk]

You have to download the alpaca_4bit repo, it's version of GPTQ (compile it) and use a newer PEFT.

do you have a link to a guide on how to do that?

[ghost]

@Ph0rk0z how could I train in 4-bit using monkey patches? I've tried everything I can find coming up with different errors.

[github-actions]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

[ziat007]

Any news on GPTQ lora training that's more up to date?