Generalize GPTQ_loader, support any model

[mayaeary]

Improved version of #521

Generalized version of quantized loader, now it auto-detecting model from model file. It allows loading GPT-J and Pygmalion-6b without joggling of repositories.

I'll try to make generalized offload version, but for now only llama supports it.

You can quantize models using my fork - https://github.com/mayaeary/GPTQ-for-LLaMa/tree/gptj-v2.

Pre-quantized --wbits 4 --groupsize 128

• Pygmalion (original) - https://huggingface.co/mayaeary/pygmalion-6b-4bit-128g
• Pygmalion (dev) - https://huggingface.co/mayaeary/pygmalion-6b_dev-4bit-128g
• PPO_Pygway-V8p4_Dev-6b - https://huggingface.co/mayaeary/PPO_Pygway-V8p4_Dev-6b-4bit-128g

# Download
python download-model.py https://huggingface.co/mayaeary/pygmalion-6b-4bit-128g

# Launch
python server.py --model pygmalion-6b-4bit-128g --wbits 4 --groupsize 128 --cai-chat

[Ph0rk0z]

It needs a PR to GPTQ.. technically GPT-NEOX and NEO can also be done this way. In here they don't want to swap around the GPTQ repo.

The steps are get this merged upstream and then get support for the loras into upstream and PEFT as well.. Otherwise these never get merged.

[oobabooga]

This is very impressive.

[oobabooga]

I have done some basic sanity tests to check if everything is equivalent to the current code, and the answer is that yes.

@Ph0rk0z I agree that it would be nice to have this functionality merged upstream in GPTQ-for-LLaMa, but I see no reason to use Maya's code until that happens. The only caveat is that we will have to watch for eventual changes in the upstream make_quant and load_quant functions.

For reference, these are the VRAM usages that I have seen for pygmalion:

• Soon after loading: 4.5GB
• Full context length: 7.8GB

Test results

Prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Who is best waifu, Rei or Asuka?
### Response:

Alpaca-30B-Int4

python server.py --wbits 4 --model Alpaca-30B-Int4 --listen

Maya:

Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Who is best waifu, Rei or Asuka?
### Response:
 It's a difficult choice, but I think Rei is the best waifu. She is kind, caring, and loyal, and she always puts others before herself. She is also a powerful warrior and a great pilot, making her a great choice for a waifu.

Output generated in 4.48 seconds (13.39 tokens/s, 60 tokens, context 47)

Main:

Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Who is best waifu, Rei or Asuka?
### Response:
 It's a difficult choice, but I think Rei is the best waifu. She is kind, caring, and loyal, and she always puts others before herself. She is also a powerful warrior and a great pilot, making her a great choice for a waifu.

Output generated in 4.50 seconds (13.33 tokens/s, 60 tokens, context 47)

alpaca-native-4bit

python server.py --model alpaca-native-4bit --wbits 4 --groupsize 128 --listen

Maya:

Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Who is best waifu, Rei or Asuka?
### Response:
 Personally, I think Rei is the best waifu. She is wise, kind, and always looks out for her friends. She is also strong and courageous, never backing down from a challenge. Asuka is also very powerful, but she can be impulsive and reckless at times. Rei, on the other hand, is more thoughtful and calculated in her decisions.

Output generated in 2.36 seconds (34.81 tokens/s, 82 tokens, context 47)

Main:

Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Who is best waifu, Rei or Asuka?
### Response:
 Personally, I think Rei is the best waifu. She is wise, kind, and always looks out for her friends. She is also strong and courageous, never backing down from a challenge. Asuka is also very powerful, but she can be impulsive and reckless at times. Rei, on the other hand, is more thoughtful and calculated in her decisions.

Output generated in 2.33 seconds (35.18 tokens/s, 82 tokens, context 47)

[treshphilip]

Can someone convert that model https://huggingface.co/TehVenom/PPO_Pygway-V8p4_Dev-6b to 4bit please? I have not enough of GPU memory to do that(

[mayaeary]

Can someone convert that model https://huggingface.co/TehVenom/PPO_Pygway-V8p4_Dev-6b to 4bit please? I have not enough of GPU memory to do that(

Done, added to first post

[treshphilip]

Can someone convert that model https://huggingface.co/TehVenom/PPO_Pygway-V8p4_Dev-6b to 4bit please? I have not enough of GPU memory to do that(

Done, added to first post

Thank you very much! I will go test that)