Skip to content

8. Quantization

Jump to bottom
AlpinDale edited this page Mar 11, 2024 · 8 revisions

Aphrodite supports a variety of quantization methods (most of them, in fact). All the supported model classes can be run quantized, provided the originating quant library supports them. The ones we currently support are:

Let's go through how you can run each of them.

GPTQ

The GPTQ quantization in Aphrodite uses the ExllamaV2 kernels for boosting throughput. The bit sizes supported are: 2, 3, 4, and 8. Most models usually have a GPTQ converted version on Hugging Face, but you can manually convert using the transformers library:

from transformers import AutoModelForCausalLM, GPTQConfig

gptq_config = GPTQConfig(
    bits=4,
    dataset="wikitext2",
    group_size=128,
    desc_act=True,
    use_cuda_fp16=True
)
model_id = "/path/to/model"  # can also be a HF model

model = AutoModeForCausalLM.from_pretrained(model_id, quantization_config=gptq_config, attn_implementation="sdpa")
model.config.quantization_config.dataset = None
model.save_pretrained(f"{model_id}-GPTQ")

Replace the model_id with either the local path to your model on disk, or a HuggingFace model ID.

To load your GPTQ model into Aphrodite, simply pass the path or ID to the --model flag, it'll figure everything out.

AWQ

Similar to GPTQ, you can either find conversions on HF, or conver it yourself. The conversion process is very similar:

from transformers import AutoModelForCausalLM, AwqConfig

gptq_config = AwqConfig(
    bits=4,
    dataset="wikitext2",
    group_size=128,
    desc_act=True,
    use_cuda_fp16=True
)
model_id = "/path/to/model"  # can also be a HF model

model = AutoModeForCausalLM.from_pretrained(model_id, quantization_config=awq_config, attn_implementation="sdpa")
model.config.quantization_config.dataset = None
model.save_pretrained(f"{model_id}-AWQ")

Loading process is the same.

Exllama V2

There aren't as many Exllama V2 models on HF, so you may need to convert some of them. Please refer to this page for instructions: https://github.com/turboderp/exllamav2/blob/master/doc/convert.md

Similar to all quant methods, just point --model to the directory where your exl2 model is.

GGUF

Most models should have a GGUF variant uploaded to HF. Unlike other models, GGUF is contained within a single file, so you cannot pass a HuggingFace ID to the --model flag. Download the GGUF file, and point the --model to it.

One important thing to note is that Aphrodite needs a PyTorch model, and GGUF is a binary format (kind of). We first need to convert the GGUF into a PyTorch-compatible state_dict. This is done automatically and the user doesn't need to do anything, but it will use a fair bit of CPU RAM (2x the model size in GBs). If you can't budget for that, then please convert your GGUF model using this script. Run it with --help to see all the options.

To convert an FP16 model to GGUF in the first place, see here.

Marlin

Marlin needs a GPTQ model that satisfies a few conditions:

  • bits=4
  • group_size= -1 or 128
  • desc_act=False

You can follow the GPTQ conversion guide above but with these specific parameters. Then see here for how to convert to Marlin. Loading process same as others.

SmoothQuant+

No conversion is needed here. Simply load an FP16 model with either of these two flags:

  • --load-in-4bit

This is the fastest Quant method currently available, beats both GPTQ and Exllamav2. The start time is a bit slow as it needs to convert the model to 4bit. The quality, however, is very good. Reportedly as good or better than AWQ. You can also load AWQ models with this flag for faster speeds!

  • --load-in-smooth

This flag loads your FP16 model in 8bit using SmoothQuant+. It's slower than FP16, but is almost lossless at half the memory usage.

Bitsandbytes

Similar to SmoothQuant+, Bitsandbytes quantization is implicit and done automatically. Only 8bit is currently supported. Simply load your FP16 model with --load-in-8bit. This option is very slow, so please use --load-in-smooth if you need implicit 8-bit conversion.

QuIP#

Please use this library to quantize your model to QuIP#. These are very rare, so you likely won't find an HF upload.

SqueezeLLM

This 4-bit quantization method is also rare on HF. The speeds aren't good, so they're not very popular. Please refer to this repo for conversion.

AQLM

AQLM, similar to what QuIP# offers, is a state-of-the-art 2-bit quantization method. It's very slow, however, so I don't recommend using this unless for experimentation purposes. It takes a long time to quantize a model (12 days on 8xA100s for a 70B model), and the inference is too slow to be practical (over 100 seconds to generate from a 70B on a 3090 GPU). See here for a list of quantized AQLM models, and here for how to quantize a model yourself.

Clone this wiki locally