33B llama quantization post-inference time

[ACoder-J]

Why is it that after I quantize, my inference is 2 times slower than the original `model?

Quantize command：python main.py $MODEL_PATH $DATASET_PATH --nsamples=1024 --num_codebooks=1 --nbits_per_codebook=16 --in_group_size=8 --relative_mse_tolerance=0.01 --finetune_relative_mse_tolerance=0.001 --finetune_batch_size=32 --local_batch_size=1 --offload_activations --wandb --save $SAVE_PATH --dtype float32

Quantitative results：

In addition, looking at the GPU usage, it seems that it has become a parallel pipeline. I use four cores to load the model, but only one GPU utilization is 100% at the same time.

Have you encountered the same problem as me again? Is the inference speed of your quantized model normal?

[BlackSamorez]

Hi @ACoder-J !
The inference speed is really bottlenecked by transformers for multiple reasons (transformers being slow and kernel launches taking long).
One way to achieve proper speedup is to compile your model into a cuda graph. It's a new feature added to PyTorch in 2.0.0 and refined in 2.2.1. People of transformers have also been working hard to properly support and document it. To check out what you can do with it already, please refer to this AQLM demo.
Another way to properly serve AQLM is to use the vllm integration. It's not complete yet but it might be usable already (not sure).

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.