How long does it take to quantize?

[fahadh4ilyas]

I'm been using quantization tools like GPTQ, Exllama, or QUIP#. Those tools is quite fast to do quantization in a single A6000 gpu. But, this tool takes a really long time even though I'm using two A6000 gpu. How long does it take for quantizing Mistral 7B using two A6000 gpu and this parameters:

python main.py ../models/my-mistral-7B wikitext2 --nsamples=1024 --num_codebooks=1 --nbits_per_codebook=16 --in_group_size=8 --relative_mse_tolerance=0.01 --finetune_relative_mse_tolerance=0.001 --finetune_batch_size=32 --local_batch_size=1 --save ../models/my-mistral-7B-AQLM --model_seqlen 8192 --offload_activations

[iamwavecut]

See #28

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.