from auto_round import AutoRoundConfig

[CrispStrobe]

... did not work for me right now, whereas previously it did. Cannot check this further at the moment, but maybe you might want to. Environment was kaggle and colab, occured after !pip install auto-round, seemingly resolved per:

!pip uninstall auto_round -y
!pip cache remove auto_round
!pip install --no-cache-dir auto_round
from auto_round.auto_quantizer import AutoRoundConfig

[wenhuach21]

Sorry for the confusion. The AutoRoundConfig was introduced after version 0.3.0. We'll update the documentation to clarify this. In the meantime, you can install the latest version from the source

 git clone https://github.com/intel/auto-round.git && cd auto-round && pip install -vvv --no-build-isolation -e ..

For version 0.3.0, we recommend using:

from auto_round.auto_quantizer import AutoHfQuantizer

In this version, the device is automatically chosen with GPU and HPU taking priority over CPU. To use the CPU on a CUDA machine, you'll need to modify the model's configuration file.

If you're working with CUDA, you'll need to install from the source to compile the kernel, as we couldn't include the it in the package for various reason

[wenhuach21]

updated readme, also the introduction of format you might be interested.
Additionally, we plan to release a new version next month, which will address this issue.

[CrispStrobe]

many thanks, perfect - and wow, that was swift!

[sahibpreetsingh12]

Is the issue still there because I am still getting the issue.

[wenhuach21]

Apologies for the delay in the release. In the meantime, please use the following import statement for version v0.3

from auto_round.auto_quantizer import AutoHfQuantizer

[sahibpreetsingh12]

yes thanks @wenhuach21 the issue i found is
While quantisation it does it with 'auto-roun' format but during inference since we don't have support for that it causes the issue.
Do correct me if i am wrong ?

[wenhuach21]

In quantization, the model operates in floating-point format and undergoes fake quantization to simulate the quantization behavior. After the tuning process is complete, this fake model is converted into a true int4 model that adheres to your specified format. So for real inference, we need to import that code for auto-round format or install auto_gptq for gptq format.

Please refer to the Model Inference in README for more details.

[sahibpreetsingh12]

Sure
I installed optimum and auto-gptq
but still getting
ImportError: Loading a GPTQ quantized model requires optimum (pip install optimum) and auto-gptq library (pip install auto-gptq)
I am on a Kaggle kernel so can't refresh

[wenhuach21]

Sure I installed optimum and auto-gptq but still getting ImportError: Loading a GPTQ quantized model requires optimum (pip install optimum) and auto-gptq library (pip install auto-gptq) I am on a Kaggle kernel so can't refresh

Which format are you using, auto_round or auto_gptq?

For the GPTQ format, installing Optimum and Auto-GPTQ should suffice. If Transformers still throws an exception, please check your environment; you might have multiple environments in use.

For the auto_round format, please follow our readme

[sahibpreetsingh12]

The format is 'auto_gptq'
i installed optimum and auto-gptq and auto-round==0.3.0
and for now i am quantising 'facebook/opt-125m' and my kaggle kernel is crashing
it's 12GB of P-100 GPU
and this is what i am getting

[wenhuach21]

remove the second line and try again, if it's ok, it seems a bug in our code.
It it's not ok, try to inference this model ybelkada/opt-125m-gptq-4bit in your env.

[sahibpreetsingh12]

I did removed the second line and for my model 'facebook/opt-125m' and i got

but when i did inference with 'ybelkada/opt-125m-gptq-4bit' yes i got results

[wenhuach21]

Then you export to AutoRound format model rather than AutoGPTQ format. For AutoRound format, you'll need to install it from source with CUDA support. I recommend switching to the Auto-GPTQ format, but please note that it may have accuracy issues with asymmetric quantization.

autoround.save_quantized(output_dir, format='auto_gptq', inplace=True) 

[sahibpreetsingh12]

Yes I already saved in 'auto-gptq' format

[sahibpreetsingh12]

But still no inference