When building from source getting errors quantization.

[choochtech]

Hi,

First of all thanks for this fantastic work. Great job !

After building intel-extension-for-transformers from source we noticed that we get errors when quantizing the model.

After the initial Llama 2 7B model is converted to ne_llama_f32.bin (Float 32) the library fails to quantize the model getting the below error.

quant_model: failed to quantize model from 'runtime_outs/ne_llama_f32.bin'
Traceback (most recent call last):
File "/home/ubuntu/run.py", line 9, in
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=woq_config, trust_remote_code=True)
File "/opt/conda/envs/llm_hakan/lib/python3.9/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 265, in from_pretrained
model.init(
File "/opt/conda/envs/llm_hakan/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/init.py", line 124, in init
assert os.path.exists(quant_bin), "Fail to quantize model"
AssertionError: Fail to quantize model

What are we missing ? Do we need to compile another package as well ?

Thanks

[LJ-underdog]

@choochtech , thank you for using Itrex, is the model you are using from llama2-hf-7b on Huggingface? If not, please provide the URL of the corresponding model.

[choochtech]

Your main branch doesn't work properly.

We had to download 1.3 (https://github.com/intel/intel-extension-for-transformers/releases/tag/v1.3) and build. That worked with no issues,

Thanks

[Zhenzhong1]

Hi, so happy it worked with no issues. I closed this issue~ Thank you.