fix causallm weights compression via quantizaer

[eaidova]

What does this PR do?

this PR prevent issue with NotImplementedError: NNCF is not yet supported OpenVINO data type: bf16 on some platforms.
bf16 injected using preprocessing for modifying past key values precision based on inference precision hint applied before model compilation

[eaidova]

@echarlaix @AlexKoff88 @helena-intel could you please take a look

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[canyurt7]

Hi

using latest OpenVino, I run into same issue with openvinotoolkit/openvino_notebooks#1514

Llama3 8B inference on CPU with OpenVino Optimum.

I see merge is there and I should not have run into the issue if it was fixed?

[AlexKoff88]

Hi @canyurt7, the fix is in progress in NNCF and OpenVINO but as a workaround you can cast model to float or float16 before optimizing it with OpenVINO/NNCF.

[helena-intel]

Hi @canyurt7, when using optimum-intel with weight compression, you should not run into this issue. I have tested Llama-3-8B on 5th generation Xeon (with AMX/BF16). It's worth trying to update all your dependencies: pip install --upgrade --upgrade-strategy eager "optimum-intel[openvino]"@git+https://github.com/huggingface/optimum-intel.git and OpenVINO to the latest nightly version: pip install --upgrade --pre openvino --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly.

If that does not help, can you give some more information about what goes wrong and if possible post the output of the command/code you're running?