Add dynamic quantization config

[echarlaix]

Add dynamic quantization configuration

from optimum.intel import OVDynamicQuantizationConfig, OVModelForCausalLM

quantization_config = OVDynamicQuantizationConfig(bits=8, activations_group_size=32)
int8_model = OVModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[echarlaix]

Removed calibration_dataset argument @nikita-savelyevv

[AlexKoff88]

@echarlaix, it looks like we will have two ways to enable weight-only quantization + dynamic quantization:

1. as you did it in the PR.
2. Quantize weights and pass OVConfig with runtime option that enables DQ.

Am I right here?

Two more things to note. This flow is still under development for GPU. It is also used along with 8-bit KV-cache quantization which performs pretty well and helps to reduce the memory footprint with almost no accuracy degradation.

[nikita-savelyevv]

Some minor comments

[nikita-savelyevv]

With calibration_dataset removal from from_pretrained() arguments, the workflow such as here will no longer be available. I personally am ok with this, but as I understand this will limit capabilities of hybrid quantization in the future. Currently, string only dataset is enough for it, but from discussion we had with @l-bat this is not the only use case.

Possibly, we should rework the hybrid quantization workflow to be called only through OVQuantizer which accepts custom calibration dataset.

@AlexKoff88, what do you think?

[echarlaix]

@echarlaix, it looks like we will have two ways to enable weight-only quantization + dynamic quantization:

1. as you did it in the PR.

2. Quantize weights and pass OVConfig with runtime option that enables DQ.

Am I right here?

Two more things to note. This flow is still under development for GPU. It is also used along with 8-bit KV-cache quantization which performs pretty well and helps to reduce the memory footprint with almost no accuracy degradation.

Yes but would be in favor of using 1. in our examples / documentation to have the same API across all quantization strategies

[AlexKoff88]

@echarlaix, shall we turn 8-bit KV-cache quantization as well? It is essentially a per-token INT8 quantization and it is safe in terms of accuracy degradation?