RuntimeError: Could not infer the backend framework from the model type when used with stable diffusion.

[ranjithum]

i m not sure if this is the right group for discussion, but asking it anyways ..
i m trying to run compress_weights on pretrained stable diffusion pipeline.. and this is the snippet

from optimum.intel.openvino import OVStableDiffusionXLPipeline
from nncf import compress_weights, CompressWeightsMode
model_id = "stabilityai/stable-diffusion-xl-base-1.0"


ov_pipe_bf16 = OVStableDiffusionXLPipeline.from_pretrained(model_id, compile=False, export=True, load_in_8bit=True)

ov_pipe_bf16 = compress_weights(ov_pipe_bf16, mode=CompressWeightsMode.INT4_SYM, group_size=128, ratio=0.8)
ov_pipe_bf16.save_pretrained("./openvino_ir_xl_base-1.0")

But i get follwoing exception

RuntimeError: Could not infer the backend framework from the model type because the framework is not available or the model type is unsupported. The available frameworks found: Torch, ONNX, OpenVINO.

Can some please help me with this problem...

Note:- i m trying to run this locally on my macbook, which has intel cpu. no GPU.

[MaximProshin]

@andreyanufr , @ljaljushkin , @AlexKoff88 please take a look. I guess the model passed to compress_weights() is in int8 and this might be the reason. Perhaps we need a better error trace for it.

[ljaljushkin]

The error happens because compress_weights takes not expected object OVStableDiffusionXLPipeline. It is supposed to be the openvino.Model. OVStableDiffusionXLPipeline contains many Optimum wrappers on top of openvino.Model object:

...
"text_encoder": [
    "optimum",
    "OVModelTextEncoder"
  ],
"text_encoder_2": [
   "optimum",
   "OVModelTextEncoder"
],
....

They can be compressed separately, as follows:

from nncf import compress_weights, CompressWeightsMode
ov_text_encoder_model = ov_pipe_bf16.text_encoder.model
compressed_text_encoder = compress_weights(ov_text_encoder_model, mode=CompressWeightsMode.INT4_SYM, group_size=128, ratio=0.8)

As @MaximProshin noticed, load_in_8bit=True option for OVStableDiffusionXLPipeline applies 8-bit weight compression on loading.
Currently, compression of 8-bit model to 4-bit one is not supported. Weight compression API expects floating-point model.

BTW, 4bit options were added for OVQuantizer in the Optimum: https://github.com/huggingface/optimum-intel/pull/469/files#diff-4672e27e75b94153148b701383fd97a0aa48a8106c0fe52fb94075879fce7a0cR258
There might be some plans for extending it for OVStableDiffusionXLPipeline. @AlexKoff88

from optimum.intel import OVQuantizer
from optimum.intel import OVConfig
quantizer = OVQuantizer.from_pretrained(ov_optimum_wrapper)
quantizer.quantize(save_directory='', weights_only=True, quantization_config=OVConfig(compression={"type": "int4_sym_g128", "ratio": 0.8}),)

[l-bat]

@ranjithum you can load pretrained stable diffusion pipeline from optimum-intel with compressed weights by specifying the quantization_config

Please install the latest version

pip install optimum-intel==1.15.2

Eaxmple

from optimum.intel import OVStableDiffusionXLPipeline, OVWeightQuantizationConfig

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
quantization_config = OVWeightQuantizationConfig(bits=4, sym=True, ratio=0.8, group_size=64)
ov_pipe_bf16 = OVStableDiffusionXLPipeline.from_pretrained(model_id, compile=False, export=True, quantization_config=quantization_config)