refactor to support SDXL classes

[noskill]

No description provided.

[Necr0x0Der]

Partly works, but doesn't work with .safetensors files (besides prompt2im)

[Necr0x0Der]

The issue is that when we pass model_type=ControlnetType.stable_diffusion_xl to Cond2ImPipe and use .safetensors at the same time, this piece of code is used

if model_id.endswith('.safetensors'):
        try:
            self.pipe = StableDiffusionPipeline.from_single_file(model_id, **args)
        except TypeError as e:
            pass
        self.pipe = StableDiffusionXLPipeline.from_single_file(model_id, **args)

and StableDiffusionXLPipeline is used. While XL class is used, it is used for a different reason, and a wrong class is used (not control net). Apparently, something wrong with this logic. Maybe, we don't need specifically model_type=ControlnetType.stable_diffusion_xl, which is not taken into account in the base class, but could try introducing generic model_type parameter in the base class constructor.

[noskill]

the code should work after pr in diffusers is merged huggingface/diffusers#5179

[Necr0x0Der]

Still doesn't work for me, but I'm ok with merging it and raising an issue.

[noskill]

@Necr0x0Der the issue in duffusers was with loading StableDiffusionPipeline from safetensors, not StableDiffusionXL

from diffusers import ControlNetModel, StableDiffusionControlNetPipeline


model_id = "realismEngine/realismEngine_v10.safetensors"
cnet_id = "DionTimmer/controlnet_qrcode-control_v1p_sd15"

cnets = [ControlNetModel.from_pretrained(cnet_id)]
StableDiffusionControlNetPipeline.from_single_file(model_id, controlnet=cnets)

My pr was merged in main branch but not yet in current release from 29.09.2023 available on pypi