[community pipeline] SpeechToImage

[stevhliu]

Hi @MikailINTech, the copy-and-paste example from the Speech to image pipeline is failing to initialize correctly with the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-2-3d6478569897>](https://localhost:8080/#) in <cell line: 34>()
     32 
     33 diffuser_pipeline.enable_attention_slicing()
---> 34 diffuser_pipeline = diffuser_pipeline.to(device)
     35 
     36 output = diffuser_pipeline(speech_data)

1 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py](https://localhost:8080/#) in components(self)
   1830 
   1831         if set(components.keys()) != expected_modules:
-> 1832             raise ValueError(
   1833                 f"{self} has been incorrectly initialized or {self.__class__} is incorrectly implemented. Expected"
   1834                 f" {expected_modules} to be defined, but {components.keys()} are defined."

ValueError: SpeechToImagePipeline {
  "_class_name": "SpeechToImagePipeline",
  "_diffusers_version": "0.22.0.dev0",
  "_name_or_path": "CompVis/stable-diffusion-v1-4",
  "feature_extractor": [
    "transformers",
    "CLIPImageProcessor"
  ],
  "scheduler": [
    "diffusers",
    "PNDMScheduler"
  ],
  "speech_model": [
    "transformers",
    "WhisperForConditionalGeneration"
  ],
  "speech_processor": [
    "transformers",
    "WhisperProcessor"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}
 has been incorrectly initialized or <class 'diffusers_modules.git.speech_to_image_diffusion.SpeechToImagePipeline'> is incorrectly implemented. Expected {'feature_extractor', 'unet', 'vae', 'safety_checker', 'text_encoder', 'speech_model', 'scheduler', 'tokenizer', 'speech_processor'} to be defined, but dict_keys(['speech_model', 'speech_processor', 'vae', 'text_encoder', 'tokenizer', 'unet', 'scheduler', 'feature_extractor']) are defined.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[cbstyle]

How to solve this problem finally?

[yukarinoki]

I am encountering the same issue with the following code:

from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_config("runwayml/stable-diffusion-v1-5", use_safetensors=True)
pipeline.to("cuda")

This is just information, not a workaround, but with diffusers==0.13.1, the code runs without any errors.

However, with version 0.14.0, the same error occurs.

workaround ?

In my case, the issue seemed to be resolved when I used from_pretrained instead of the deprecated from_config.

pipeline = diffusers.DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True)
pipeline.to("cuda")

[MqLeet]

I encounter the same error when trying to load my custom pipeline to GPU

ensemble_pipe.to(f"cuda:{gpu_id}")

ValueError: AdaptiveDDIMPipelineMore {
  "_class_name": "AdaptiveDDIMPipelineMore",
  "_diffusers_version": "0.17.0.dev0",
  "scheduler": [
    "diffusers",
    "DDIMScheduler"
  ],
  "unets": [
    "modules",
    "ModuleList"
  ]
}
 has been incorrectly initialized or <class 'diffusers.pipelines.ddim_more.pipeline_ddim_more.AdaptiveDDIMPipelineMore'> is incorrectly implemented. Expected {'scheduler', 'unet_list'} to be defined, but dict_keys(['unets', 'scheduler']) are defined.

Any solutions to this error?