Fix TypeError when using prompt_embeds and negative_prompt

[At-sushi]

Features

• Fixed issue that StableDiffusionPipeline.__call__() causes TypeError when both prompt_embeds and negative_prompt (not negative_prompt_embeds) are given as arguments.

This is my first PR so if there is problems please inform me.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Hey @At-sushi,

Could you maybe add a quick code snippet that shows the current problem with prompt_embeds and negative_prompt? :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Great PR, thanks!

[At-sushi]

I apologize for being late for replies.

this PR is to fix the error with negative_prompts

Examples code:

from diffusers import StableDiffusionPipeline, AutoencoderKL

model_id = "model"
device = "cuda"

negative_prompt = "some negative prompts"

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16, revision='fp16')
pipe = pipe.to(device)
tokenizer = pipe.tokenizer
text_encoder = pipe.text_encoder

num_samples = 2

def infer(prompt):
    text_input = tokenizer(
        [prompt] * num_samples,
        padding="max_length",
        max_length=tokenizer.model_max_length,
        truncation=True,
        return_tensors="pt",
    )
    prompt_embeds = text_encoder(
        text_input.input_ids.to(device),
        attention_mask=text_input.attention_mask.to(device),
    )[0]

    # Somehow prompt type check is still active
    images = pipe(prompt_embeds=prompt_embeds, negative_prompt=[negative_prompt] * num_samples)["images"]

    return images

then it occurs such as:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 915, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.9/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "<ipython-input-2-61e1bb04fb24>", line 49, in infer
    images = pipe(prompt_embeds=prompt_embeds, negative_prompt=[negative_prompt] * num_samples)["images"]
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 628, in __call__
    prompt_embeds = self._encode_prompt(
  File "/usr/local/lib/python3.9/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 363, in _encode_prompt
    raise TypeError(
TypeError: `negative_prompt` should be the same type to `prompt`, but got <class 'list'> != <class 'NoneType'>.