IP adapter basic example causes ValueError: too many values to unpack (expected 3)

[AlmogDeepKeep]

Describe the bug

I tried to run the basic example from this tutorial . Building the model works, but it crashes on inference.

Reproduction

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
pipeline.set_ip_adapter_scale(0.6)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_diner.png")
generator = torch.Generator(device="cpu").manual_seed(0)
images = pipeline(
    prompt="a polar bear sitting in a chair drinking a milkshake", 
    ip_adapter_image=image,
    negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
    num_inference_steps=100, 
    generator=generator,
).images

Code is from this tutorial

Logs

images = pipeline(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 1241, in __call__
    noise_pred = self.unet(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/unets/unet_2d_condition.py", line 1121, in forward
    sample, res_samples = downsample_block(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 1199, in forward
    hidden_states = attn(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/transformers/transformer_2d.py", line 391, in forward
    hidden_states = block(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention.py", line 366, in forward
    attn_output = self.attn2(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 512, in forward
    return self.processor(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 2195, in __call__
    ip_key = attn.head_to_batch_dim(ip_key)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 551, in head_to_batch_dim
    batch_size, seq_len, dim = tensor.shape
ValueError: too many values to unpack (expected 3)

System Info

• diffusers version: 0.26.3
• Platform: Linux-6.2.0-1018-aws-x86_64-with-glibc2.35
• Python version: 3.9.17
• PyTorch version (GPU?): 1.13.1+cu117 (True)
• Huggingface_hub version: 0.20.3
• Transformers version: 4.33.0.dev0
• Accelerate version: 0.21.0
• xFormers version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@DN6 @yiyixuxu

[AlmogDeepKeep]

I installed Xformers. For some reason, that fixed it 🤷‍♂️

[fabiorigano]

installing xFormers, pip uninstalled torch-1.13 and installed torch-2.2

the IP-Adapter attention processor for pytorch 2 is different, so you no longer have this problem

IMO, this issue should be fixed with a PR

[yiyixuxu]

@fabiorigano
ohh our ip-adapter does not work with IPAdapterAttnProcessor??

[fabiorigano]

hi @yiyixuxu

The output tensors of to_k_ip and to_v_ip are 4D, but they are expected to be 3D (batch size, sequence length and embedding dimension) for head_to_batch_dim

the second dimension has always shape 1, so the tensors can be easily reshaped. this issue occurs only in IPAdapterAttnProcessor and not in IPAdapterAttnProcessor2_0

here it is the code to fix the isse

# for ip-adapter
        for current_ip_hidden_states, scale, to_k_ip, to_v_ip, mask in zip(
            ip_hidden_states, self.scale, self.to_k_ip, self.to_v_ip, ip_adapter_masks
        ):
            ip_key = to_k_ip(current_ip_hidden_states)
            ip_value = to_v_ip(current_ip_hidden_states)

            ip_key = ip_key.reshape(ip_key.shape[0], ip_key.shape[-2], ip_key.shape[-1])  # Add this
            ip_value = ip_value.reshape(ip_value.shape[0], ip_value.shape[-2], ip_value.shape[-1]) # and this

            ip_key = attn.head_to_batch_dim(ip_key)
            ip_value = attn.head_to_batch_dim(ip_value)

[yiyixuxu]

@fabiorigano
looks good! wanna open a PR?

[fabiorigano]

sure!

[fabiorigano]

the second dimension has always shape 1, so the tensors can be easily reshaped. this issue occurs only in IPAdapterAttnProcessor and not in IPAdapterAttnProcessor2_0

@yiyixuxu my mistake here, I was not thinking about multiple images
the PR now fixes the issue