Colabs do not run

[cinjon]

I've tried running both colabs and I get a sequence of errors:

1. Setting the offset in ptp_utils.py L163 doesn't work with the newest diffusers.
2. After turning off the offset kwarg above, I get TypeError: forward() got an unexpected keyword argument 'attention_mask'.
3. I reverted to diffusers==0.3.0 like in the requirements.txt and I get that TypeError: getattr(): attribute name must be string when trying to load the model (ldm_stable = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", use_auth_token=MY_TOKEN).to(device)).

What am I missing?

[fulltrend]

you need to add missing parameters in your forward() implementation here

prompt-to-prompt/ptp_utils.py

Line 181 in 8186230

def forward(x, context=None, mask=None):

Something like
def forward(x, context=None, mask=None, encoder_hidden_states=None, attention_mask=None):

Refer to the new mplementation here

[JFKraasch]

Doing that results into a new error message:

`
File "/home/ubuntu/miniconda3/envs/null-txt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/miniconda3/envs/null-txt/lib/python3.10/site-packages/diffusers/models/attention.py", line 498, in forward
self.attn2(
File "/home/ubuntu/miniconda3/envs/null-txt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/null-text-inversion/p2p/ptp_utils.py", line 138, in forward
k = self.to_k(context)
File "/home/ubuntu/miniconda3/envs/null-txt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/miniconda3/envs/null-txt/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (8192x320 and 768x320)

Process finished with exit code 1
`

[Johnson-yue]

@JFKraasch I have the same bug :
mat1 and mat2 shapes cannot be multiplied (262144x5 and 320x320)

[JFKraasch]

@Johnson-yue I got it running again
you just have to revert diffusers to 0.8.1 and transformers to 4.24.0

[ouhenio]

hey @cinjon and @Johnson-yue, here is a working version :)

[Weifeng-Chen]

any solution for new version?(>0.11.0)

hey @cinjon and @Johnson-yue, here is a working version :)

[yasserben]

Hi ,
I tried to run the two notebooks : null_text_w_ptp.ipynb and prompt-to-prompt_stable.ipynb and I get the same kind of error :
For prompt-to-prompt_stable.ipynb when I reach this cell :

g_cpu = [torch.Generator().manual_seed(8888)]
prompts = ["A painting of a squirrel eating a burger"]
controller = AttentionStore()
image, x_t = run_and_display(prompts, controller, latent=None, run_baseline=False, generator=g_cpu)
show_cross_attention(controller, res=16, from_where=("up", "down"))

I get the following error :

mat1 and mat2 shapes cannot be multiplied (8192x320 and 768x320)

For null_text_w_ptp.ipynb, when I reach the following cell :

image_path = "./example_images/gnochi_mirror.jpeg"
prompt = "a cat sitting next to a mirror"
(image_gt, image_enc), x_t, uncond_embeddings = null_inversion.invert(image_path, prompt, offsets=(0,0,200,0), verbose=True)
print("Modify or remove offsets according to your image!")

I get the following error :

mat1 and mat2 shapes cannot be multiplied (4096x320 and 768x320)

Here's the full error message for prompt-to-prompt_stable.ipynb :

RuntimeError                              Traceback (most recent call last)
Cell In[7], line 4
      2 prompts = ["A painting of a squirrel eating a burger"]
      3 controller = AttentionStore()
----> 4 image, x_t = run_and_display(prompts, controller, latent=None, run_baseline=False, generator=g_cpu)
      5 show_cross_attention(controller, res=16, from_where=("up", "down"))

Cell In[6], line 6, in run_and_display(prompts, controller, latent, run_baseline, generator)
      4     images, latent = run_and_display(prompts, EmptyControl(), latent=latent, run_baseline=False, generator=generator)
      5     print("with prompt-to-prompt")
----> 6 images, x_t = ptp_utils.text2image_ldm_stable(ldm_stable, prompts, controller, latent=latent, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=GUIDANCE_SCALE, generator=generator, low_resource=LOW_RESOURCE)
      7 ptp_utils.view_images(images)
      8 return images, x_t

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/autograd/grad_mode.py:27, in _DecoratorContextManager.__call__.<locals>.decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File ~/projects/prompt-to-prompt/ptp_utils.py:167, in text2image_ldm_stable(model, prompt, controller, num_inference_steps, guidance_scale, generator, latent, low_resource)
    165 model.scheduler.set_timesteps(num_inference_steps, **extra_set_kwargs)
    166 for t in tqdm(model.scheduler.timesteps):
--> 167     latents = diffusion_step(model, controller, latents, context, t, guidance_scale, low_resource)
    169 image = latent2image(model.vae, latents)
    171 return image, latent

File ~/projects/prompt-to-prompt/ptp_utils.py:70, in diffusion_step(model, controller, latents, context, t, guidance_scale, low_resource)
     68 else:
     69     latents_input = torch.cat([latents] * 2)
---> 70     noise_pred = model.unet(latents_input, t, encoder_hidden_states=context)["sample"]
     71     noise_pred_uncond, noise_prediction_text = noise_pred.chunk(2)
     72 noise_pred = noise_pred_uncond + guidance_scale * (noise_prediction_text - noise_pred_uncond)

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py:424, in UNet2DConditionModel.forward(self, sample, timestep, encoder_hidden_states, class_labels, attention_mask, return_dict)
    422 for downsample_block in self.down_blocks:
    423     if hasattr(downsample_block, "has_cross_attention") and downsample_block.has_cross_attention:
--> 424         sample, res_samples = downsample_block(
    425             hidden_states=sample,
    426             temb=emb,
    427             encoder_hidden_states=encoder_hidden_states,
    428             attention_mask=attention_mask,
    429         )
    430     else:
    431         sample, res_samples = downsample_block(hidden_states=sample, temb=emb)

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py:777, in CrossAttnDownBlock2D.forward(self, hidden_states, temb, encoder_hidden_states, attention_mask)
    775     else:
    776         hidden_states = resnet(hidden_states, temb)
--> 777         hidden_states = attn(hidden_states, encoder_hidden_states=encoder_hidden_states).sample
    779     output_states += (hidden_states,)
    781 if self.downsamplers is not None:

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/diffusers/models/attention.py:216, in Transformer2DModel.forward(self, hidden_states, encoder_hidden_states, timestep, return_dict)
    214 # 2. Blocks
    215 for block in self.transformer_blocks:
--> 216     hidden_states = block(hidden_states, encoder_hidden_states=encoder_hidden_states, timestep=timestep)
    218 # 3. Output
    219 if self.is_input_continuous:

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/diffusers/models/attention.py:498, in BasicTransformerBlock.forward(self, hidden_states, encoder_hidden_states, timestep, attention_mask)
    492 if self.attn2 is not None:
    493     # 2. Cross-Attention
    494     norm_hidden_states = (
    495         self.norm2(hidden_states, timestep) if self.use_ada_layer_norm else self.norm2(hidden_states)
    496     )
    497     hidden_states = (
--> 498         self.attn2(
    499             norm_hidden_states, encoder_hidden_states=encoder_hidden_states, attention_mask=attention_mask
    500         )
    501         + hidden_states
    502     )
    504 # 3. Feed-forward
    505 hidden_states = self.ff(self.norm3(hidden_states)) + hidden_states

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/projects/prompt-to-prompt/ptp_utils.py:188, in register_attention_control.<locals>.ca_forward.<locals>.forward(x, context, mask, encoder_hidden_states, attention_mask)
    186 is_cross = context is not None
    187 context = context if is_cross else x
--> 188 k = self.to_k(context)
    189 v = self.to_v(context)
    190 q = self.reshape_heads_to_batch_dim(q)

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/promptop/lib/python3.10/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -> Tensor:
--> 114     return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (8192x320 and 768x320)

I get the same error message for null_text_w_ptp.ipynb at the same function :

k = self.to_k(context)

I work with the following versions :

cuda==11.6
pytorch==1.13.1
diffusers == 0.11.1
transformers == 4.25.1
xformers == 0.0.16.dev430+git.bac8718 py310_cu11.6_pyt1.13.1

[ntlm1686]

@yasserben

Just downgrade your diffusers to 0.8.1, and it will work.

pip uninstall "diffusers[torch]"
pip install "diffusers[torch]==0.8.1"

[yasserben]

@ntlm1686

it works now, thank you very much ! 👍

[Hades32]

@ouhenio nice work! Did you get it to work on a free Collab? The readme stated it should run with 12GB of VRAM, but combined with SD it takes more than 16GB for me...

[ouhenio]

Hey @Hades32! Thanks for pointing this out. I just realized that I developed this using Colab Pro. So my answer is no.

I guess you could try with the "fp16" version of SD, since its less memory hungry. If this doesn't work, then it becomes tricky. Memory optimizations of SD tend to touch the attention pipeline, and this paper implementation uses a specific attention scheme. 😕

[Hades32]

Thanks @ouhenio, that got me a good bit further but seems I fp16 needs to put in lots of places as I'm running in type errors. And the first produces just black images, so I'm not sure this will actually works.
So, thanks anyway, I'll try somewhere with more vram

[GongXinyuu]

@yasserben

Just downgrade your diffusers to 0.8.1, and it will work.

pip uninstall "diffusers[torch]"
pip install "diffusers[torch]==0.8.1"

After doing this, I met an error of TypeError: '<' not supported between instances of 'TorchVersion' and 'Version'.

[SleeplessBegonia]

@yasserben
Just downgrade your diffusers to 0.8.1, and it will work.

pip uninstall "diffusers[torch]"
pip install "diffusers[torch]==0.8.1"

After doing this, I met an error of TypeError: '<' not supported between instances of 'TorchVersion' and 'Version'.

me too,how did you solve?

[Weifeng-Chen]

【自动恢复】来信已收到，我将尽快回复你！