Disabling FreeU in StableDiffusionXLPipeline if its not enabled throws an error

[Vargol]

Describe the bug

If you disable FreeU before enabling it you get an error, at least of SDXL not relied any others

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-5-210568d4bac8>](https://805ldlttw9r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231024-060124_RC00_576097381#) in <cell line: 9>()
      7 
      8 #pipe.enable_freeu(s1=0.95, s2=1.0, b1=1.0, b2=1.05)
----> 9 pipe.disable_freeu()
     10 

2 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py](https://805ldlttw9r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231024-060124_RC00_576097381#) in disable_freeu(self)
    619     def disable_freeu(self):
    620         """Disables the FreeU mechanism if enabled."""
--> 621         self.unet.disable_freeu()
    622 
    623     @torch.no_grad()

[/usr/local/lib/python3.10/dist-packages/diffusers/models/unet_2d_condition.py](https://805ldlttw9r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231024-060124_RC00_576097381#) in disable_freeu(self)
    792         for i, upsample_block in enumerate(self.up_blocks):
    793             for k in freeu_keys:
--> 794                 if hasattr(upsample_block, k) or getattr(upsample_block, k) is not None:
    795                     setattr(upsample_block, k, None)
    796 

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://805ldlttw9r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231024-060124_RC00_576097381#) in __getattr__(self, name)
   1693             if name in modules:
   1694                 return modules[name]
-> 1695         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1696 
   1697     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'CrossAttnUpBlock2D' object has no attribute 'b1'

Reproduction

pip install --upgrade  git+https://github.com/huggingface/diffusers.git transformers accelerate 

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
      "stabilityai/stable-diffusion-xl-base-1.0",
      torch_dtype=torch.float16,
      use_safetensors=True,
      variant="fp16",
      ).to('cuda')

pipe.disable_freeu()

Logs

No response

System Info

• diffusers version: 0.22.0.dev0
• Platform: Linux-5.15.120+-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.1.0+cu118 (True)
• Huggingface_hub version: 0.17.3
• Transformers version: 4.34.1
• Accelerate version: 0.24.0
• xFormers version: not installed
• Using GPU in script?: YES
• Using distributed or parallel set-up in script?: NO

System is Free Tier Google Colab

Who can help?

@yiyixuxu @sayakpaul @DN6 @patrickvonplaten

[sayakpaul]

This is not a fully reproducible code snippet. We need a fully reproducible code snippet to look further.

[Vargol]

I'm sorry what do you think is missing, running that will fail with the error above, its the disable_freeu call that fails.

[sayakpaul]

Ah okay, got it. Will fix.

[sayakpaul]

#5552 should fix it.

[Vargol]

Yes the seems to fix the issue.

[sayakpaul]

@DN6 #5552 is not yet merged 😅