[LoRA] fix: disabling hooks when loading loras.

[sayakpaul]

What does this PR do?

Fixes huggingface/accelerate#3672

from diffusers import AutoPipelineForText2Image, DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()
pipe.load_lora_weights(
    "renderartist/retrocomicflux", 
    weight_name="Retro_Comic_Flux_v1_renderartist.safetensors", 
    adapter_name="test"
)

Fails with:

AttributeError: 'NoneType' object has no attribute '__dict__'. Did you mean: '__dir__'?

[yiyixuxu]

thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

Thanks!