Using Dreambooth Model that wasn't trained with inpainting

[AmanKishore]

model = EditAnythingLoraModel(base_model_path="nitrosocke/Ghibli-Diffusion",
                                  controlmodel_name='LAION Pretrained(v0-4)-SD15', extra_inpaint=True,
                                  lora_model_path=None, use_blip=True)

Hello when running the above code, I am getting the following error:

Traceback (most recent call last):
  File "/home/amankishore/.local/lib/python3.8/site-packages/gradio/routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/amankishore/.local/lib/python3.8/site-packages/gradio/blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "/home/amankishore/.local/lib/python3.8/site-packages/gradio/blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/amankishore/.local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/amankishore/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/amankishore/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/amankishore/mirage/deep-learning/EditAnything/sam2edit_lora.py", line 446, in process
    x_samples = self.pipe(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/amankishore/mirage/deep-learning/EditAnything/utils/stable_diffusion_controlnet_inpaint.py", line 1086, in __call__
    down_block_res_samples, mid_block_res_sample = self.controlnet(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py", line 125, in forward
    down_samples, mid_sample = controlnet(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/controlnet.py", line 526, in forward
    sample, res_samples = downsample_block(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/unet_2d_blocks.py", line 867, in forward
    hidden_states = attn(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/transformer_2d.py", line 265, in forward
    hidden_states = block(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/attention.py", line 331, in forward
    attn_output = self.attn2(
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/attention_processor.py", line 267, in forward
    return self.processor(
  File "/home/amankishore/.local/lib/python3.8/site-packages/diffusers/models/attention_processor.py", line 689, in __call__
    key = attn.to_k(encoder_hidden_states)
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/amankishore/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (240x768 and 1024x320)

[gasvn]

Can you give me more details? Current version support the lora model without inpaiting ability. Also, can you be sure that nitrosocke/Ghibli-Diffusion is a base model based on SD1.5?

[AmanKishore]

I see, I am trying to use a Dreambooth that was trained without inpainting, is that possible? Yes the model was trained on SD 1.5!

[gasvn]

Thanks for more details. This is a bug caused by the default settings for condition model in gradio GUI. I have fixed this. You can try the latest version.

[AmanKishore]

Awesome thank you!