Example workflow for SDXL

[jkcarney]

My research organization received access to SDXL. ComfyUI seems to work with the stable-diffusion-xl-base-0.9 fine, but when I try to add in the stable-diffusion-xl-refiner-0.9, I run into issues.

High likelihood is that I am misunderstanding how I use both in conjunction within comfy. If there was an example workflow or method for using both the base and refiner in one workflow, that would be very helpful.

Here's my current workflow

Here's the error:

!!! Exception during processing !!!
Traceback (most recent call last):
  File "/home/carnejk1/ComfyUI/execution.py", line 145, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/carnejk1/ComfyUI/execution.py", line 75, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/carnejk1/ComfyUI/execution.py", line 68, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/carnejk1/ComfyUI/nodes.py", line 1028, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/home/carnejk1/ComfyUI/nodes.py", line 998, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/home/carnejk1/ComfyUI/comfy/sample.py", line 88, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/home/carnejk1/ComfyUI/comfy/samplers.py", line 676, in sample
    samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/k_diffusion/sampling.py", line 128, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/samplers.py", line 308, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/k_diffusion/external.py", line 114, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/k_diffusion/external.py", line 140, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/samplers.py", line 296, in apply_model
    out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
  File "/home/carnejk1/ComfyUI/comfy/samplers.py", line 274, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
  File "/home/carnejk1/ComfyUI/comfy/samplers.py", line 251, in calc_cond_uncond_batch
    output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
  File "/home/carnejk1/ComfyUI/comfy/model_base.py", line 55, in apply_model
    return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 834, in forward
    h = forward_timestep_embed(module, h, emb, context, transformer_options)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 97, in forward_timestep_embed
    x = layer(x, context, transformer_options)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/attention.py", line 692, in forward
    x = block(x, context=context[i], transformer_options=transformer_options)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/attention.py", line 524, in forward
    return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 123, in checkpoint
    return func(*inputs)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/attention.py", line 624, in _forward
    n = self.attn2(n, context=context_attn2, value=value_attn2)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/ldm/modules/attention.py", line 419, in forward
    k = self.to_k(context)
  File "/home/carnejk1/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/carnejk1/ComfyUI/comfy/ops.py", line 18, in forward
    return torch.nn.functional.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x2048 and 1280x768)

[comfyanonymous]

The refiner has a different conditioning than the base model so you have to use the CLIP from the refiner to sample with the refiner.

[jkcarney]

So effectively the refiner can be thought of as an additional step that is almost like img2img, am I correct?:

[comfyanonymous]

Yes the refiner is basically an img2img step. If you want a workflow that matches more closely (but not exactly) the SDXL workflow you can try this. Copy paste it on the UI to load it.

{
  "last_node_id": 20,
  "last_link_id": 37,
  "nodes": [
    {
      "id": 8,
      "type": "VAEDecode",
      "pos": [
        1209,
        188
      ],
      "size": {
        "0": 210,
        "1": 46
      },
      "flags": {},
      "order": 10,
      "mode": 0,
      "inputs": [
        {
          "name": "samples",
          "type": "LATENT",
          "link": 28
        },
        {
          "name": "vae",
          "type": "VAE",
          "link": 8
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            9
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "VAEDecode"
      }
    },
    {
      "id": 15,
      "type": "PrimitiveNode",
      "pos": [
        -52,
        -156
      ],
      "size": [
        402,
        188
      ],
      "flags": {},
      "order": 0,
      "mode": 0,
      "outputs": [
        {
          "name": "STRING",
          "type": "STRING",
          "links": [
            18,
            19
          ],
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          },
          "slot_index": 0
        }
      ],
      "title": "Positive Prompt",
      "properties": {},
      "widgets_values": [
        "beautiful scenery nature glass bottle landscape, , purple galaxy bottle,"
      ]
    },
    {
      "id": 16,
      "type": "PrimitiveNode",
      "pos": [
        -51,
        77
      ],
      "size": [
        398,
        140
      ],
      "flags": {},
      "order": 1,
      "mode": 0,
      "outputs": [
        {
          "name": "STRING",
          "type": "STRING",
          "links": [
            21,
            22
          ],
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          },
          "slot_index": 0
        }
      ],
      "title": "Negative Prompt",
      "properties": {},
      "widgets_values": [
        "text, watermark"
      ]
    },
    {
      "id": 6,
      "type": "CLIPTextEncode",
      "pos": [
        445,
        155
      ],
      "size": [
        210,
        54
      ],
      "flags": {},
      "order": 5,
      "mode": 0,
      "inputs": [
        {
          "name": "clip",
          "type": "CLIP",
          "link": 3
        },
        {
          "name": "text",
          "type": "STRING",
          "link": 19,
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          }
        }
      ],
      "outputs": [
        {
          "name": "CONDITIONING",
          "type": "CONDITIONING",
          "links": [
            24
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "CLIPTextEncode"
      },
      "widgets_values": [
        "beautiful scenery nature glass bottle landscape, , purple galaxy bottle,"
      ]
    },
    {
      "id": 7,
      "type": "CLIPTextEncode",
      "pos": [
        449,
        264
      ],
      "size": [
        210,
        54
      ],
      "flags": {},
      "order": 6,
      "mode": 0,
      "inputs": [
        {
          "name": "clip",
          "type": "CLIP",
          "link": 5
        },
        {
          "name": "text",
          "type": "STRING",
          "link": 22,
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          }
        }
      ],
      "outputs": [
        {
          "name": "CONDITIONING",
          "type": "CONDITIONING",
          "links": [
            25
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "CLIPTextEncode"
      },
      "widgets_values": [
        "text, watermark"
      ]
    },
    {
      "id": 5,
      "type": "EmptyLatentImage",
      "pos": [
        389,
        431
      ],
      "size": {
        "0": 315,
        "1": 106
      },
      "flags": {},
      "order": 2,
      "mode": 0,
      "outputs": [
        {
          "name": "LATENT",
          "type": "LATENT",
          "links": [
            26
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "EmptyLatentImage"
      },
      "widgets_values": [
        1024,
        1024,
        1
      ]
    },
    {
      "id": 4,
      "type": "CheckpointLoaderSimple",
      "pos": [
        -60,
        319
      ],
      "size": [
        397,
        98
      ],
      "flags": {},
      "order": 3,
      "mode": 0,
      "outputs": [
        {
          "name": "MODEL",
          "type": "MODEL",
          "links": [
            27
          ],
          "slot_index": 0
        },
        {
          "name": "CLIP",
          "type": "CLIP",
          "links": [
            3,
            5
          ],
          "slot_index": 1
        },
        {
          "name": "VAE",
          "type": "VAE",
          "links": [
            8
          ],
          "slot_index": 2
        }
      ],
      "properties": {
        "Node name for S&R": "CheckpointLoaderSimple"
      },
      "widgets_values": [
        "sd_xl_base_0.9.safetensors"
      ]
    },
    {
      "id": 9,
      "type": "SaveImage",
      "pos": [
        1451,
        189
      ],
      "size": [
        426.0842159050553,
        437.66388141222205
      ],
      "flags": {},
      "order": 12,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 9
        }
      ],
      "properties": {},
      "widgets_values": [
        "base_output"
      ]
    },
    {
      "id": 11,
      "type": "CheckpointLoaderSimple",
      "pos": [
        16,
        -385
      ],
      "size": [
        336,
        98
      ],
      "flags": {},
      "order": 4,
      "mode": 0,
      "outputs": [
        {
          "name": "MODEL",
          "type": "MODEL",
          "links": [
            34
          ],
          "shape": 3,
          "slot_index": 0
        },
        {
          "name": "CLIP",
          "type": "CLIP",
          "links": [
            11,
            13
          ],
          "shape": 3,
          "slot_index": 1
        },
        {
          "name": "VAE",
          "type": "VAE",
          "links": [
            37
          ],
          "shape": 3,
          "slot_index": 2
        }
      ],
      "properties": {
        "Node name for S&R": "CheckpointLoaderSimple"
      },
      "widgets_values": [
        "sd_xl_refiner_0.9.safetensors"
      ]
    },
    {
      "id": 12,
      "type": "CLIPTextEncode",
      "pos": [
        876,
        -355
      ],
      "size": [
        219,
        54
      ],
      "flags": {},
      "order": 7,
      "mode": 0,
      "inputs": [
        {
          "name": "clip",
          "type": "CLIP",
          "link": 11
        },
        {
          "name": "text",
          "type": "STRING",
          "link": 18,
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          },
          "slot_index": 1
        }
      ],
      "outputs": [
        {
          "name": "CONDITIONING",
          "type": "CONDITIONING",
          "links": [
            32
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "CLIPTextEncode"
      },
      "widgets_values": [
        "beautiful scenery nature glass bottle landscape, , purple galaxy bottle,"
      ]
    },
    {
      "id": 13,
      "type": "CLIPTextEncode",
      "pos": [
        884,
        -241
      ],
      "size": [
        210,
        54
      ],
      "flags": {},
      "order": 8,
      "mode": 0,
      "inputs": [
        {
          "name": "clip",
          "type": "CLIP",
          "link": 13
        },
        {
          "name": "text",
          "type": "STRING",
          "link": 21,
          "widget": {
            "name": "text",
            "config": [
              "STRING",
              {
                "multiline": true
              }
            ]
          },
          "slot_index": 1
        }
      ],
      "outputs": [
        {
          "name": "CONDITIONING",
          "type": "CONDITIONING",
          "links": [
            33
          ],
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "CLIPTextEncode"
      },
      "widgets_values": [
        "text, watermark"
      ]
    },
    {
      "id": 20,
      "type": "KSampler",
      "pos": [
        1184,
        -343
      ],
      "size": {
        "0": 315,
        "1": 262
      },
      "flags": {},
      "order": 11,
      "mode": 0,
      "inputs": [
        {
          "name": "model",
          "type": "MODEL",
          "link": 34
        },
        {
          "name": "positive",
          "type": "CONDITIONING",
          "link": 32
        },
        {
          "name": "negative",
          "type": "CONDITIONING",
          "link": 33
        },
        {
          "name": "latent_image",
          "type": "LATENT",
          "link": 35
        }
      ],
      "outputs": [
        {
          "name": "LATENT",
          "type": "LATENT",
          "links": [
            36
          ],
          "shape": 3,
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "KSampler"
      },
      "widgets_values": [
        382444612533601,
        "randomize",
        15,
        8,
        "dpmpp_2m",
        "normal",
        0.25
      ]
    },
    {
      "id": 18,
      "type": "VAEDecode",
      "pos": [
        1625,
        -332
      ],
      "size": {
        "0": 210,
        "1": 46
      },
      "flags": {},
      "order": 13,
      "mode": 0,
      "inputs": [
        {
          "name": "samples",
          "type": "LATENT",
          "link": 36
        },
        {
          "name": "vae",
          "type": "VAE",
          "link": 37
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            30
          ],
          "shape": 3,
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "VAEDecode"
      }
    },
    {
      "id": 19,
      "type": "SaveImage",
      "pos": [
        1904,
        -327
      ],
      "size": [
        315,
        270
      ],
      "flags": {},
      "order": 14,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 30
        }
      ],
      "properties": {},
      "widgets_values": [
        "refiner_output"
      ]
    },
    {
      "id": 17,
      "type": "KSampler",
      "pos": [
        779,
        158
      ],
      "size": {
        "0": 315,
        "1": 262
      },
      "flags": {},
      "order": 9,
      "mode": 0,
      "inputs": [
        {
          "name": "model",
          "type": "MODEL",
          "link": 27
        },
        {
          "name": "positive",
          "type": "CONDITIONING",
          "link": 24
        },
        {
          "name": "negative",
          "type": "CONDITIONING",
          "link": 25
        },
        {
          "name": "latent_image",
          "type": "LATENT",
          "link": 26
        }
      ],
      "outputs": [
        {
          "name": "LATENT",
          "type": "LATENT",
          "links": [
            28,
            35
          ],
          "shape": 3,
          "slot_index": 0
        }
      ],
      "properties": {
        "Node name for S&R": "KSampler"
      },
      "widgets_values": [
        443819846053208,
        "randomize",
        20,
        6,
        "dpmpp_2s_ancestral",
        "normal",
        1
      ]
    }
  ],
  "links": [
    [
      3,
      4,
      1,
      6,
      0,
      "CLIP"
    ],
    [
      5,
      4,
      1,
      7,
      0,
      "CLIP"
    ],
    [
      8,
      4,
      2,
      8,
      1,
      "VAE"
    ],
    [
      9,
      8,
      0,
      9,
      0,
      "IMAGE"
    ],
    [
      11,
      11,
      1,
      12,
      0,
      "CLIP"
    ],
    [
      13,
      11,
      1,
      13,
      0,
      "CLIP"
    ],
    [
      18,
      15,
      0,
      12,
      1,
      "STRING"
    ],
    [
      19,
      15,
      0,
      6,
      1,
      "STRING"
    ],
    [
      21,
      16,
      0,
      13,
      1,
      "STRING"
    ],
    [
      22,
      16,
      0,
      7,
      1,
      "STRING"
    ],
    [
      24,
      6,
      0,
      17,
      1,
      "CONDITIONING"
    ],
    [
      25,
      7,
      0,
      17,
      2,
      "CONDITIONING"
    ],
    [
      26,
      5,
      0,
      17,
      3,
      "LATENT"
    ],
    [
      27,
      4,
      0,
      17,
      0,
      "MODEL"
    ],
    [
      28,
      17,
      0,
      8,
      0,
      "LATENT"
    ],
    [
      30,
      18,
      0,
      19,
      0,
      "IMAGE"
    ],
    [
      32,
      12,
      0,
      20,
      1,
      "CONDITIONING"
    ],
    [
      33,
      13,
      0,
      20,
      2,
      "CONDITIONING"
    ],
    [
      34,
      11,
      0,
      20,
      0,
      "MODEL"
    ],
    [
      35,
      17,
      0,
      20,
      3,
      "LATENT"
    ],
    [
      36,
      20,
      0,
      18,
      0,
      "LATENT"
    ],
    [
      37,
      11,
      2,
      18,
      1,
      "VAE"
    ]
  ],
  "groups": [],
  "config": {},
  "extra": {},
  "version": 0.4
}

[jkcarney]

Thank you; I will close this. I think an example of a SDXL workflow in the ui prior to the full release would be wise, as I think there are plenty of users who are more clueless than myself!

[Tigwin]

I put the safetensors files for both base and refiner into the models/ folder and tried the workflow above. I get errors.
Any idea? Do I need the other files that came with base and refiner?

!!! Exception during processing !!!
Traceback (most recent call last):
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 141, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 68, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 351, in load_checkpoint
out = comfy.sd.load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True, embedding_directory=folder_paths.get_folder_paths("embeddings"))
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 1161, in load_checkpoint_guess_config
model = load_model_weights(model, sd, verbose=False, load_state_dict_to=load_state_dict_to)
File "C:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 28, in load_model_weights
m, u = model.load_state_dict(sd, strict=False)
File "C:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for BaseModel:
size mismatch for diffusion_model.output_blocks.2.0.in_layers.0.weight: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
size mismatch for diffusion_model.output_blocks.2.0.in_layers.0.bias: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
size mismatch for diffusion_model.output_blocks.2.0.in_layers.2.weight: copying a param with shape torch.Size([1280, 1920, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280, 2560, 3, 3]).
size mismatch for diffusion_model.output_blocks.2.0.skip_connection.weight: copying a param with shape torch.Size([1280, 1920, 1, 1]) from checkpoint, the shape in current model is torch.Size([1280, 2560, 1, 1]).
size mismatch for diffusion_model.output_blocks.3.0.in_layers.0.weight: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
size mismatch for diffusion_model.output_blocks.3.0.in_layers.0.bias: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
size mismatch for diffusion_model.output_blocks.3.0.in_layers.2.weight: copying a param with shape torch.Size([640, 1920, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280, 2560, 3, 3]).
size mismatch for diffusion_model.output_blocks.3.0.in_layers.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.0.emb_layers.1.weight: copying a param with shape torch.Size([640, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280]).
size mismatch for diffusion_model.output_blocks.3.0.emb_layers.1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.0.out_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.0.out_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 3, 3]).
size mismatch for diffusion_model.output_blocks.3.0.out_layers.3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.0.skip_connection.weight: copying a param with shape torch.Size([640, 1920, 1, 1]) from checkpoint, the shape in current model is torch.Size([1280, 2560, 1, 1]).
size mismatch for diffusion_model.output_blocks.3.0.skip_connection.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.1.norm.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.1.norm.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
size mismatch for diffusion_model.output_blocks.3.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280]).
size mismatch for diffusion_model.output_blocks.3.1.proj_in.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).

edit: figured it out. needed to update comfy before it would run

[Hinkiii]

Any idea how to have it also work with a lora? I just get errors when trying