SDXL Diffusers checkpoint support?

[kalkal11]

Hey, this is for the purpose of model development - we end up with a lot of large checkpoints and being able to only load in unet separately and reference the same clip model and vae would be any helpful. (save a huge amount of space)

Is there currently a way to load in SDXL diffusers models at all?

The advanced (and depreciated) diffusers loader doesn't seem to load in the config correctly.

Loading in each component doesn't seem to allow to specify a config - so it's looking like this might not be possible but perhaps I'm missing something.

Thanks

[ltdrdata]

ComfyUI supports, unet, diffusers loading for SDXL, already.

acf9519

If it isn't working properly. It should be issue.

[kalkal11]

Well woah there with the issue - there's no guide on usage so before we go there, would like to confirm I'm doing everything correctly first. Will raise an issue in the event that it's confirmed that I'm doing everything correctly and it still isn't working as intended.

I've tried every combo I can think of, loading in clip/vae from a ckpt and unet like you've suggested here, loading in each component with the corresponding loader too also the depreciated diffusers loader - however as I've seen no reference to the correct usage, I can't be absolutely certain I've set it up correctly.

So I've been in a setup where all the diffusers components have been split across the corresponding folders in the models folder. I've also tried loading in via a full diffusers model inside the diffusers folder.

[ltdrdata]

That is true. We need documentation.

I think @comfyanonymous is the only one who can explain this.

Maybe comfy implemented this for other researchers.

[comfyanonymous]

Don't use the diffusers loader,

use: Loaders -> Load VAE, it will work with diffusers vae files.

Advanced -> loaders -> UNET loader will work with the diffusers unet files.

Advanced -> loaders -> DualClipLoader (For SDXL base) or Load CLIP (for other models) will work with diffusers text encoder files.

[kalkal11]

Ah, this is actually the first thing I tried, perhaps I should open an issue after all?

Error occurred when executing UNETLoader:

'down_blocks.0.attentions.1.transformer_blocks.0.attn2.to_k.weight'

File "/home/ubuntu/ComfyUI/execution.py", line 145, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/ubuntu/ComfyUI/execution.py", line 75, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/ubuntu/ComfyUI/execution.py", line 68, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/ubuntu/ComfyUI/nodes.py", line 598, in load_unet
model = comfy.sd.load_unet(unet_path)
File "/home/ubuntu/ComfyUI/comfy/sd.py", line 1132, in load_unet
match["context_dim"] = sd["down_blocks.0.attentions.1.transformer_blocks.0.attn2.to_k.weight"].shape[1]

[comfyanonymous]

Is this base or the refiner?

[TheLastBen]

same here, base model

[kalkal11]

@comfyanonymous finetuned base

[comfyanonymous]

Should be fixed now.

[kalkal11]

Hey thanks for the prompt fix, loading unet doesn't cause a crash now, however - ksampler does instead -

Error occurred when executing KSampler:

No operator found for `memory_efficient_attention_forward` with inputs:
query : shape=(20, 4096, 1, 64) (torch.float16)
key : shape=(20, 4096, 1, 64) (torch.float16)
value : shape=(20, 4096, 1, 64) (torch.float16)
attn_bias :
p : 0.0
`flshattF` is not supported because:
xFormers wasn't build with CUDA support
Operator wasn't built - see `python -m xformers.info` for more info
`tritonflashattF` is not supported because:
xFormers wasn't build with CUDA support
requires A100 GPU
Only work on pre-MLIR triton for now
`cutlassF` is not supported because:
xFormers wasn't build with CUDA support
Operator wasn't built - see `python -m xformers.info` for more info
`smallkF` is not supported because:
xFormers wasn't build with CUDA support
dtype=torch.float16 (supported: {torch.float32})
max(query.shape[-1] != value.shape[-1]) > 32
Operator wasn't built - see `python -m xformers.info` for more info
unsupported embed per head: 64

File "/home/ubuntu/ComfyUI/execution.py", line 145, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/ubuntu/ComfyUI/execution.py", line 75, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/ubuntu/ComfyUI/execution.py", line 68, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/ubuntu/ComfyUI/nodes.py", line 1082, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "/home/ubuntu/ComfyUI/nodes.py", line 1052, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "/home/ubuntu/ComfyUI/comfy/sample.py", line 88, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 671, in sample
samples = getattr(k_diffusion_sampling, "sample_{}".format(self.sampler))(self.model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/sampling.py", line 138, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 308, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/external.py", line 125, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/home/ubuntu/ComfyUI/comfy/k_diffusion/external.py", line 151, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 296, in apply_model
out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 274, in sampling_function
cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
File "/home/ubuntu/ComfyUI/comfy/samplers.py", line 251, in calc_cond_uncond_batch
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
File "/home/ubuntu/ComfyUI/comfy/model_base.py", line 61, in apply_model
return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float()
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 834, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 97, in forward_timestep_embed
x = layer(x, context, transformer_options)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/attention.py", line 695, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/attention.py", line 527, in forward
return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 123, in checkpoint
return func(*inputs)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/attention.py", line 590, in _forward
n = self.attn1(n, context=context_attn1, value=value_attn1)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1522, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1531, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/ComfyUI/comfy/ldm/modules/attention.py", line 440, in forward
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=self.attention_op)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/xformers/ops/fmha/__init__.py", line 192, in memory_efficient_attention
return _memory_efficient_attention(
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/xformers/ops/fmha/__init__.py", line 290, in _memory_efficient_attention
return _memory_efficient_attention_forward(
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/xformers/ops/fmha/__init__.py", line 306, in _memory_efficient_attention_forward
op = _dispatch_fw(inp)
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/xformers/ops/fmha/dispatch.py", line 94, in _dispatch_fw
return _run_priority_list(
File "/home/ubuntu/ve_comfy/lib/python3.8/site-packages/xformers/ops/fmha/dispatch.py", line 69, in _run_priority_list
raise NotImplementedError(msg)

[comfyanonymous]

xFormers wasn't build with CUDA support

That sounds like an xformers issue, you can try running it with: --use-pytorch-cross-attention

[TheLastBen]

UNet loader works fine for me now