Using a custom pipeline with from_ckpt.

[sertdfyguhi]

Hey, it seems like loading custom pipelines with from_ckpt is currently not implemented. Is there a workaround for this and is this feature planned for the future? Thanks in advance!

[patrickvonplaten]

We have indeed not added such a functionality yet. Could you copy-paste your use case here and I'm sure we'll find a nice workaround :-)

[sertdfyguhi]

Sure! I'm currently making a stable diffusion UI with diffusers so that's why I would love to have the functionality to load ckpt and safetensors files while still be able to use the LPW stable diffusion pipeline for its infinite prompt length and prompt weighting. Some example code would be like this:

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_ckpt(
    "path/to/ckpt/file", custom_pipeline="lpw_stable_diffusion"
).to("cuda")

image = pipe(
    prompt="user prompt",
    num_inference_steps=20,
).images[0]

# Use the image to show in UI

[alexblattner]

@sertdfyguhi I just copied the entirety of txt2img pipeline, modified it and then called by its new name like it was nothing. works with from_ckpt

[patrickvonplaten]

@sertdfyguhi,

Could you for now use this script to load the custom pipeline with from_ckpt:

#!/usr/bin/env python3
from diffusers import StableDiffusionPipeline
from diffusers.utils import get_class_from_dynamic_module

pipe = StableDiffusionPipeline.from_ckpt("path/to/ckpt/file").to("cuda")

LPWStableDiffusionPipeline = get_class_from_dynamic_module("lpw_stable_diffusion", module_file="lpw_stable_diffusion.py")
lpw_pipe = LPWStableDiffusionPipeline(**pipe.components)


image = lpw_pipe(
    prompt="user prompt",
    num_inference_steps=20,
).images[0]
# Use the image to show in UI

[sertdfyguhi]

@patrickvonplaten Thank you very much! Will absolutely do.

[sertdfyguhi]

Hey, it seems like this code has broken due to version differences (perhaps because the LPW stable diffusion file was updated in this repo?). Here's the error I got:

File "/Users/test/programming/onGAU/onGAU/imagen/base.py", line 56, in <module>
    StableDiffusionLongPromptWeightingPipeline = get_class_from_dynamic_module(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/diffusers/utils/dynamic_modules_utils.py", line 445, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/diffusers/utils/dynamic_modules_utils.py", line 264, in get_cached_module_file
    available_versions = get_diffusers_versions()
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/diffusers/utils/dynamic_modules_utils.py", line 46, in get_diffusers_versions
    return sorted(releases, key=StrictVersion)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/version.py", line 54, in __init__
    self.parse(vstring)
  File "/opt/homebrew/lib/python3.11/site-packages/setuptools/_distutils/version.py", line 157, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '0.17.0rc0'

[patrickvonplaten]

see: #3703

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[agarwalml]

I'm trying the same thing using from_single_file() (assuming this is a renamed from_ckpt().

So far, this is what I have:
pipe = StableDiffusionPipeline.from_single_file(
checkpoint,
torch_dtype=torch.float16,
variant="fp16",
clip_skip=2,
)

# Load custom VAE for sketch and anime checkpoints
if "sketch" in checkpoint:
    vae = AutoencoderKL.from_single_file("models/vaes/klf8vae.safetensors", torch_dtype=torch.float16)
    pipe.vae = vae
    print(f"Loaded custom VAE for sketch")
    
# Load the custom LPW pipeline
from diffusers.utils import get_class_from_dynamic_module
LPWStableDiffusionPipeline = get_class_from_dynamic_module("lpw_stable_diffusion", module_file="lpw_stable_diffusion.py")

# Filter out 'image_encoder' from components
components = {k: v for k, v in pipe.components.items() if k != 'image_encoder'}
pipe = LPWStableDiffusionPipeline(**components)

This file keeps getting reloaded depending on me loading the VAE or not, which causes my server to shut down. Any clue how I can keep the .py file stable?