Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental support for loading .safetensors models #163

Merged
merged 1 commit into from
May 25, 2024

Conversation

monstruosoft
Copy link
Contributor

As the title says, this PR adds experimental support for loading models in safetensors format; I call it experimental because, even if it technically works, I've noticed that rebuilding the pipeline a couple of times, for example by enabling/disabling ControlNet, causes RAM usage to increase considerably making it impractical for its use on low-end PCs.

Some other things to note in this PR:

  • Loading safetensors models only works in LCM-LoRA mode, I think it's the only mode where it makes sense.
  • LCM-LoRA mode was always considerably slower than plain LCM mode on my machine, about 50% slower, I always thought that it was a speed hit from the LCM-LoRA mode itself but I noticed that by commenting this line:
    pipeline.unet.to(memory_format=torch.channels_last)
    inference speed automatically became much better. Is there any particular reason why this line is used?.
  • By going further and always fusing the LCM-LoRA with the base model, inference speed in LCM-LoRA mode gets almost as fast as inference speed in plain LCM mode.

Add experimental support for loading .safetensors models
@captainerd
Copy link

pipe = pipeline_class(
TypeError: StableDiffusionPipeline.init() got an unexpected keyword argument 'text_encoder_2'

I tried to load sd_xl_base_1.0.safetensors

It loads fine the same model as diffusers

Am i missing something?

@monstruosoft
Copy link
Contributor Author

Sorry, should have mentioned that this PR works only for SD 1.5 models. A similar code should work for SDXL but unfortunately I can't run SDXL on my machine.

@rupeshs
Copy link
Owner

rupeshs commented May 25, 2024

@monstruosoft Going to merge this PR,thanks

@rupeshs rupeshs merged commit 6a86059 into rupeshs:main May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants