Merge pipelines and/or checkpoints

[jslegers]

Is your feature request related to a problem? Please describe.

I can't find a way to merge pipelines and/or checkpoints with each other.

Describe the solution you'd like.

I'd like to merge different Stable Diffusion models using diffusers.

AFAIK, diffusers doesn't provide that feature yet.

Right now I merge my LORAs with my models in diffusers, then save them to ckpt, and then use the script below to merge my models.

It would save me a lot of time if I could just merge straight from a StableDiffusionXLPipeline and I could skip the step in between.

But, right now, even just adding this script (or a variation of it) to diffusers/scripts would already be an improvement.

Describe alternatives you've considered.

I currently use the following code from eyriewow/merge-models :

import os
import argparse
import torch
from tqdm import tqdm

parser = argparse.ArgumentParser(description="Merge two models")
parser.add_argument("model_0", type=str, help="Path to model 0")
parser.add_argument("model_1", type=str, help="Path to model 1")
parser.add_argument("--alpha", type=float, help="Alpha value, optional, defaults to 0.5", default=0.5, required=False)
parser.add_argument("--output", type=str, help="Output file name, without extension", default="merged", required=False)
parser.add_argument("--device", type=str, help="Device to use, defaults to cpu", default="cpu", required=False)
parser.add_argument("--without_vae", action="store_true", help="Do not merge VAE", required=False)

args = parser.parse_args()

device = args.device
model_0 = torch.load(args.model_0, map_location=device)
model_1 = torch.load(args.model_1, map_location=device)
theta_0 = model_0["state_dict"]
theta_1 = model_1["state_dict"]
alpha = args.alpha

output_file = f'{args.output}-{str(alpha)[2:] + "0"}.ckpt'

# check if output file already exists, ask to overwrite
if os.path.isfile(output_file):
    print("Output file already exists. Overwrite? (y/n)")
    while True:
        overwrite = input()
        if overwrite == "y":
            break
        elif overwrite == "n":
            print("Exiting...")
            exit()
        else:
            print("Please enter y or n")


for key in tqdm(theta_0.keys(), desc="Stage 1/2"):
    # skip VAE model parameters to get better results(tested for anime models)
    # for anime model，with merging VAE model, the result will be worse (dark and blurry)
    if args.without_vae and "first_stage_model" in key:
        continue
        
    if "model" in key and key in theta_1:
        theta_0[key] = (1 - alpha) * theta_0[key] + alpha * theta_1[key]

for key in tqdm(theta_1.keys(), desc="Stage 2/2"):
    if "model" in key and key not in theta_0:
        theta_0[key] = theta_1[key]

print("Saving...")

torch.save({"state_dict": theta_0}, output_file)

print("Done!")

It usually gets the job done for ckpt files that are the same size, but that's it.

[asomoza]

Hi, there's a script in the examples you can use:

https://github.com/huggingface/diffusers/blob/main/examples/community/checkpoint_merger.py

I understand what you want and that this would help your use case, but the pipelines are used by researchers and people that wants to learn. Adding this will make the pipelines more complex with a functionality that most of the time won't be used so it's better to just keep it as a separate script.

You can still learn and adapt it so you can use it in your projects without the need to call an external file

[jslegers]

the pipelines are used by researchers and people that wants to learn

I'm a former senior full stack web developer who's currently trying to revive a dead (due to burnout) career.

I'd like to build a new career either as a "concept artist", a custom model developer or something related. Since the market for this type of job is only just starting to emerge, my exact focus isn't clear yet.

I'm not sure I qualify as a "researcher, but I most definitely fall under the category "people who want to learn".

Adding this will make the pipelines more complex with a functionality that most of the time won't be used

Merging models is pretty common in the Stable Diffusion ecosystem, and a standalone safetensors file has become the norm within this ecosystem.

While I agree it doesn't make sense to add mergers of checkpoints and/or conversion to various other formats to a generic DiffusionPipeline, IMHO they make perfect sense for a StableDiffusionXLPipeline.

In my experience, using Stable Diffusion since 2022, there's quite a demand for such features. And with the AUTOMATIC1111 web UI currently being blocked on free Google Colab tiers, many Stable Diffusion users, including myself, don't really have a decent alternative to do their model conversions & mergers.

Hi, there's a script in the examples you can use:

https://github.com/huggingface/diffusers/blob/main/examples/community/checkpoint_merger.py

I tried that one already.

As the documentation explains :

The checkpoint merging is currently memory intensive as it modifies
the weights of a DiffusionPipeline object in place. Expect at least 13GB
RAM usage on Kaggle GPU kernels and on Colab you might run out of
the 12GB memory even while merging two checkpoints.

Since, for the time being, I'm not making any money from this, and I don't have a desktop or laptop PC powerful enough to run Stable Diffusion locally, I'm stuck with the free tier of Google Collab.

Trying to initialize a CheckpointMergerPipeline with two SDXL checkpoints is pretty much impossible on a tree tier of Google Colab since you run out of your available RAM while the pipeline is loading.

With the merger script I've been using thusfar, I don't have that issue.

The merger script I'm using is pretty lightweight but produces results nontheless. I used it back in 2023 to create Epic Diffusion, by merging more than a dozen custom models.

[asomoza]

I don't really like to give my personal opinion as you're giving yours because people in Stable Diffusion tend to get very angry when someone doesn't share their opinion so I just stick to the facts:

Merging models is pretty common in the Stable Diffusion ecosystem

We support this but as SDXL is a big model, you will need good hardware for this, there's some people that still use SD 1.5 because they have low end GPUs but we can't do anything about that either.

standalone safetensors file has become the norm within this ecosystem

We fully support single file checkpoints.

there's quite a demand for such features. And with the AUTOMATIC1111 web UI currently being blocked on free Google Colab tiers, many Stable Diffusion users, including myself, don't really have a decent alternative to do their model conversions & mergers.

We don't get that much requests for model merging, most people that use this library are more interested in the model training or finetuning, you can just search for this in the issues or discussions.

Trying to initialize a CheckpointMergerPipeline with two SDXL checkpoints is pretty much impossible on a tree tier of Google Colab since you run out of your available RAM while the pipeline is loading.

You will have the same problem with what you're asking since you'll still need to initialize two SDXL pipelines which have everything loaded, you will need the VRAM or RAM to fit both of them at the same time.

Since you wrote that you want to learn, merging two SDXL checkpoints should be doable in a free tier colab, you will have to change the script a little, to merge just the unets first and then the text encoders (if needed) and
you should use the GPU (16GB VRAM instead of 12 GB of RAM).

Also maybe you can also try this one from kohya which has the option to just merge the unets and will save RAM and VRAM.

[jslegers]

We support this but as SDXL is a big model, you will need good hardware for this, there's some people that still use SD 1.5 because they have low end GPUs but we can't do anything about that either.

Besides the CheckpointMergerPipeline, the free tier of Google Colab is good enough for my needs.

We fully support single file checkpoints.

A StableDiffusionXLPipeline can load only from a single checkpoint.

And to save to a single file checkpoint, you first need to save to diffusers format and then call convert_diffusers_to_original_sdxl.py, which is convoluted & buggy AF. See eg. #8974

Also, I have no idea how to use LoraLoaderMixin.save_lora_weights if I want to save just my adapter. The documentation is very unclear regarding the expected values for unet_lora_layers and text_encoder_lora_layers.

most people that use this library are more interested in the model training or finetuning

Possibly because model merging is such a mess when using diffusers.

I presume most just use the AUTOMATIC1111 web UI. But if you don't have access to that, you have little to no alternatives.

You will have the same problem with what you're asking since you'll still need to initialize two SDXL pipelines which have everything loaded, you will need the VRAM or RAM to fit both of them at the same time.

Using the script in my original post, I can merge just fine without my RAM getting maxed out.

Also maybe you can also try this one from kohya which has the option to just merge the unets and will save RAM and VRAM.

That's one of the first things I tried.

As far as I remember, it only supports merging LORAs, and it doesn't work for all LORAs.

The most effective method I found for merging LORas thusfar is running first load_lora_weights, then set_adapters, then fuse_lora and then save_pretrained on a DiffusionPipeline in diffusers. However, that gives you the model + the LORAS.

I have not yet found a reliable way to merge & save just the LORAs.

[asomoza]

It's really hard for me to tell if you're just here to criticize us or to ask for help. You just disregard any kind of help I give and just center on pointing your personal opinion to tell us that we're convoluted, a mess, have bad documentation and that automatic1111 is better.

I must repeat, free tier collab has 16GB of VRAM, you should be able to merge models with it, when I have time I can do a quick demo of it but for now I can just give you tips like I did in on my first answer.

I could answer all the other points but I don't think you would care about that because you already made your mind about them.

[jslegers]

It's really hard for me to tell if you're just here to criticize us or to ask for help.

I'm just frustrated, man.

I've wasted multiple days wading through the diffusers documentation & code just to get the most basic things done (features I expected to be supported out-of -the-box) in a way that's even remotable workable.

The more time I spend trying to get literally anything done with diffusers, the more I understand the dude who wrote this rant on Reddit.

You just disregard any kind of help I give

Everything you tell me to do is stuff I tried days ago, which I couldn't manage to get to work.

Also, you come off as quite elitist IMO. "Just get more budget" or "just get better hardware" isn't an option for everyone. Yet they doesn't mean they shouldn't be able to use diffusers.

SDXL actually runs pretty well in diffusers on a free Google Colab tier when all you need to do is loading a model, running a prompt & displaying its output.

I though there was no way I could run an SDXL model and generate output at 1024 x 1024 without running out of VRAM. Yet, quite to my surprise, that actually worked smoothly, and faster than some hosted solutions for Stable Diffusion. Even loading multiple LORAs with 3 or 4 LORAs, using different ratios, works surprisingly well on a free Google Colab tier. So, kudos for that!!

Merging different checkpoints also works fine when I use the script from my original post. Using that code, I'm able to achieve what CheckpointMergerPipeline is supposed to achieve, without blowing up available (V)RAM.

It's just an extremely tedious process that requires me to...

1. individually load the two models I'm trying to merge into a DiffusionPipepline and save to full precision diffusers format
2. convert both of the models I'm trying to merge individually to ckpt format using convert_diffusers_to_original_sdxl.py
3. run the merge script from my initial post to merge the two ckpt files into one
4. load the new ckpt into a DiffusionPipepline and save it half precision diffusers format

And that's not including the various saves to hub or loads from hub I need to do in the process to save progress.

It works... but it's a pain to have to redo this over and over for every merge.

and just center on pointing your personal opinion to tell us that we're convoluted, a mess, have bad documentation and that automatic1111 is better.

Besides the bloat, AUTOMATIC1111 is objectively better than diffusers for doing anything (or almost anything) with Stable Diffusion. This is why it became the norm, and so popular that Google that effectively banned it on free Google Collab tiers, to save on resources.

Since that means AUTOMATIC1111 is no longer an option for me, I'm currentlly trying to use diffusers to replace AUTOMATIC1111.

Do you want to become better than AUTOMATIC1111? Then, maybe you'll want to listen to the feedback of someone who has been using Stable Diffusion ever since it came out back in summer 2022, and who released his first custom model ever since January 2023.

If you want to capture the target audience of AUTOMATIC1111, then give people the features they're looking for in AUTOMATIC1111 that diffusers either doesn't provide or only poorly.

Ignore user feedback, and you'll make yourselves obsolete in the foreseeable future.

I must repeat, free tier collab has 16GB of VRAM, you should be able to merge models with it

I am... with the code from my original post.

Using CheckpointMergerPipeline? Maybe that's just good enough for SD 1.x models, but I care only about SDXL models. And, even when it's the first thing I do, my (V)RAM just can't handle that.

I have time I can do a quick demo of it but for now I can just give you tips like I did in on my first answer.

If you think you can make a CheckpointMergerPipeline work with SDXL models on a free tier of Google Collab, feel free to give it at try.

I'd love to learn what I may be overlooking.

I could answer all the other points but I don't think you would care about that because you already made your mind about them.

I care enough that I decided to convert my workflow on Google Collab into demo with a maintainable modular codebase.

I'll post a link to the demo when I'm finished.

If you get a better idea of my workflow, maybe you'll understand aforementioned issues.

[asomoza]

Thanks for your reply, It really matter to me when someone comes more sincerely and ask for help/guidance.

I also come from a full stack background and I can relate on what you're going through. I'll answer the help part here and do another one for the stuff that it's more related to diffusers in general.

Also, you come off as quite elitist IMO. "Just get more budget" or "just get better hardware" isn't an option for everyone.

I'm really sorry if I sound like that, it's not my intention, but the models comes with some restrictions which we cannot fix for the users. SDXL its a 6B param model and it wont directly fit inside a GPU with less than 6-7GB of free VRAM (in fp16), and to run it in one go without cpu offload or sequential offload, you will need a 12-13GB of free VRAM GPU, this is a fact and not something like "you must buy expensive GPU to run it" but more of a "hardware requirements" for a model, pretty much like in games. This is even higher for some of the latest model, for example Kolors will run only on a 3090 or a 4090.

This for merging means, that depending on where you're doing the merging (CPU or GPU) you will need double the amount of VRAM or RAM since you will need to load both models at the same time.

Saying all from before, there's some techniques you can do to overcome this limit, for example, SDXL that people commonly think it's just one model in fact its 4, so you can merge each one separately which lowers significantly the memory requirement, most of the time you won't need to even merge the text encoders and certainly you don't need to merge the VAE which is the same one for every model.

The code you're using works for you because it's very simple, it just goes through one checkpoint and compares it to the other one, if they match, it then merges them with an "alpha", after that it does a second pass and just copies the part of the second model that isn't part of the first one, so in that code the order matters and also it's very unreliable because it doesn't really load the models.

To be able to do what you want, you'll need to learn more about tensors, torch and more of a low level understanding on what's going under the hood, it's not something that can be explained in just a simple docstring or "usage" documentation. I can help you achieve what you want but I need a couple of answers first:

• From what I understand your final objective is to merge just loras right? or you will still need to merge whole checkpoints?
• If it is loras, can you link me to two of them you want merged?
• if it's checkpoints, can you link me to two of them you want merged? Also is there a reliable way to test if two checkpoints are being merged correctly? For me it seems kind of a lottery and also I don't really use them, I have my filters in the model repos as "show only trained" because for my needs I like to only work with the base models and trained fine tunes.

Also there's a lot of "types" of LoRAs and there's always some new/obscure ones that won't load with diffusers, I still don't understand why people do that because for me the bests ones are just the normal ones.

[asomoza]

About the hugging face and diffusers comments, I can answer a little of it since I joined recently but this still will by my personal opinion and doesn't mean the main view of the team or hugging face.

Do you want to become better than AUTOMATIC1111? Then, maybe you'll want to listen to the feedback of someone who has been using Stable Diffusion ever since it came out back in summer 2022, and who released his first custom model ever since January 2023.

Actually there's a misunderstanding here, we don't want to be better than any of the webuis, we are a library, not a complete solution, we build what should be under the hood and not a front end. So this comparison is not fair for them, we don't except a normal user that doesn't know how to code to be able to use diffusers, this in a full stack environment is like expecting a wordpress user to be able to make a react app from scratch or to code the php backend of a website.

Also we don't compete with other open source projects, we want the whole open source scene to be better and to collaborate with each other, that's why you don't see us posting the "we're better than you" typical posts and we only post about other projects when we have good vibes or praise for them.

The more time I spend trying to get literally anything done with diffusers, the more I understand the dude who wrote this rant on Reddit

This comes as a shock for me, didn't know that post. I really don't understand that point of view. When I was learning about AI and ML in general a couple of years ago, there wasn't any kind of good documentation unless it came from hugging face.

Transformers and diffusers are very educational, all the code is well documented (for developers), have notebooks and even workshops you can follow. We have extensive tests too which are pretty much non existent in other projects and we actually answer to the questions and issues. I would really like to know to which other projects we're being compared here.

For example, you did bring this part of the code:

unet_lora_layers (Dict[str, torch.nn.Module] or Dict[str, torch.Tensor]) — State dict of the LoRA layers corresponding to the unet.
text_encoder_lora_layers (Dict[str, torch.nn.Module] or Dict[str, torch.Tensor]) — State dict of the LoRA layers corresponding to the text_encoder. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers.

unet_lora_layers and text_encoder_lora_layers are really well explained on what they are, so really don't understand when you say:

The documentation is very unclear regarding the expected values

So I think what you don't understand is the meaning of state dict, layers, Tensor or nn.Module and on how to get those, which is something expected to be in the knowledge of someone that's looking at that part of the code. If you want to understand that, you will need to first start on a more basic level which is not in the scope of diffusers but more in machine learning and pytorch in general.

[asomoza]

I had a little free time, so I did a PoC of how this should be done. Here is the link to the colab.

It should work for any pair of unets that have the same config. This is just for the unets, if you need the text encoders and can't do it yourself, tell me and I'll do it when I have time.

I did a mix to divide the memory consumption between the CPU and the GPU:

This is slower than just using the GPU, but it's decent IMO:

The saving actually takes more time, I don't really know why is that slow in the google colab, in my PC the saving its done in a couple of seconds.

[jslegers]

About the hugging face and diffusers comments, I can answer a little of it since I joined recently but this still will by my personal opinion and doesn't mean the main view of the team or hugging face.

Fair enough

Actually there's a misunderstanding here, we don't want to be better than any of the webuis, we are a library, not a complete solution, we build what should be under the hood and not a front end.

Well, duuuh...

Do you really think I don't know the difference between a web UI and a utility library?

Obviously I was refering only to the things both projects share, comparing performance, API consistency, API completeness, stability, ease of use, documentation completeness, organization of code, etc.

So this comparison is not fair for them, we don't except a normal user that doesn't know how to code to be able to use diffusers

I have more than a decade of professional experience as a software developer, having coded in mostly ABAP (SAP), PHP, frontend JavaScript & NodeJS.

Between 2017 & 2019, I worked for a company called Luciad (now Hexagon), where I worked as a senior software developer, doing maintainance and development for a high end GIS library called LuciadRIA :

I also authored my own CSS framework back in 2013, which had better browser support than Bootstrap (the leading competitor), and which was fully integrated with jQuery UI :

Additionally, in the 11 years since I became a member of StackOverflow, I managed to stack up a reputation of +46K and almost 400 badges over there :

Python is pretty new to me, though, but with a background in both JavaScript & PHP, it's pretty easy to get a hang of the language.

Having actually done maintenance and development on a high end GIS library for a living, I know the challenges associated with eg. code architecture & documentation associated with high end utility libraries. And I also know how to compare.

And, in my experience, both the code & documentation of diffusers are a mess.

Also we don't compete with other open source projects, we want the whole open source scene to be better and to collaborate with each other

Good.

Then let's collaborate and start listening to what I and others in the community are saying, instead of talking down to me like I'm a n00b coder who barely knows how to write a Hello World program.

This comes as a shock for me, didn't know that post. I really don't understand that point of view.

You sound like someone in his early 20s with mostly academic experience & little to no corporate experience.

Either way, I think you need to come out of your ML silo more often and communicate with developers who have senior level experience maintaining large libraries for high end customers.

The more you do, the better you'll understand the Reddit rant... the better you'll understand me... and the more capable you will be of helping diffusers grow in the right direction.

That is, at least, if you stop acting all condescending towards developers with more experience than yourself and you first acknowledge that your actual knowledge of maintaining large codebases is pretty limited.

When I was learning about AI and ML in general a couple of years ago, there wasn't any kind of good documentation unless it came from hugging face.

OK, fair. I have little to no knowledge of ML. We clearly have different backgrounds.

But, as I hope I have aptly demonstrated, I have more then enough experience as a software developer to know what documentation & code should look like for a utility library like diffusers.

At a first glance, I very much liked diffusers. Stuff like loading a model from the hub, switching scheduler & loading a LORA are all fairly straightforward and require very little to know knowledge of the actual file formats or maths involved.

But then you try to do some other stuff, that should be just as straightforward and just as easy to use... and you end up with a headache.

all the code is well documented (for developers)

As a developer, I disagree.

We have extensive tests too which are pretty much non existent in other projects

In academic projects maybe. In many corporate projects, you can't commit any code to production unless you have created new unit tests or modified existing ones.

Also, your tests are clearly insufficient. #8974 would have immediately been detected with proper unit testing. And, had you properly integrated this conversion into the core library, this bug would never have occured in the first place.

unet_lora_layers (Dict[str, torch.nn.Module] or Dict[str, torch.Tensor]) — State dict of the LoRA layers corresponding to the unet.
text_encoder_lora_layers (Dict[str, torch.nn.Module] or Dict[str, torch.Tensor]) — State dict of the LoRA layers corresponding to the text_encoder. Must explicitly pass the text encoder LoRA state dict because it comes from 🤗 Transformers.

unet_lora_layers and text_encoder_lora_layers are really well explained

You call that a good explanation?

Where can I find the LoRA layers corresponding to the unet? I can't find any documentation on this anywhere.

Where can I find the LoRA layers corresponding to the text_encoder? I can't find any documentation on this anywhere.

Also, where can I find the state dict of these LoRA layers?

A simple demo would have probably answered all of these questions.

Code examples are often the best documentation.

So I think what you don't understand is the meaning of state dict, layers, Tensor or nn.Module and on how to get those, which is something expected to be in the knowledge of someone that's looking at that part of the code.

Nonsense.

The whole point of using a library like diffusers, is to make this stuff simple enough for coders with zero experience with ML.

Your documentation should either...

1. Encapsulate the state dict, layers, Tensor or nn.Module for basic LORA usage, so people only interested in basic LORA usage should not need to familiarize themselves with these terms
2. Use proper API docs, to tell a dev where to find the state dict, layers, Tensor or nn.Module
3. Use demo code to tell a dev where to find the state dict, layers, Tensor or nn.Module

diffusers does neither. And that's one of the main issues with this library.

If you can't properly encapsulate or document this stuff, then what's even the point of using diffusers?

If you want to understand that, you will need to first start on a more basic level which is not in the scope of diffusers but more in machine learning and pytorch in general.

At that point, there's literally no reason to use diffusers at all.

In fact, I just looked at the code of txt2img demo of CompVis/stable-diffusion, which was implemented directly in torch, and I was shocked by how simple this demo was.

Using diffusers should be simpler than that, not harder.

[asomoza]

We posted at almost the same time, I was going to start helping you, but you still seem to be very aggressive.

I'm really tired of a discussion that seems to go nowhere, you seem to know me better than myself, so no point on trying to explain more and also you seem to know everything and also better than me.

If you think that using the original CompVis code it's simple and easier, by all means, please use that to merge models. If you think that the documentation of the webuis are better and that they answer better the issues, I don't really know why are you trying to give me a lecture here.

By the way, I'm probably older than you, but if in any way I was giving a "condescending tone" or "insulted you" I'm sorry and that was never my intention. I was trying to help you and answer you in a genuine way but I seem to have made a mistake somewhere.

I'll take note on the documentation, testing and maintaining large code bases.

In case you missed my post, it's here

Take care and I really wish you can solve your problem.

[jslegers]

It should work for any pair of unets that have the same config.

Now that's really cool. Very much appreciated!!

Is there any way at all to merge unets that don't have the same config?

This is just for the unets, if you need the text encoders and can't do it yourself, tell me and I'll do it when I have time.

Maybe I could do it myself if you tell me what I need to do differently.

Either way, once I have a working version for the text encoders, I'll probably write my own equivalent to the CheckpointMergerPipeline based this logic. I could push it to this repo as a community pipeline if you wish.

I did a mix to divide the memory consumption between the CPU and the GPU

That would be a really cool feature to have in the actual library!

The saving actually takes more time, I don't really know why is that slow in the google colab, in my PC the saving its done in a couple of seconds.

Saving to GDrive is pretty suboptimal in my experience.

Not sure about saving to runtime.

I was going to start helping you

You already did help me with the Collab example you provided. And I'm very grateful for that.

but you still seem to be very aggressive.

Sorry, man...

As I said, I'm just frustrated.

And I can't stand it when i'm being talked down to or my concerns are ignored.

Maybe it's just too long since I've been active on StackOverflow,

If you think that using the original CompVis code it's simple and easier, by all means, please use that to merge models.

I was actually seriously considering reverse engineering AUTOMATIC1111 and ComfyUI, with the amim of extracting the parts I need & building my own library on top of it.

But, now, maybe I don't need to.

By the way, I'm probably older than you

Maybe.

I'm 42.

if in any way I was giving a "condescending tone" or "insulted you" I'm sorry and that was never my intention. I was trying to help you and answer you in a genuine way but I seem to have made a mistake somewhere.

Apology accepted.

On my end, sorry if I came off as aggressive.

I'll take note on the documentation, testing and maintaining large code bases.

Thank you.

With respect to that, could you please tell me where I could find the state dict of the LoRA layers corresponding to respectively the unet and text_encoder?

With that info, I should be able to save my merged LORAs directly instead of blending them into a base model... which I generally prefer.

Same as with the checkpoint merger, I could push my result to https://huggingface.co/docs/diffusers/en/api/loaders/lora as an example, although I'm not sure if this is possible on my end and where I could do this.

[jslegers]

OK, I think I got it...

I just copy-pasted your code & did the same for both text encoders + the vae.

Then I copied the rest of the data from one of the models...

Then I loaded the data into a pipeline.

And boom...

At first sight everything seems to work fine.

I'm going to tweak this a bit further & post my code after that.

Either way, thank you so much for the headstart.

You saved me a hell of a lot of time with this!

[jslegers]

@asomoza

OK, so, I converted your code for merging the unets into a composable pipeline that allows for easy loading and unloading of different models.

The pipeline can be used as follows :

composable = ComposableStableDiffusionXLPipeline

pipe = composable \
  .load_model() \
  .compose(
    name = "my_merge",
    unet = composable.merge("SG161222/RealVisXL_V4.0", "RunDiffusion/Juggernaut-XL-v9", model = "unet", alpha = 0.4),
    text_encoder = composable.merge("stabilityai/stable-diffusion-xl-base-1.0", "SG161222/RealVisXL_V4.0", model = "text_encoder", alpha = 0.4),
    text_encoder_2 = composable.merge("stabilityai/stable-diffusion-xl-base-1.0", "RunDiffusion/Juggernaut-XL-v9", model = "text_encoder_2", alpha = 0.2),
    vae = composable.merge("SG161222/RealVisXL_V4.0", "RunDiffusion/Juggernaut-XL-v9", model = "vae", alpha = 0.5)
  ) \
  .from_loaded() \
  .to(dev)

I build a small demo on Google Collab, which can be found here :
https://colab.research.google.com/drive/14xjerkyfcm059VhMhScJ1nYqR5zCeIKh

I could only achieve this with the help of the merge code your provided, so thank you, again, for that.

[asomoza]

Glad to read and that your issue was resolved.

FYI, for SDXL there isn't custom vaes like in SD 1.5, there are just two of them, the original which needs to be upcasted and this one. I recommend to just skip vae merging and just add the fixed one which requires less VRAM.

Also for the text encoders, there's a lot of models that don't train them (slower training or not enough GPU power) so you can probably check if they're different or not before the merging.