[Experiment] Transfer Control to Other SD1.X Models

[lllyasviel]

News

This post is out-of-date and obsolete. Please directly use Mikubill' A1111 Webui Plugin to control any SD 1.X models. No transfer is needed. Results are a bit better than the ones in this post.

Previous Method (Obsolete)

This is guideline to transfer the ControlNet to any other community model in a relatively “correct” way.

This post is prepared for SD experts. You need to have some understandings about the neural network architecture of Stable Diffusion to perform this experiment.

Let us say we want to use OpenPose to control Anything V3, then the overall method is

AnythingV3_control_openpose = AnythingV3 + SD15_control_openpose – SD15

More specifically,

# Inside Control Net
Any3_control.control_model.weights 
= SD15_control.control_model.weights + Any3.model.diffusion_model.weights - SD15.model.diffusion_model.weights

# Inside Base Model (less important, but better to have)
Any3_control.model.diffusion_model.weights 
= SD15_control.model.diffusion_model.weights + Any3.model.diffusion_model.weights - SD15.model.diffusion_model.weights

You can download necessary files from

AnythingV3: https://huggingface.co/Linaqruf/anything-v3.0
SD1.5: https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main
ControlNet: https://huggingface.co/lllyasviel/ControlNet/tree/main/models

Important things to keep in mind:

1. Replacing the base model in control net MAY work but is WRONG. This is because control net may be trained with some SD layers unlocked. See the ending part of “SD_locked” in the official training guideline. You need to compute the offset even in the base diffusion model. (Obsolete) Some experiments show that results are equally good without such offsets. Please directly use Mikubill' A1111 Webui Plugin.
2. The difference of CLIP text encoder must be considered. In many anime models, because of that well-known reason, a dominant majority of models need “clip_skip=2” and a 3x longer Token length. Note that this is also influencing the SoftMax averaging because the length is different.
3. In some applications like human pose, your input image should not be anime images. It should be real person photos because that image is only read by the OpenPose human pose detector. That image will not be visible for SD/ControlNet. Also, OpenPose is bad at processing anime images.

I have done all these preparations for you.

You may open the "tool_transfer_control.py" and then edit some file paths

path_sd15 = './models/v1-5-pruned.ckpt'
path_sd15_with_control = './models/control_sd15_openpose.pth'
path_input = './models/anything-v3-full.safetensors'
path_output = './models/control_any3_openpose.pth'

You can define the output filename with "path_output". You need to make sure that all other 3 filenames are correct and exist. Then

python tool_transfer_control.py

Then you will get the file

 models/control_any3_openpose.pth

Then, you need to hack the gradio codes to read your new models, and hack the CLIP encoder with "clip_skip=2" and 3x token length.

Taking openpose as an example, you can hack "gradio_pose2image.py" in this way

from share import *
from cldm.hack import hack_everything


hack_everything(clip_skip=2)


import config
import cv2
import einops
import gradio as gr
import numpy as np
import torch

from pytorch_lightning import seed_everything
from annotator.util import resize_image, HWC3
from annotator.openpose import OpenposeDetector
from cldm.model import create_model, load_state_dict
from ldm.models.diffusion.ddim import DDIMSampler


apply_openpose = OpenposeDetector()

model = create_model('./models/cldm_v15.yaml').cpu()
model.load_state_dict(load_state_dict('./models/control_any3_openpose.pth', location='cpu'))
model = model.cuda()
ddim_sampler = DDIMSampler(model)


def process ...

Then, results will be like:

("1girl")

("1girl, masterpiece, garden")

And other controls like Canny edge:

("1girl, garden, flowers, sunshine, masterpiece, best quality, ultra-detailed, illustration, disheveled hair")

[Mikubill]

Great! How about make it into an automatic1111 webui plugin? Something like applying the control net to other fine-tuned models without conversion etc..

[HaFred]

It is done already. See this in Chinese tho.

[hansolocambo]

Available for a few days already. It's a HUGE game changer. Truly amazing.
https://github.com/Mikubill/sd-webui-controlnet
how to, just in case : https://www.youtube.com/watch?v=YephV6ptxeQ

[ninjasaid2k]

Available for a few days already. It's a HUGE game changer. Truly amazing. https://github.com/Mikubill/sd-webui-controlnet how to, just in case : https://www.youtube.com/watch?v=YephV6ptxeQ

you realize you gave the creator of the repo their own repo.

[Jaxkr]

Amazing job on this resource, coolest thing I’ve ever seen. I think this secures the stable diffusion 1.5 dominance forever. My use case is posing characters using depth2img and this blows it out of the water.

EDIT: Works great with almost every SD 1.5 checkpoint I try!

[SlimeVRX]

Hi everybody!

Thanks for the great work of the author!

Can someone share with me the model after the merge "control_any3.pth"?
I run on Colab, the computer does not have enough VRAM to merge model. Need at least 21.11 Gb RAM to load 3 model.
I need "Canny edge" model.

Thank you very much!

[SlimeVRX]

Hi! Thank you very much!

[entmike]

Here! https://drive.google.com/file/d/1zMiDvWZLnWbcRPE9bEDePagn8NmdIfaA/view?usp=sharing

Seems to be gone, now.

[toyxyz]

Here! https://drive.google.com/file/d/1zMiDvWZLnWbcRPE9bEDePagn8NmdIfaA/view?usp=sharing

Seems to be gone, now.

The merged model is here.

https://huggingface.co/toyxyz/Control_any3/tree/main

You no longer need a merged model. With this extension, you can use it with any model you use.
https://github.com/Mikubill/sd-webui-controlnet

[Trevor-Z]

You mean, we don't need models like control_any3_openpose.pth anymore?

How does that work, I didn't find anything relating to that in the repo.

[Trevor-Z]

Oh, I guess this is it?

Mikubill/sd-webui-controlnet#35

What a preposterous amount of progress in such a short time.

[toyxyz]

Works very well!

[OlegBatrakov]

Hey! Thanks a lot for your work, it's awesome! Thinking how we can integrate it in our gamedev studio process!

[toyxyz]

I uploaded models that merged Anything 3.0 and ControlNet to huggingspace. Those who cannot merge due to lack of VRAM, feel free to use it!
https://huggingface.co/toyxyz/Control_any3

[SlimeVRX]

Thank you many!

[throttlekitty]

Is this a merge of all the ControlNet models + Anything 3.0, or just one of them?

[toyxyz]

Is this a merge of all the ControlNet models + Anything 3.0, or just one of them?

[toyxyz]

Canny edge works very well too! I wonder when ControlNet with Anime Line Drawing will be released.

[cryptowooser]

Can this be transferred to a 2.x SD model (like the newly release WD1.5) as well, or is that outside its present capabilities?

[sALTaccount]

The technique should be able to be applied, but the pretrained ControlNet models are SD 1.5 based so not gonna work with SD 2.1 based models like WD1.5. We probably have to train WD 1.5 for like a month or two before it gets really good, so I'll look into making controlnet models for SD 2.x models in the mean time. Hopefully codebase doesn't have to be changed too much (or at all if they natively support) Very cool technique.

[GreenLandisaLie]

Can someone make a Low RAM (16Gb) version of the tool_transfer_control.py script? Even if it involves unloading/reloading models plus saving/deleting temporary models multiple times?

[gitmithy]

16GB memory is required, 12GB is not available?

[GreenLandisaLie]

I've made a script that works on 16G PCs.
It runs successfully and makes a model with the exact same size as the working anythingv3-pose model provided in this discussion.
However - for some reason - when doing inference all it returns is noise. There must be a problem somewhere in the script but I'm no programmer and not familiar with python so I'm having trouble figuring out.
Can someone take a look? Link: https://pastebin.com/H03JPgPb

[lorestateman]

Has anyone tried using ControlNet to control Dreambooth models?

[2blackbar]

yes i do with this extension and it works amazing , can do much higher than 512x512 i think controlned was trained on up to 1024x1024 , set weight to 0.5 https://github.com/Mikubill/sd-webui-controlnet

[FurkanGozukara]

With this extension there is no need for merging.
https://github.com/Mikubill/sd-webui-controlnet

I tried but the output quite dark and the background is simple gradient color, is the setting or prompts I get wrong?

yes it works awesome. that Mikubill is genius seriously

i made a tutorial for how to use : https://youtu.be/vhqqmkTBMlU

[a-toms]

Clear tutorial here for those who prefer pure code: https://github.com/haofanwang/ControlNet-for-Diffusers
I'll be using this for my product https://amazing.photos

[paulo-coronado]

Amazing tutorial @a-toms! The only "issue" I am facing is that I compared the images generated by the model created (e.g. control_any3_openpose.pth) and by ControlNet extension (by Mikubill), using same seed and configs, and the results are a bit different... I don't know why... any thoughts?

[paulo-coronado]

Just try for yourself. The first image I got using SD+ControlNet (merged), and the second (WebUI by Mikubill).

[2blackbar]

Hey man can you modify the code so it extracts just controlnet weights and leaves off SD.15 thats built in in weights from huggingface ?
I want to try if it works with extension cause it kinda ignores SD1.5 and uses own models which means it requires just controlnet weights on its own and this would save sapce and speedup loading times , at this moment we are kinda loading SD1.5 twice .
I use this extension https://github.com/Mikubill/sd-webui-controlnet

[mth233]

Could we just save the "SD15_control_openpose – SD15" and add the customized model when we actually use it? Could this way skip the transfer model step?

[ghost]

@toyxyz Could you also try the chilloutmix_Ni model?

https://civitai.com/api/download/models/10332

[HaFred]

@AO2233 Hi buddy, would you mind showing how to transfer on chilloutmix_Ni? I tried to replace path_sd15 = './models/v1-5-pruned.ckpt' but it seemed to be not correct.

[ghost]

@HaFred
To reproduce the picture

1. It is recommended to use stable-diffusion-webui and sd-webui-controlnet to do it.
2. Use "chilloutmix.safetensors" of ChilloutMix.
   Attention: 'lora:KoreanDollLikeness_v10:0.66' here refers to the additional Lora model (Low-Rank Adaptation of Large Language Models). And vae is here.

Check it out here for more information: korean-doll-likeness.
In fact, you can see that the image I show here is his model example image.

I think this example is just to help illustrate the generalizability of "ControlNet". What kind of effect or style you want, just look for the corresponding SD fine-tured model or train it by yourself. For example, without large graphics memory, if I want to generate some pictures of an anime character, just collect some image, fine-tuning AnythingV3 by way of Lora, finally use ControlNet to control the Image generation results. No need to insist on reproducing this one picture.

As the author says: a relatively “correct” way:
AnythingV3_control_openpose = AnythingV3 + SD15_control_openpose – SD15
So we can also have AnySD1.x_model_control_{control_type} = AnySD1.x_model + SD15_control_{control_type} – SD15.
Personal view: If SD1.X do not have change in main structure in the future, this ControlNet weights may be possible to used for any SD1.X model.

[HaFred]

I just wanna remove the facial distortion, but it seems like using controlnet gradio could not work at the moment. Perhaps using the whole WebUI-stable-diffuision pipeline with controlnet plugin will do the work.

[Huntersxsx]

When I run the 'tool_transfer_control.py', I meet the error: 'Segmentation fault (core dumped)'. I have no idea about it, can you help me or offer the weights of merging ControlNet and chilloutmix? Thanks

[blx0102]

'Segmentation fault (core dumped)' as well when using tool_transfer_control.py, any idea?

[Nacurutu]

Hi, May I ask whats the difference between:

control_sd15_canny.pth
5.71 GB

control_canny-fp16.safetensors
723 MB

and

diff_control_sd15_canny_fp16.safetensors
723 MB

[Sansui233]

The first one has an sd v1.5 model in it.
The second one is the pure controlNet.
The third one implements a different algothrithm discussed here.

[steelywing]

I still don't know the different between control_canny-fp16.safetensors and diff_control_sd15_canny_fp16.safetensors, both of they are

SD15_control.control_model.weights - SD15.model.diffusion_model.weights

isn't it?

[offchan42]

control_canny-f16 is supposed to be SD15_control.control_model.weights; no subtraction

[xiaoyu220302]

直接起飞

[liuquande]

Dear author, very amazing work!!!

Though I can intuitively understand that the ControlNet trained together with SD can thansfer to other SD-like architecture model, but is there any official explaination for that, or any relevant literature that I need to read to figure it out.

Besides, seems the control net only contain the encoder and a middle block of the Unet.
So for this transferring function:

# Inside Control Net
Any3_control.control_model.weights 
= SD15_control.control_model.weights + Any3.model.diffusion_model.weights - SD15.model.diffusion_model.weights

If Any3.model.diffusion_model.weights, SD15.model.diffusion_model.weights only denotes those weights contained in the control net, instead of all the model weights? Otherwise, the weight number will not be the same.

Many thanks.

Best.

[williamkmlau]

Hi Lvmin, got some question regarding the weights add/subtracts as I'm not an expert in data science field.

I don't know if model weights are commutative in the sense that is:

custom_model_control.weights = SD15_control.control_model.weights + custom_model.weights - SD15.model.diffusion_model.weights

the same as:

custom_model_control.weights = SD15_control.control_model.weights - SD15.model.diffusion_model.weights + custom_model.weights

Currently in the community, there is an "extracted" version of the SD1.5 ControlNet weights, which I'm presuming is the diff of

SD15_control.control_model.weights - SD15.model.diffusion_model.weights

My understanding is that it should be commutative or else there was no point to extract this in the first place. With the diff weights I assume we can just do:

diff.weights + custom_model.weights

But I'm also reading feedbacks from the community that the custom merged ControlNet models seems to be different from using the diff models and that the custom merged models seem to be better. Would this be the case?

[paulo-coronado]

Could you please explain the following step:

"...hack the gradio codes to read your new models, and hack the CLIP encoder with "clip_skip=2" and 3x token length."

I did both tests (hacking and not hacking the gradio codes), both ways actually work (and generate good quality images), but they generate different results (using same prompt, seed, sampler etc.)

[illtellyoulater]

So guys I know this is a highly technical thread and for what I can I am trying to following through... I've just installed ControlNet in WebUI and could verify the basics seem to be working ok with SD base model and a few other models I quickly went through...
But then I read the above topic about Any3 and I started getting a bit confused...
If you are merging ControlNet with other models, does that imply that by default ControlNet can only achieve good results with SD base model?

[korzen]

Hi, do I need to transfer ControlNet to custom models based on SD 1.5 (e.g. Dreamshaper, Deliberate, Anything) when using Diffusers? Or is it already taken care of by Diffusers. Thanks!

[geroldmeisinger]

May be a stupid question but now that:

This post is out-of-date and obsolete. Please directly use Mikubill' A1111 Webui Plugin to control any SD 1.X models. No transfer is needed.

I wonder how and why from a technical point of view this is possible in the first place?
TODO search for corresponding issue in sd-webui-controlnet

ControlNet paper v2: "Transferring to community models. Since ControlNets do not change the network topology of pretrained SD models, it can be directly applied to various models in the stable diffusion community, such as Comic Diffusion [60] and Pro- togen 3.4 [16], in Figure 12."