Model Merging Method, anyone seen or tried?

[ClashSAN]

#1462

https://github.com/samuela/git-re-basin

Came across, wanted to know what it means.

[affableroots]

This means merging 2 (or 1000) models without losing details of either. It means you can finetune/dreambooth multiple characters, multiple objects, multiple concepts, styles etc. It means distributed training and sharing.

[ogkalu2]

@affableroots Do you know how to use the code to achieve that though ? I know that's not pytorch but if i had converted TF models, would it be possible to merge then with re-basin ?

[ClashSAN]

update!
https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion
needs pytorch 1.11.0 or lower

python SD_rebasin_merge.py --model_a nameofmodela.ckpt --model_b nameofmodelb.ckpt

[snsd21116890]

update! https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion needs pytorch 1.11.0 or lower

python SD_rebasin_merge.py --model_a nameofmodela.ckpt --model_b nameofmodelb.ckpt

I had run it sucsessful.But after 10 hours it still didn't finished.

[ogkalu2]

update! https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion needs pytorch 1.11.0 or lower
python SD_rebasin_merge.py --model_a nameofmodela.ckpt --model_b nameofmodelb.ckpt

I had run it sucsessful.But after 10 hours it still didn't finished.

Shouldn't take so long with the new update now (if you have a gpu and enough ram, it will take minutes) but there's still a problem. All merged models are currently biased to model b.

[MV799]

But did actually anybody tried it?

[ogkalu2]

@snsd21116890
Are you still having this problem ? You shouldn't be anymore

[runner22k]

@snsd21116890 Are you still having this problem ? You shouldn't be anymore

@ogkalu2 can you please give us clear steps to use this script correctly without loss and bias under windows and linux? I want to merge two of my favorite ckpts into one. OR can i follow @ChashSAN's steps?

[runner22k]

You can try it now. Most people probably have windows and python, so then:

1. create a venv: `python -m venv rebasin_venv`

2. activate your "rebasin_venv" in cmd, type: `rebasin_venv\Scripts\activate.bat`

3. install these packages (enter this): `pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 scipy pytorch_lightning`

4. Run the script like this as an example: `python SD_rebasin_merge.py --model_a Anything-V3.0-pruned-fp16.ckpt --model_b Clonewars.ckpt --alpha 0.25`

you can mix 2gb(fp16) and 4gb(fp32) When you're done, you can just delete all the packages because everything installed is in the rebasin_venv folder

I am under windows and I am getting below error. I dont have any prior knowledge, I am following your steps. Please help

PS D:\ckptMerger_without_distortion> python -m venv rebasin_venv
PS D:\ckptMerger_without_distortion> rebasin_venv\scripts\activate.bat
PS D:\ckptMerger_without_distortion> pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 scipy pytorch_lightning
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu113
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/torch/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/torch/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/torch/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/torch/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/torch/
Could not fetch URL https://pypi.org/simple/torch/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/torch/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /whl/cu113/torch/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /whl/cu113/torch/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /whl/cu113/torch/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /whl/cu113/torch/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /whl/cu113/torch/
Could not fetch URL https://download.pytorch.org/whl/cu113/torch/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='download.pytorch.org', port=443): Max retries exceeded with url: /whl/cu113/torch/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: none)
ERROR: No matching distribution found for torch==1.11.0+cu113
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Could not fetch URL https://download.pytorch.org/whl/cu113/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='download.pytorch.org', port=443): Max retries exceeded with url: /whl/cu113/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
WARNING: There was an error checking the latest version of pip.

[runner22k]

I found a solution for the above installation problem. Using anacond prompt will allow me to install without above error.

For anyone who is new like me, above command needs 2.2+ GB packages to be install. So, plan accordingly.

[runner22k]

@ogkalu2 I am getting below error while trying to merge. and "--device gpu" is not working.

(rebasin_venv2) (base) d:\ckptMerger_without_distortion>python SD_rebasin_merge.py --model_a model1.ckpt --model_b model2.ckpt --alpha 0.9 --output merged_model
Using half precision

    ---------------------
         ITERATION 1
    ---------------------

new alpha = 0.09

Traceback (most recent call last):
  File "d:\ckptMerger_without_distortion\SD_rebasin_merge.py", line 55, in <module>
    theta_0 = {key: (1 - (new_alpha)) * theta_0[key] + (new_alpha) * value for key, value in theta_1.items() if "model" in key and key in theta_1}
  File "d:\ckptMerger_without_distortion\SD_rebasin_merge.py", line 55, in <dictcomp>
    theta_0 = {key: (1 - (new_alpha)) * theta_0[key] + (new_alpha) * value for key, value in theta_1.items() if "model" in key and key in theta_1}
KeyError: 'model_ema.decay'

(rebasin_venv2) (base) d:\ckptMerger_without_distortion>

[Babak2127]

@AUTOMATIC1111
That's so interesting! is there any plan to implement this method to webui's checkpoint merger?

[diStyApps]

I made basic ui for it if you want to try.

https://github.com/diStyApps/Merge-Stable-Diffusion-models-without-distortion-gui

[runner22k]

Merged two ckpt files using Git-re-Basin Script:

NOTE: MASKED THE FACE INTENTIONALLY. THOSE ARE PERSONAL PHOTOS.

Used anaconda prompt as I was facing some SSH problem. Below is the command I used.
python SD_rebasin_merge.py --model_a dreamlike.ckpt --model_b model.ckpt --alpha 0.4 --output sk_dl0.4

This was merged using Automatic1111's default merging Method "Weighted Sum"

In the images on the top are ckpt file names with the alpha/Multiplier values I selected.
on the left, I have use ((attention)) to dig out the person's features as far as possible. same with dreamlike, we can use ((dreamlikeart)) to exaggerate dreamlike style.

NOTE: MASKED THE FACE INTENTIONALLY. THOSE ARE PERSONAL PHOTOS.

Right now Git-re-basin doesn't works with all ckpt files. Maybe because there is no one standard for all ckpt files. I don't know. This Method didn't gave the result I expected. I thought it will be able to completely merge, both ckpt files, without loosing details of the person. If we observe the above images, I chose a custom trained person's model ckpt file, and the second one is dreamlike ckpt. I chose them because I can use "sks" and check if the person data is preserved while merging or not. This Method of merging is similar to Weighted Sum, we have to choose the percentage of lose, which in a way we can call how much of the B file to merge with the A file. It doesn't mean that we are atleast having complete A. We will be sacrificing some of it too.

Using git-re-basin Method is giving just a little more preserving of the person details, based on the above example. But I could be wrong, or git-re-basin could be more capable if GPU is allowed, OR maybe code or method could be improved little more to make it perform well. I don't know because i am not a programmer. I can only test and see if works/helps me with my requirement.

SO, in conclusion... Git-re-basin is little better than usual Weighted Sum. and it works in reverse. Usually B is added to A, based on my limited understanding. But git-re-basin adds A to B. We have a multiplier M, but this has "alpha" and also "iterations"

Use below file for the settings and prompt I used