Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An error during merging #95

Closed
miasik opened this issue Dec 26, 2023 · 17 comments
Closed

An error during merging #95

miasik opened this issue Dec 26, 2023 · 17 comments
Assignees
Labels
bug Something isn't working

Comments

@miasik
Copy link

miasik commented Dec 26, 2023

Tried different models with the same result.

debugs =  ['elemental merge']
use_extra_elements =  True
 - mm_max_models =  4
config hash =  f79746d04db933c7fd3d9ae6cb70b3bd562a09ed41e48bc894b4fec58d5259e1
  - mm_use [True, False, False, False]
  - model_a Photo\nextphoto_v30.safetensors [1c1f913f3b]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\rundiffusionFX_v10.safetensors [ad1a10552b]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.75]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Photo_nextphoto_v30
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\nextphoto_v30.safetensors...
isxl = False
compact_mode =  False
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_rundiffusionFX_v10...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\rundiffusionFX_v10.safetensors...
mode = Add-Diff, alpha = 0.75
Stage #1/2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1132/1132 [00:15<00:00, 74.62it/s]
Check uninitialized #2/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1132/1132 [00:00<00:00, 189153.90it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg337:  24%|██████████Special layer P_bg337 found███████████████▉                                                                                                                                                                       | 125/519 [00:12<00:25, 15.58it/s]
P_bg265: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 519/519 [00:38<00:00, 13.33it/s]
P_bg176: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 519/519 [00:35<00:00, 14.43it/s]
P_bg176: new - old = 0.0
weight order changed layers = ['P_bg371', 'P_bg324', 'P_bg337', 'P_bg358']
Save unchanged weights #2/2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Apply fine tune [0.99, 1.02, 0.99, 1.02, 0.99, [0.0, 0.0, 0.0, 0.0]]
Clip is fine
 - merge processing in 95.3s (prepare: 0.4s, merging: 94.9s).
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3746, in before_process
        sd_models.send_model_to_cpu(sd_models.model_data.sd_model)
      File "E:\SD\automatic1111\modules\sd_models.py", line 576, in send_model_to_cpu
        m.to(devices.cpu)
      File "e:\SD\automatic1111\venv\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 54, in to
        return super().to(*args, **kwargs)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
        return self._apply(convert)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      [Previous line repeated 1 more time]
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
        param_applied = fn(param)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
        return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    NotImplementedError: Cannot copy out of meta tensor; no data!

---
*** Error completing request
*** Arguments: ('task(26bs7tv83v5lryc)', 'A real life analog photo of a provocative adult woman,\nred lips, camisole, low leg jeans shorts, boobs drop, side boobs, groin,\njewelry,\ndetailed skin, skin pores, (freckles:0.6), (moles:0.6), (pigmentation:0.5),\ncatwalking on a street at night,\n[[Gabrielle Union | Laura Vandervoort]:0.1],\nphotography achievement, depth of field, film grain', '(worst quality:1.3), (bad quality:1.2), (low quality:1.1),\n[deformed | disfigured], poorly drawn, [bad : wrong] anatomy, [extra | missing | floating | disconnected] limb, (mutated hands and fingers)', [], 32, 'DPM++ 3M SDE Karras', 1, 1, 7, 768, 512, False, 0.3, 2, '4x_NMKD-Siax_200k', 24, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x0000029F347CFBB0>, 0, False, '', 0.8, 1518790282, False, -1, 0, 0, 0, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, True, 'MEAN', 'AD', 1, False, 1, 1, 0, 0, -1, 0, 0, 0, 0, 0, True, -1, 1, 0, '1,1', 'Horizontal', '', 2, 1, True, 'Photo\\nextphoto_v30.safetensors [1c1f913f3b]', 'v1-5-pruned-emaonly.safetensors [6ce0161689]', 4, '1,1,1,0,0,0,0,0', {'calcmodes': ('Rebasin', 'Normal', 'Normal', 'Normal'), 'save_settings': ['fp16', 'prune', 'safetensors'], 'calc_settings': ['GPU']}, True, False, False, False, 'Photo\\rundiffusionFX_v10.safetensors [ad1a10552b]', 'None', 'None', 'None', 'Add-Diff', 'Sum', 'Sum', 'Sum', 0.75, 0.5, 0.5, 0.5, False, True, True, True, [], [], [], [], [], [], [], [], '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', False, False, False, False, '', '', '', '', False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "E:\SD\automatic1111\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "E:\SD\automatic1111\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "E:\SD\automatic1111\modules\processing.py", line 721, in process_images
        sd_models.reload_model_weights()
      File "E:\SD\automatic1111\modules\sd_models.py", line 764, in reload_model_weights
        sd_model = reuse_model_from_already_loaded(sd_model, checkpoint_info, timer)
      File "E:\SD\automatic1111\modules\sd_models.py", line 713, in reuse_model_from_already_loaded
        send_model_to_cpu(sd_model)
      File "E:\SD\automatic1111\modules\sd_models.py", line 576, in send_model_to_cpu
        m.to(devices.cpu)
      File "e:\SD\automatic1111\venv\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 54, in to
        return super().to(*args, **kwargs)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
        return self._apply(convert)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      [Previous line repeated 1 more time]
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
        param_applied = fn(param)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
        return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    NotImplementedError: Cannot copy out of meta tensor; no data!

---
@wkpark
Copy link
Owner

wkpark commented Dec 26, 2023

Thank you for your reporting,
I have the same issue with the recent 1.7.x sd-webui (currently, I'm using sd-webui dev branch)

Please check some settings "Settings"->"Optimization" and turn off "FP8" optimization (Enable to Disable)

@miasik
Copy link
Author

miasik commented Dec 26, 2023

Thank you for your reporting, I have the same issue with the recent 1.7.x sd-webui (currently, I'm using sd-webui dev branch)

Please check some settings "Settings"->"Optimization" and turn off "FP8" optimization (Enable to Disable)

I don't see/find such an option

@wkpark wkpark self-assigned this Dec 26, 2023
@wkpark wkpark added the bug Something isn't working label Dec 26, 2023
@wkpark
Copy link
Owner

wkpark commented Dec 26, 2023

I guess this is a similar issue. #29

@miasik
Copy link
Author

miasik commented Dec 26, 2023

I guess this is a similar issue. #29
Should I add more logs?

@miasik
Copy link
Author

miasik commented Dec 27, 2023

It looks working now for Sum mode, but Add-Diff/Rebasing still does not.
It shows its progress and a1111 renders a picture in this mode thу picture never changes. If I change mode the picture follows corresponding merges but it freezes again if I switch back to Add-Diff/Rebasing.
Also, what Rebasing and Fast Rb are?

debugs =  ['elemental merge', 'merge', 'adjust']█████████████████                                            | 32/64 [00:08<00:08,  3.85it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  73453409346ffcb55fca326b0e055e898aef8cacd938fcbc304fd6e4a1143e13
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.25]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.25
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:03<00:00, 237.75it/s]
Check uninitialized #2/2: 100%|████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 146382.88it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.03it/s]
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.53it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3482, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

---
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:08<00:00,  3.66it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.71it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.89it/s]

@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

It looks working now for Sum mode, but Add-Diff/Rebasing still does not. It shows its progress and a1111 renders a picture in this mode thу picture never changes. If I change mode the picture follows corresponding merges but it freezes again if I switch back to Add-Diff/Rebasing. Also, what Rebasing and Fast Rb are?

debugs =  ['elemental merge', 'merge', 'adjust']█████████████████                                            | 32/64 [00:08<00:08,  3.85it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  73453409346ffcb55fca326b0e055e898aef8cacd938fcbc304fd6e4a1143e13
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.25]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.25
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:03<00:00, 237.75it/s]
Check uninitialized #2/2: 100%|████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 146382.88it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.03it/s]
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.53it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3482, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

---
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:08<00:00,  3.66it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.71it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.89it/s]

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently.
cond_stage_model.logit_scale key is not for SD1.5

"Fast rebasin" only checks special weights groups.
(rebasin itself checks all permutation groups and sorts keys internally results possible permutations. but almost all orders of permutation groups are not changed (identical) except special groups. so Fast Rebasin will result identical results if it does not found special groups on merging blocks)

@miasik
Copy link
Author

miasik commented Dec 28, 2023

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently. cond_stage_model.logit_scale key is not for SD1.5

It couldn't be the reason, because both models are 1.5 — they were posted as 1.5 and they look fully functional as 1.5 models. Actually, I can merge them as 1.5 models by Sum mode.
Is there a way to just skip unexpected keys and continue merging?

@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently. cond_stage_model.logit_scale key is not for SD1.5

It couldn't be the reason, because both models are 1.5 — they were posted as 1.5 and they look fully functional as 1.5 models. Actually, I can merge them as 1.5 models by Sum mode. Is there a way to just skip unexpected keys and continue merging?

OK. I see, It needs to be checked the validity or acceptable keys in the weight matching routine.

@wkpark wkpark closed this as completed in f9f39fb Dec 28, 2023
@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

This issue has been automatically closed. Please reopen if this issue is not resolved

@miasik
Copy link
Author

miasik commented Dec 28, 2023

As I see — nothing has changed. Still getting the same error and the rendered picture doesn't chande

*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3487, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 839, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if _valid_key(k)}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 839, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if _valid_key(k)}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

wkpark added a commit that referenced this issue Dec 28, 2023
 * fixed mistake
@wkpark wkpark reopened this Dec 28, 2023
@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

thanks for your reporting. I've fixed logic bug again

@miasik
Copy link
Author

miasik commented Dec 28, 2023

Nothing new again. Rebasin doesn't change the image

debugs =  ['elemental merge', 'merge', 'adjust']████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.44it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  0abd1b6d1456626614caacd09c282b776869dcac07788f0289caa9e1a80dbfb5
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model Cartoons\fluffity_v21.safetensors [68536065c9]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.01]
  - adjust
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False , sd2 = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.01
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:13<00:00, 62.83it/s]
Check uninitialized #2/2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 125476.75it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10.52it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
Save unchanged weights #2/2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Clip is fine
 - merge processing in 17.5s (prepare: 0.5s, merging: 17.1s).
 - loading scripts.patches...
 - Fail to patch lora
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3684, in before_process
        base_dict[key] = theta_0[k]
    KeyError: 'cond_stage_model.logit_scale'

---
Using already loaded model Cartoons_fluffity_v21 + (Photo_epicphotogasm_lastUnicorn - v1-5-pruned-emaonly.safetensors [6ce0161689]) x alpha_0(0.25).safetensors [955683620d]: done in 0.5s (send model to device: 0.5s)
Using already loaded model Cartoons_fluffity_v21 + (Photo_epicphotogasm_lastUnicorn - v1-5-pruned-emaonly.safetensors [6ce0161689]) x alpha_0(0.25).safetensors [955683620d]: done in 0.0s
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.37it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.37it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.43it/s]

wkpark added a commit that referenced this issue Dec 28, 2023
 * simple check acceptable keys in the partial update routine
@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

OK. this is another bug. rebasin works just fine. fixed another bug found at the partial update routine.

@miasik
Copy link
Author

miasik commented Dec 28, 2023

Ok, this time i noticed some minor changes and I cant get if it's OK
I set XYZ plot with [Model Mixer] alpha B for X and 0,0.25,0.5,0.75,1 as parameter
Then I got a grid but the changes almost invisible

New errors, one for each X value:

debugs =  ['elemental merge', 'merge', 'adjust']█▍                                                                                                                 | 30/150 [00:07<00:28,  4.26it/s]
use_extra_elements =  True
 - mm_max_models =  4
XYZ: alpha b -> 0.25
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in before_process
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in <listcomp>
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
    ValueError: could not convert string to float: ''

wkpark added a commit that referenced this issue Dec 28, 2023
@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

Ok, this time i noticed some minor changes and I cant get if it's OK I set XYZ plot with [Model Mixer] alpha B for X and 0,0.25,0.5,0.75,1 as parameter Then I got a grid but the changes almost invisible

New errors, one for each X value:

debugs =  ['elemental merge', 'merge', 'adjust']█▍                                                                                                                 | 30/150 [00:07<00:28,  4.26it/s]
use_extra_elements =  True
 - mm_max_models =  4
XYZ: alpha b -> 0.25
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in before_process
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in <listcomp>
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
    ValueError: could not convert string to float: ''

thanks again. you just found another bug again.

and fixed now

@miasik
Copy link
Author

miasik commented Dec 28, 2023

Fantastic! ;-)
I see my new grid and it looks like we've finally done here :-D
Thank you so much!

Next I want to ensure that Rebasin works not worse than Train Difference. Because TD is amazing.

@miasik miasik closed this as completed Dec 28, 2023
@wkpark
Copy link
Owner

wkpark commented Dec 28, 2023

Thank you for your patience and quick feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants