[Bug]: ControlNet fails miserably when settings are set by Config-Presets (model not loading properly)

[marcsyp]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

Took a few hours to track down what was happening here, but I have found the issue and the workaround. Essentially the issue is this -- if you use the Config-Presets extension (absolutely essential for complex workflow management) to manage your generation parameters and tasks, and if you add controlnet parameters to the tracking list, everything appears to work perfectly. Parameters are saved and loaded properly from the config.

HOWEVER -- when you try to generate images in img2img, the results are absolutely nonsense fuzziness and weirdness.

WORKAROUND: Click and unclick the Pixel Perfect option on your CN config. I suspect you have to do this for each controlnet that you have set up in your preset, but I have only tested with a single controlnet. I noticed when you change this option, it refreshes the model list and I think that this is what fixes the problem, which (speculation) may be that the model is not being loaded properly when the model selection is done programmatically.

Sample Image

Source:

Bad result:

Result after workaround:

Steps to reproduce the problem

1. Install Config-Presets
2. Add control net fields to your tracking list (see Add'l information for keys to add to your config file)
3. Reload UI
4. Go to img2img, load up a file and set some parameters, including a CN model with tile_resample and tile model, select enable and Pixel Perfect.
5. Save a preset (reloads UI)
6. Go to img2img again, load the preset, and supply a source image.
7. GENERATE - See fuzzy results
8. Click off Pixel Perfect (model will regenerate), click back on.
9. GENERATE - Results as expected
10. If you then load a different preset with same model -- results are as expected (probably because model is still properly loaded
11. If you load a different preset with a different model -- I suspect results will be off again because the old model will still be loaded (not tested)

What should have happened?

Model should load properly when Config-Presets is used, and then...
Beautiful CN goodness at the push of a button
... also good to ensure that previous models are unloaded when Config-Presets makes a change

Commit where the problem happens

controlnet: 1.1.173
python: 3.10.6  •  torch: 1.13.1+cu117  •  xformers: 0.0.16rc425  •  gradio: 3.23.0  •  commit: 22bcc7be

What browsers do you use to access the UI ?

No response

Command Line Arguments

set COMMANDLINE_ARGS=--opt-channelslast --use-cpu interrogate --theme dark --no-half-vae --xformers

Console logs

none out of the ordinary:

Restarting UI...
ControlNet v1.1.173
Image Browser: ImageReward is not installed, cannot be used.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 13.5s (load scripts: 0.4s, create ui: 12.9s, gradio launch: 0.1s).
[Config-Presets] Changed to: Upscale 3x (Stage 1) - Early Release
[Config-Presets] Changed to: Upscale 2x (Stage 2) - Guidance End Early
4.5
Canva size: 1536x2304
Image size: 768x1136
Scale factor: 3
Upscaling iteration 1 with scale factor 3
Tile size: 2304x768
Tiles amount: 3
Grid: 3x1
Redraw enabled: True
Seams fix mode: NONE
Loading model: control_v11f1e_sd15_tile [a371b31b]
Loaded state_dict from [C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.pth]
Loading config: C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml
ControlNet model control_v11f1e_sd15_tile [a371b31b] loaded.
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:13<00:00,  1.15s/it]
could not find upscaler named <empty string>, using None as a fallback
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:13<00:00,  1.15s/it]
could not find upscaler named <empty string>, using None as a fallback
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:14<00:00,  1.22s/it]
could not find upscaler named <empty string>, using None as a fallback
Total progress: 36it [26:59, 45.00s/it]
4.5al progress: 36it [26:59,  1.11s/it]
Canva size: 1536x2304
Image size: 768x1136
Scale factor: 3
Upscaling iteration 1 with scale factor 3
Tile size: 2304x768
Tiles amount: 3
Grid: 3x1
Redraw enabled: True
Seams fix mode: NONE
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:14<00:00,  1.22s/it]
could not find upscaler named <empty string>, using None as a fallback                 | 12/36 [00:12<00:25,  1.05s/it]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
 42%|██████████████████████████████████▌                                                | 5/12 [00:06<00:09,  1.38s/it]
could not find upscaler named <empty string>, using None as a fallback                 | 17/36 [00:24<00:31,  1.63s/it]
Total progress:  47%|███████████████████████████████▏                                  | 17/36 [00:29<00:33,  1.74s/it]
4.5al progress:  47%|███████████████████████████████▏                                  | 17/36 [00:29<00:31,  1.63s/it]
Canva size: 1536x2304
Image size: 768x1136
Scale factor: 3
Upscaling iteration 1 with scale factor 3
Tile size: 2304x768
Tiles amount: 3
Grid: 3x1
Redraw enabled: True
Seams fix mode: NONE
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:14<00:00,  1.24s/it]
could not find upscaler named <empty string>, using None as a fallback                 | 12/36 [00:12<00:25,  1.06s/it]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:14<00:00,  1.18s/it]
could not find upscaler named <empty string>, using None as a fallback                 | 24/36 [00:34<00:13,  1.13s/it]
Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
Loading preprocessor: tile_resample
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.RESIZE
raw_H = 832
raw_W = 2368
target_H = 832
target_W = 2368
estimation = 832.0
preprocessor resolution = 832
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:13<00:00,  1.16s/it]
could not find upscaler named <empty string>, using None as a fallback█████████████████| 36/36 [00:55<00:00,  1.12s/it]

Additional information

ControlNet (extension)

img2img_controlnet_ControlNet-0_controlnet_enable_checkbox
img2img_controlnet_ControlNet-0_controlnet_low_vram_checkbox
img2img_controlnet_ControlNet-0_controlnet_pixel_perfect_checkbox
img2img_controlnet_ControlNet-0_controlnet_preprocessor_preview_checkbox
img2img_controlnet_ControlNet-0_controlnet_preprocessor_dropdown
img2img_controlnet_ControlNet-0_controlnet_model_dropdown
img2img_controlnet_ControlNet-0_controlnet_control_weight_slider
img2img_controlnet_ControlNet-0_controlnet_start_control_step_slider
img2img_controlnet_ControlNet-0_controlnet_ending_control_step_slider
img2img_controlnet_ControlNet-0_controlnet_control_mod_radio
img2img_controlnet_ControlNet-0_controlnet_resize_mode_radio
img2img_controlnet_ControlNet-1_controlnet_enable_checkbox
img2img_controlnet_ControlNet-1_controlnet_low_vram_checkbox
img2img_controlnet_ControlNet-1_controlnet_pixel_perfect_checkbox
img2img_controlnet_ControlNet-1_controlnet_preprocessor_preview_checkbox
img2img_controlnet_ControlNet-1_controlnet_preprocessor_dropdown
img2img_controlnet_ControlNet-1_controlnet_model_dropdown
img2img_controlnet_ControlNet-1_controlnet_control_weight_slider
img2img_controlnet_ControlNet-1_controlnet_start_control_step_slider
img2img_controlnet_ControlNet-1_controlnet_ending_control_step_slider
img2img_controlnet_ControlNet-1_controlnet_control_mod_radio
img2img_controlnet_ControlNet-1_controlnet_resize_mode_radio
img2img_controlnet_ControlNet-2_controlnet_enable_checkbox
img2img_controlnet_ControlNet-2_controlnet_low_vram_checkbox
img2img_controlnet_ControlNet-2_controlnet_pixel_perfect_checkbox
img2img_controlnet_ControlNet-2_controlnet_preprocessor_preview_checkbox
img2img_controlnet_ControlNet-2_controlnet_preprocessor_dropdown
img2img_controlnet_ControlNet-2_controlnet_model_dropdown
img2img_controlnet_ControlNet-2_controlnet_control_weight_slider
img2img_controlnet_ControlNet-2_controlnet_start_control_step_slider
img2img_controlnet_ControlNet-2_controlnet_ending_control_step_slider
img2img_controlnet_ControlNet-2_controlnet_control_mod_radio
img2img_controlnet_ControlNet-2_controlnet_resize_mode_radio

[marcsyp]

I should add that if you use the workflow above (with workaround to get good images) for a little while, eventually you start getting OOM errors, which indicates to me that there is something not being properly unloaded in certain scenarios as well. A UI refresh does not solve this, you typically need to reboot the server to recapture the memory.

Just thought I would point that out since I speculated in the original issue that model unloading is likely an issue too.

[marcsyp]

I'll also note that setting ControlNet values with a Preset does not always trigger the correct sliders to appear, depending on the Pixel Perfect setting. Any thoughts @lllyasviel or @Mikubill? Thank you!

P.s. The pace and quality of development on this project is absolutely mindblowing. Thank you to all the contributors.

[marcsyp]

After a little digging in the gradio api and looking through the controlnet + configpresets code, I believe that this has to do with the gradio event listeners not working smoothly with the way that ConfigPresets is executing its update cycle. ControlNet appears to use the appropriate listeners (e.g., .change()), which should respond to programmatic changes, but perhaps the order or timing of execution from ConfigPresets is not optimal.

Unfortunately I'm not a gradio or python expert (getting back into python after many years) so I'm currently at a loss on where to start on looking for a solution, but in general it would seem that solving this problem paves the way for successful implementation of other programmatic change paradigms (like repopulating ControlNet parameters from gen params, for instance).

If anyone has any ideas on where I could start looking for a potential solution please let me know.

Thanks!

[lllyasviel]

I think perhaps we add wontfix tag to this. handling this seems causing deadlock in some logics

[marcsyp]

Hi @lllyasviel -- no problem, I totally understand if it causes problems in other parts of the logic to fix. I can live with the workaround. Would you mind explaining a bit more about the solution you tested and the logic it breaks so that I tinker around with my own branch as a learning exercise? Thank you!