Reduced sharpness in img2img / inpainting at lower strength since release v1.9.0, presumably due to step scaling with strength

[gomezzz]

Hi!

First off thank you for this fantastic plugin. I have been using it for quite a while now and have to say you have done a remarkable work here! Thanks a lot. :)

To the problem: I noticed since release v1.9.0 I have a lot of problems with inpainting and img2img processes with the sharpness of the image that seem to be related to the scaling of the number of steps introduced in that version. (Sorry if there is another issue about it, didn't find any, I am currently on version 1.14)

To illustrate, take some image created at 100% Strength (Dreamshaper XL Lightning in case it matters):

Now, if before 1.9.0 I could run at different strengths to do img2img with various bias towards the existing image. Since 1.9.0 I get blurry results for this (this is 50% strength, notice the bird's neck, feathers and eyes as well as background):

If I manually double the stepsize (also 50% strength) to compensate for the scaling it works flawlessly:

This is problem is especially prevalent for inpainting.

Below an inpaint example where a small snippet in the center is inpainted with left column original, center 50% Strength with scaling, right 50% Strength manually doubled steps.

Neither inpaint is perfect (I just picked the first one to avoid cherrypicking) but the scaled one is clearly blurry.

Would it be possible to add a checkbox option to disable the strength scaling? At the moment, I am manually switching styles as a workaround.

With some guidance where to add it I might be able to create a PR for it. :)

(PS: I quickly created this example specifically for the post, I have seen it much more prominently in other examples but I didn't save them.)

[Acly]

I think this is an issue of Turbo/Lightning merges specifically, they are not flexible regarding sample counts and noise schedules.

Technically using 50% strength means running the latter half of the full denoising process, so it should only be half the steps. The same noise is added that would be added in the last 50% of a txt2img generation. It gives the most consistent results and honors the set amount of sample steps best.

I get similar issues with DreamshaperXL_Lightning though, using half the steps gives blury result. Using the same number of steps for a 50% generation as you would for a full gives bad results too though. IMO your 3rd image is noticeably overcooked, it looks worse than the initial generation (1st image).

Not exactly sure yet what the best solution is, if you have more experiments/images let me know.

[gomezzz]

Hi @Acly , thanks for the reply.

May well be and I see what you mean. Maybe providing the user with some control over the strength scaling could be a solution? E.g. a slider for a scaling factor might help?

E.g.
0 - no scaling of strength with steps
0.5 - 75% of max steps with 50% strength etc.
1 - half number of steps with 50% etc. (default value)

That way it could be adjusted by the user depending on the model? There may be smarter / automatic solutions though, my knowledge on the denoising etc. is limited :)

[Acly]

I've added a minimum of 4 steps in the latest release, which might help in some cases. Although not if your total is 8 and you consider 4 at 50% too few (no change there).

No great automatic solution yet, unfortunately it depends a lot on the checkpoint (regular SDXL, LCM, Lightning merge...). Custom scaling factor is quite complex, but might be an option for some kind of presets #497

[RadioVs]

yes, option for minimum steps would be good, i did change it in workflow.py to 8 and now quality is much better

[Acly]

In 1.17.2 it can now also be configued in samplers.json for each preset. Maybe higher values make sense for some models/schedulers.

I kept the default 4, as it works fine for a lot of scenarios and regular checkpoints. My conclusion after some more experimentation is still that turbo/lightning checkpoints are really not great for img2img no matter the step count. They're very inflexible regarding noise schedules, any change in that area quickly leads to poor results.