CLIP blending and concatenation

[catboxanon]

Based on some findings from an anon on 4chan, for models trained primarily with the penultimate CLIP layer, there seems to be a benefit to blending (and optionally concatenating) a combination of the final and penultimate CLIP layers in order to bring back higher-level, detailed concepts from the final layer, while retaining the quality of the penultimate layer the model was trained with. In the own words of anon:

Some atmospheric tags, like "dark persona", "cyberpunk" (as in you want to character to be cyberpunk and not just standing in front of a cyberpunk city), only get activated at last CLIP layer so I blend the two together

In the future there could even be per-token blends. There really are a lot of possibilities

Some qualitative observations:

1. Negative ratio (meaning the z2 mix being negative) really brings out those atmospheric tags, however:
2. Very much needed to concat z2 if the first ratio is negative. Without z2 there is no image coherence and you get boobs everywhere.
3. Negative ratio doesn't work with most embeds. I think most embeds are overtrained.

This is not necessarily based on analytical evidence and only trial and error, but the results were interesting enough I collaborated with this anon and made a script to implement this, with additional x/y plot support. The change is small enough it's feasible to distribute as a standalone script for now to drop in the scripts directory, but I believe it could also be cloned to the extensions directory if you like.

This is a normal x/y plot for the last layer and penultimate layer of CLIP, as it is in the webui currently:

Full size w/ PNG info

And this is an x/y plot utilizing the script, using several blend ratios, as well as an option for concatenating an additional CLIP layer blend.

Full size w/ PNG info (47mb!)

(Note that for the left-hand chart, you may only focus on the first row, as all others are the same, because CLIP concatenation is disabled for that chart)

These examples are fairly basic and you'll need to write your own prompts to investigate the results for yourself to get a feel for the difference. I've personally found for the default blend defined in the script as-is with concatenation enabled, you can get some very interesting results sometimes compared to having neither enabled. The results for the upper rows of the right-hand chart are also of note for very different results from normal CLIP behavior.

Because this is currently still WIP, saving generation params to PNG info hasn't been implemented yet.

You can find the script here: https://gist.github.com/catboxanon/bdcf23a2699c920da0d8177dbfa34e15

Note that the filename is important for x/y plot support, as the webui parses scripts in alphabetical order, and is unable to add custom x/y axis options if this script is parsed before the xyz grid script.

[warrockmill]

Hi, looks very interesting but having trouble enabling it. The script is running as I see the options on my X/Y/Z scripts but I'm having trouble making it work. I think it's becuase I don't know what information to put in. I'm attaching a screenshot and maybe you can tell me where I'm going wrong. It's probably very obvious but any help would be appreciated. Thank you.

[catboxanon]

Check settings for CLIP Blend, it has to be enabled from there currently. Enabling it also overrides the CLIP skip value, as the way blending happens now the two layers that are blended are fixed (last and penultimate). I would add clip_blend_enabled and clip_blend_concat to your quicksettings for ease of access.

[warrockmill]

Thanks for your answer. I'll give it a go!