What exactly is clip skip? #5674
-
|
Can someone explain in plain English what does CLIP skip option do? As far as I understand, CLIP is a an embedding layer which analyses texts, and changing this option gives different results. And it seems pointless on models that weren't trained with it. But what does it exactly do and what is it for? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 11 replies
-
|
CLIP model (The text embedding present in 1.x models) has a structure that is composed of layers. Each layer is more specific than the last. Example if layer 1 is "Person" then layer 2 could be: "male" and "female"; then if you go down the path of "male" layer 3 could be: Man, boy, lad, father, grandpa... etc. Note this is not exactly how the CLIP model is structured, but for the sake of example. The 1.5 model is for example 12 ranks deep. Where in 12th layer is the last layer of text embedding. Each layer matrix of some size, and each layer is has additional matrixes. So 4x4 first layer has 4 4x4 under it... SO and so forth. So the text space is dimensionally fucking huge. Now why would you want to stop earlier in the Clip layers? Well if you want picture of "a cow" you might not care about the sub categories of "cow" the text model might have. Especially since these can have varying degrees of quality. So if you want "a cow" you might not want "a abederdeen angus bull". You can imagine CLIP skip to basically be a setting for "how accurate you want the text model to be". You can test it out, wtih XY script for example. You can see that each clip stage has more definition in the description sense. So if you have a detailed prompt about a young man standing in a field, with lower clip stages you'd get picture of "a man standing", then deeper "young man standing", "Young man standing in a forest"... etc. CLIP skip really becomes good when you use models that are structured in a special way. Like Booru models. Where "1girl" tag can break down to many sub tags that connect to that one major tag. Whether you get use of from clip skip is really just trial and error. Now keep in mind that CLIP skip only works in models that use CLIP and or are based on models that use CLIP. As in 1.x models and it's derivates. 2.0 models and it's derivates do not interact with CLIP because they use OpenCLIP. |
Beta Was this translation helpful? Give feedback.
-
|
Brilliant explanations, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
May I please know what would be the difference if I set clip_skip=0, 1 or 2. Does sitting to 0 means neurons from all the layers will be used for calculating the final text embedding and 2 means two layers will be completely skipped after each layer which has contributed to text embedding calculations |
Beta Was this translation helpful? Give feedback.
-
The counter was reversed from what it was in the past. Originally it was displayed as "Stop at clip layer n" as in 12 by default. Now it is "stop this many layers from the bottom". So clip skip on 2 would mean, that in SD1.x model where the CLIP has 12 layers, you would stop at 10th layer. If it helps, you can imagine the as floors in a building... or basements rather. You start at ground level 0. This is just the entrace point. If you skip 3 layers, you travel down the sructure to -9 level/U9/K9 or whatever system is used where you happen to be in. It was changed just to make the interface more consitent and easier to decypher. If you are asking about TI training and clip skip, then that is not considered. CLIP skip is used in image generation. Auto's repo also allows you to unload them from memory for the training phases, to save some VRAM. You can do TI-training for text embeddings, without ever generating any images. The process is simply comparing training images to eachother, then taking the average of those, and comparing that to latent space, and deriving the vectors incommon. Basically it is So CLIP skip is irrelevant with text embeddings (TI), the text embeddings is more like... Bypass. It knows exactly the locations of the vectors in Unet, it doesn't need to navigate the CLIP space. Because of this, text embedding (TI) can contain two otherwise contradicting things in it, that you couldn't otherwise get from text prompt. This is because instead of going through the system, when you get to the embedding, the AI can go fetch the results directly, instead of solving them. However this means that TI can not be adjusted to mean other things during the generation, it will only ever refer to specific things. In reality TI embedding is text outside of the CLIP, and preferably it is trained in a manner where it doesn't learn things that the text space can already do. So instead of just embedding words the text space already has in to the clip (even though this can be handy, like using embedding inspector extension, to find your favourite negative tokens and just ramming them in to one easy to use packet), you train it to have things that you can't prompt. Like if you got set of 5 images which have something that you like in common, but not something you can describe with words. You can write the parts you can describe in to the training prompts, and then have the training find the vectors for what you cant. The result of this training, as mentioned earlier, doesn't need bare any connection to "actual words". Using something like embedding inspector you can see what the vectors are closest to in the CLIP/word space, however... they aren't exactly those, and they aren't supposed to be. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
The way CLIP skip was presented in Automatic was changed at some point. However 12 being last or first is irrelevant to the decription of the function. As I wrote that near year ago, and these interfaces going to through near daily changes; take things with some salt. With the diverse range of models and base structure solutions we have nowadays, with different ways they were made the actual settings can vary. Consult the documentation of your interface of choice in order to understand the settings and how they manipulate the model and image generation process. |
Beta Was this translation helpful? Give feedback.
-
|
nice question |
Beta Was this translation helpful? Give feedback.
CLIP model (The text embedding present in 1.x models) has a structure that is composed of layers. Each layer is more specific than the last. Example if layer 1 is "Person" then layer 2 could be: "male" and "female"; then if you go down the path of "male" layer 3 could be: Man, boy, lad, father, grandpa... etc. Note this is not exactly how the CLIP model is structured, but for the sake of example.
The 1.5 model is for example 12 ranks deep. Where in 12th layer is the last layer of text embedding. Each layer matrix of some size, and each layer is has additional matrixes. So 4x4 first layer has 4 4x4 under it... SO and so forth. So the text space is dimensionally fucking huge.
Now why would …