CLIP guidance

[thibaudart]

Add a CLIP guidance like dreamstudio.
the results seems good!

[slymeasy]

Did he ever implement this?

[Ehplodor]

AFAIK, no.

[pablx-ts]

CLIP guidance is a big deal! We have it in Deforum v0.6 and in DreamStudio. Hope we get it here too!

[ZeroCool22]

It will make the generation slower?

[thibaudart]

It could be a little. On dreamstudio, the difference is 1 or 2 seconds. Maybe less.

[slymeasy]

I could have sworn I saw something about CLIP when I updated a yesterday.

[space-nuko]

I hacked on the implementation by Birch-san onto my fork, all credit goes to him: https://github.com/space-nuko/stable-diffusion-webui/tree/feature/clip-guidance

With that I get some decent results. I hope this brings auto closer to merging CLIP guidance someday!

Original without CLIP guidance

img2img with CLIP guidance, ViT-B-16-plus-240, pretrained=laion400m_e32, guidance scale 300

Note the above picture took ~4min to render on my 3090 and used up all 24GB of VRAM with batch size 1

[thibaudart]

I just copypaste your code but I think something go wrong.

[space-nuko]

I dont think you should copy-paste it but checkout my fork like this first

git clone https://github.com/space-nuko/stable-diffusion-webui --branch feature/clip-guidance

Then go into the settings and look for the Clip guidance option

[thibaudart]

I'll try to make a fresh install with your branch.

[thibaudart]

I did a fresh install. With SD2.1 512 I got this kind of renders.

ok I just saved settings once and now it's working. I think with the fresh install some settings are not correctly initialized!

[andupotorac]

I dont think you should copy-paste it but checkout my fork like this first

git clone https://github.com/space-nuko/stable-diffusion-webui --branch feature/clip-guidance

Then go into the settings and look for the Clip guidance option

Hi. I installed this following your suggested line. What does the setting look like in Settings? I can't find it when searching for "guidance".

[paniq]

there's also a test space here https://huggingface.co/spaces/EleutherAI/clip-guided-diffusion