Feat/inversion pp

[cloneofsimo]

Few utilities, including CLIP evaluation, CLIP evaluation preparation, random initialization with sigma.

So I've implemented CLIP text alignment, Image alignment in this PR #67 (comment)

Expect to see some results like above figure, from Custom Diffusion.
https://arxiv.org/abs/2212.04488

[cloneofsimo]

Bit more commits coming...

[brian6091]

Will start to look at this today. Just to clarify, the target_imgs input:

def evaluate_pipe(
pipe,
target_images: List[Image.Image],
...
):

should be a list of images matching the example prompts. I'm guessing these could be generated by the same pipe if necessary?

[cloneofsimo]

Will start to look at this today. Just to clarify, the target_imgs input:

def evaluate_pipe(
pipe,
target_images: List[Image.Image],
...
):

should be a list of images matching the example prompts. I'm guessing these could be generated by the same pipe if necessary?

Target images are the reference images, so they are like ground truth images

[brian6091]

Will start to look at this today. Just to clarify, the target_imgs input:

def evaluate_pipe(
pipe,
target_images: List[Image.Image],
...
):

should be a list of images matching the example prompts. I'm guessing these could be generated by the same pipe if necessary?

Target images are the reference images, so they are like ground truth images

Right, got it, but I imagine when we actually use this, we won't have ground truth images for all the prompts, so we can generate "ground truth" using the pipe (probably better to use the original model, rather than the trained).

[cloneofsimo]

So these seemed to work very well, I'll add these with updated example runfile + example dataset.

[cloneofsimo]

Will start to look at this today. Just to clarify, the target_imgs input:

def evaluate_pipe(
pipe,
target_images: List[Image.Image],
...
):

should be a list of images matching the example prompts. I'm guessing these could be generated by the same pipe if necessary?

Target images are the reference images, so they are like ground truth images

Right, got it, but I imagine when we actually use this, we won't have ground truth images for all the prompts, so we can generate "ground truth" using the pipe (probably better to use the original model, rather than the trained).

Oh so I was thinking, we have target subject X to train, testing on prompt Y and see how well it creates image : generated image Z should be faithful to prompt Y, and to subject X. that measure : sim(Z,Y), sim(Z,X) is what we are trying to get here

X : <Custom Yellow Clock>
Y : "photo of clock on the tree"
Z : (Photo of custom yellow clock on tree)

So our only source (ground truth) images are X, since Y are text, and Z is generated with SD
At least that's what I've understood from textual inversion paper and CLIP score papers... It's not explained clearly in the paper as well, so please correct me if I'm wrong!!

[cloneofsimo]

I am seeing incredible quality improvements combining all the fancy latest tricks. Especially giving high norm prior gave much editability unlike before.

I'm so proud of this 🤣

[cloneofsimo]

I'll merge this I guess

[brian6091]

Will start to look at this today. Just to clarify, the target_imgs input:

def evaluate_pipe(
pipe,
target_images: List[Image.Image],
...
):

should be a list of images matching the example prompts. I'm guessing these could be generated by the same pipe if necessary?

Target images are the reference images, so they are like ground truth images

Right, got it, but I imagine when we actually use this, we won't have ground truth images for all the prompts, so we can generate "ground truth" using the pipe (probably better to use the original model, rather than the trained).

Oh so I was thinking, we have target subject X to train, testing on prompt Y and see how well it creates image : generated image Z should be faithful to prompt Y, and to subject X. that measure : sim(Z,Y), sim(Z,X) is what we are trying to get here

X : <Custom Yellow Clock> Y : "photo of clock on the tree" Z : (Photo of custom yellow clock on tree)

So our only source (ground truth) images are X, since Y are text, and Z is generated with SD At least that's what I've understood from textual inversion paper and CLIP score papers... It's not explained clearly in the paper as well, so please correct me if I'm wrong!!

Ok this makes sense now. Thanks!

[brian6091]

I am seeing incredible quality improvements combining all the fancy latest tricks. Especially giving high norm prior gave much editability unlike before.

I'm so proud of this 🤣

Amazing! But you can't just drop the image without telling us what tricks you used? And what is high norm prior???

[cloneofsimo]

In this PR I made 5 changes to get it work :

1. I used multivector initialization, so it's extended latent. Which quite surprisingly isn't yet implemented in hf textual inversion.
2. Gradient accumulation
3. Face conditioned loss, but when textual inversion, we set high blur amount to recognize other features as well
4. Norm prior : so we give gaussian prior on the norm to be 0.4, and so if the norm is too large, we project it closer to 0.4
5. Full precision on textual inversion : high precision is needed during inversion. I don't know why, but seems to be the case.

[brian6091]

Thanks for the secret sauce. Very clever the multivector initialization. Does that mean your prompts include all the tokens together?

[hafriedlander]

This has gotten bigger than last time I looked :). I haven't had time to understand all the changes, but the results speak for themselves. Great work!

[cloneofsimo]

They have <krk> in prompts, and they are substituted by extended tokens( <s1><s2><s3> in my case)