[Examples] Create textual inversion training script for SDXL

[sayakpaul]

We have training for textual inversion in place: https://github.com/huggingface/diffusers/tree/main/examples/textual_inversion.

Might be worth adding one compatible with SDXL.

[AI-Casanova]

There are a few TI embeddings in the community already Example trained, I assume, on Kohya. It would be appreciated if the loading script for Diffusers was cross compatible (which would most likely just be a matter of being embedding size/order agnostic I assume)

Also users of SDNext would be thrilled if loading came before training.

[sayakpaul]

Cc: @patrickvonplaten for the loading the part as he brought in the support for this:

(from here)

[patrickvonplaten]

@AI-Casanova can you point to some SDXL TI examples that we could use to implement a loading function?

[AI-Casanova]

@patrickvonplaten the most popular one on Civitai is linked above, and here is another https://civitai.com/models/119032/unaestheticxl-or-negative-ti

They are rare enough at the moment, likely because one has to be on A1111 Dev branch or Comfyui in order to use them.

[patrickvonplaten]

Cool, I'll give it a look :-)

[jelling]

Also users of SDNext would be thrilled if loading came before training.

@patrickvonplaten can you comment on whether it will be possible to use existing TIs with XL? Or will fresh training be necessary?

[sayakpaul]

Fresh script and fresh training is necessary.

[alfredplpl]

@patrickvonplaten I would like to use negative TIs on the Hugging Face Space: https://huggingface.co/spaces/aipicasso/picasso-diffusion-latest-demo .
Are there any alternatives?

[alfredplpl]

Is this alternative method?

pipe = StableDiffusionXLPipeline.from_pretrained(model_id, torch_dtype=torch.float16)

embeddings_dict = {}
with safe_open("unaestheticXLv31.safetensors", framework="pt") as f:
    for k in f.keys():
        embeddings_dict[k] = f.get_tensor(k)

print(embeddings_dict["clip_l"].shape,embeddings_dict["clip_g"].shape)
unaestheticXLv31=""
for i in range(len(embeddings_dict["clip_l"])):
    token = f"sksd{chr(i+65)}"
    unaestheticXLv31 += token
    pipe.tokenizer.add_tokens(token)
    token_id = pipe.tokenizer.convert_tokens_to_ids(token)
    pipe.text_encoder.resize_token_embeddings(len(pipe.tokenizer))
    pipe.text_encoder_2.resize_token_embeddings(len(pipe.tokenizer))
    pipe.text_encoder.get_input_embeddings().weight.data[token_id] = embeddings_dict["clip_l"][i]
    pipe.text_encoder_2.get_input_embeddings().weight.data[token_id] = embeddings_dict["clip_g"][i]

However, I got the warning:

You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embeding dimension will be 49409. This might induce some performance reduction as *Tensor Cores* will not be available. For more details  about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

How can I fix it?

[AI-Casanova]

@alfredplpl
pipe.text_encoder.resize_token_embeddings(len(pipe.tokenizer),pad_to_multiple_of=128) pipe.text_encoder_2.resize_token_embeddings(len(pipe.tokenizer),pad_to_multiple_of=128)

Seems to work, and won't pad beyond what's necessary even with the loop.

[alfredplpl]

@AI-Casanova Thanks. I got it.

[oss-roettger]

Maybe this can help you to fix the TI huggingface pipeline for SDXL:
I' ve pnublished a TI stand-alone notebook that works for SDXL. Since it uses the huggigface API it should be easy for you to reuse it (most important: actually there are two embeddings to handle: one for text_encoder and also one for text_encoder_2):
https://github.com/oss-roettger/XL-Textual-Inversion