Simplify the `load_textual_inversion` API

[apolinario]

With Pivotal Tuning for SDXL it's common to train 2 tokens per text-encoder. The current API for loading it is as follows

pipe.load_textual_inversion(state_dict["text_encoders_0"][0], token=["<s0>"], text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)
pipe.load_textual_inversion(state_dict["text_encoders_0"][1], token=["<s1>"], text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)
pipe.load_textual_inversion(state_dict["text_encoders_1"][0], token=["<s0>"], text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)
pipe.load_textual_inversion(state_dict["text_encoders_1"][1], token=["<s1>"], text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)

I think there could be better ways, maybe

To me, the ideal scenario is:
We standardize the name of the state dicts to match the name of the text encoders, so that gets automatically mapped out. If the state dict contains multiple inputs, we allow/require those multiple tokens

Suggested API

pipe.load_textual_inversion("name/repo", weight_name="embeddings.safetensors", token=["<s0>", "<s1>"])

where in the embeddings.safetensors we have a state dict called text_encoder and another one called text_encoder_2 and that gets mapped out accordingly to the respective text_encoder and tokenizer (if the user doesn't provide text_encoder and tokenizer explicitly)

I don't think it needs to be this particular API idea, but I think any move in the direction of simplifying the API would be great!

[patrickvonplaten]

Doesn't the following already work

pipe.load_textual_inversion(state_dict["text_encoders_0"], token=["<s0>", "<s1>"], text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)
pipe.load_textual_inversion(state_dict["text_encoders_1"], token=["<s0>", "<s1>"], text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)

Also do we have lots of usage for pivotal tuning?

[apolinario]

Doesn't the following already work

It doesn't. I tried that too before creating this issue,

ValueError: You have passed a list of models of length 1, and list of tokens of length 2 Make sure both lists have the same length.

Also do we have lots of usage for pivotal tuning?

Yes, as it has become one of the community most used ways to do fine-tuning via the sdxl-cog script. Currently it uses the custom class from sdxl-cog for loading them though -- however we plan on native supporting that with a community script for trianing, so having an easier API to load would be great

[patrickvonplaten]

Doesn't the following already work

It doesn't. I tried that too before creating this issue,

ValueError: You have passed a list of models of length 1, and list of tokens of length 2 Make sure both lists have the same length.

Also do we have lots of usage for pivotal tuning?

Yes, as it has become one of the community most used ways to do fine-tuning via the sdxl-cog script. Currently it uses the custom class from sdxl-cog for loading them though -- however we plan on native supporting that with a community script for trianing, so having an easier API to load would be great

I see! We could make this work very easily!

Ok if we simplify the loading logic then to:

pipe.load_textual_inversion(state_dict["text_encoders_0"], token=["<s0>", "<s1>"], text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)
pipe.load_textual_inversion(state_dict["text_encoders_1"], token=["<s0>", "<s1>"], text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)

(and possibly to even more tokens). This means you'll never need more than two lines of code - one for each text encoder / tokenizer.

I really don't want to make it a one-liner because it means we need to create a class-specific abstraction for SDXL which bloats the code a lot.

[apolinario]

I’m ok with that! Two lines supporting n tokens one per text encoder sound like a good path forward, I’m ok with that so we don’t increase the complexity to make it a one liner

[patrickvonplaten]

@yiyixuxu are you interested in taking this issue? Essentially we should just make sure that we duplicate the state_dict, text_encodre and tokenizer to the same multiple as token if n tokens but only 1 state dict, text encoder and tokenizer are passed:

diffusers/src/diffusers/loaders/textual_inversion.py

Line 386 in ecbe27a

[patrickvonplaten]

@apolinario can you add an example checkpoint that should be better supported here?

[apolinario]

Yes, check this ones out:

• https://huggingface.co/LinoyTsaban/web_y2k_v3/tree/main
• https://huggingface.co/LinoyTsaban/3d_icon_LoRA/tree/main
• https://huggingface.co/LinoyTsaban/huggy_v28/tree/main

[patrickvonplaten]

cc @yiyixuxu

[apolinario]

Thanks @yiyixuxu ! Closed by #5837