Multi-vector Token already in tokenizer vocabulary. Please choose a different token name

[7iFinalBoSS]

Describe the bug

I'm trying to load a embedding in a pipe then i unload it after that the same embedding i'm loading so it's giving me the error

please look into this thanks

Reproduction

`from diffusers import AutoPipelineForText2Image
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

file = hf_hub_download("dn118/unaestheticXL", filename="unaestheticXLv31.safetensors")
state_dict = load_file(file)
state_dict

pipe = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", variant="fp16", torch_dtype=torch.float16)
pipe.to("cuda")

pipe.load_textual_inversion(state_dict["clip_g"], token="unaestheticXLv31", text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)
pipe.load_textual_inversion(state_dict["clip_l"], token="unaestheticXLv31", text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)`

pipe.unload_textual_inversion(tokens="unaestheticXLv31", text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer) pipe.unload_textual_inversion(tokens="unaestheticXLv31", text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)

pipe.load_textual_inversion(state_dict["clip_g"], token="unaestheticXLv31", text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2) pipe.load_textual_inversion(state_dict["clip_l"], token="unaestheticXLv31", text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)

Logs

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-6abe504f2e02> in <cell line: 1>()
----> 1 pipe.load_textual_inversion(state_dict["clip_g"], token="unaestheticXLv31", text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)
      2 pipe.load_textual_inversion(state_dict["clip_l"], token="unaestheticXLv31", text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)

2 frames
/usr/local/lib/python3.10/dist-packages/diffusers/loaders/textual_inversion.py in _extend_tokens_and_embeddings(tokens, embeddings, tokenizer)
    249                     i += 1
    250 
--> 251                 raise ValueError(
    252                     f"Multi-vector Token {multi_vector_tokens} already in tokenizer vocabulary. Please choose a different token name or remove the {multi_vector_tokens} and embedding from the tokenizer and text encoder."
    253                 )

ValueError: Multi-vector Token ['unaestheticXLv31', 'unaestheticXLv31_1', 'unaestheticXLv31_2', 'unaestheticXLv31_3', 'unaestheticXLv31_4', 'unaestheticXLv31_5', 'unaestheticXLv31_6', 'unaestheticXLv31_7'] already in tokenizer vocabulary. Please choose a different token name or remove the ['unaestheticXLv31', 'unaestheticXLv31_1', 'unaestheticXLv31_2', 'unaestheticXLv31_3', 'unaestheticXLv31_4', 'unaestheticXLv31_5', 'unaestheticXLv31_6', 'unaestheticXLv31_7'] and embedding from the tokenizer and text encoder.

System Info

Google colab

Who can help?

@sayakpaul

[tolgacangoz]

unaestheticXLv31.safetensors includes 8 tokens in total, you can see their names in the error message. You can also confirm this by looking at shape: state_dict['clip_g'].shape[0]. So, when unloading, if you give tokens=["unaestheticXLv31"] + [f"unaestheticXLv31_{i}" for i in range(1, 8)], the error disappears.

[7iFinalBoSS]

thanks