Anyone had any success with this?

[andykaufseo]

I've tested this and it does what it says, fast speed (high batch count) and low vram usage (can get as low as 4.7gb). But the quality...

Regular dreambooth needs 100-200 steps per image to get it right (and maybe another 100-200 to get it perfect).
With the LORA , after 1000 steps per image it's not even close (i can see it's getting there, it has an idea of what's going on, but the results are like with DB after 20-50 steps).

Tried different learning rates, captions / no captions, etc.

Anyone had any notable success with it?

[brian6091]

Need a bit more detail to diagnose.
What are you training? A style, a person? Are you trying to get photorealism? What is the LoRA rank you are using? Also, note that the learning rate can be much higher (like between 1-2 orders of magnitude higher).

[brian6091]

@andykaufseo
Have a look at the images here: #37
I also detailed the settings that I used; do the other settings (training text encoder, etc) match yours? Also, if you are implmenting the training wrapper yourself, make sure not to enable gradient accumulation and checkpointing, these features currently do not work for the text encoder.

Increasing rank may be useful, like much 10-20 times higher. Are you replacing the default modules? You can also check for overtraining, which is quite peculiar with LoRA. If you don't see it, you can turn up the learning rate or extend training steps.

[andykaufseo]

i did check that, i'm currently testing your colab implementation - one quick question : i have my training images captioned, can i train multiple concepts using these images? i mean not have a single instance token for the fine tuning, but to use all the captions as tuning tokens? would that be possible?

[brian6091]

@andykaufseo

Do you mean you want to insert multiple unique tokens into the tokenizer? I'll have that working in a few days.
I also think that @cloneofsimo just pushed that to master here on this repository.

However, if you are not changing the tokenizer, you can do this simply by properly captioning your images. This depends which training script you're working with. In mine, make sure not to use '{}' in the caption as this will be replaced by your instance_token, and if you are training multiple, it will not work. Rather, just write the full caption for each image, and insert the associated token(s) yourself.

[andykaufseo]

so with your training script can i just caption my images like " holding a spear inside a spaceship"? of course will be replaced with my own token - but will this image also train tokens like holding, spear, spaceship? and if use captions like this, do i need to add anything in the token form field on colab? or can i leave that empty so it trains all tokens?

[brian6091]

@andykaufseo
Yeah, so if your token is TOK, you can name your files "TOK holding a spear inside a spaceship.png", or use a textfile associated with an image (ie., image1.png has an associated image1.txt file with the prompt. The token form field will get ignored as long as you don't include "{}" in the prompts.

Effectively, the other concepts will get trained as well, howmuchso depending on the variety of your dataset.

[cloneofsimo]

Here are some success cases for LoRA within the community, including mine:

https://twitter.com/cloneofsimo/status/1603834716951830529
https://twitter.com/thibaudz/status/1604076391179378689
https://www.reddit.com/r/StableDiffusion/comments/zmr5wo/lora_training_on_myself_with_openjourney_model/
https://twitter.com/xsteenbrugge/status/1602640284445204482
https://www.reddit.com/r/StableDiffusion/comments/zk6zqf/lora_for_subject_training_amazing_results/
https://twitter.com/cloneofsimo/status/1608459563363368961
https://twitter.com/GuglielmoIozzia/status/1604234714692259840
https://twitter.com/Nitrosocke/status/1604845468302938112

Also, if you are using A1111 dreambooth extension for LoRA it might be bit suboptimal. There are 4 major updates after the text-encoding fine-tuning : Pivotal Tuning Inversion, Multiword inversion, ROI conditioned loss, MLP tuning.

My latest results with rank-1 output (0.8 MB output):

[askbekzod]

@cloneofsimo does it mean that there is no golden rule, thus for each new face we have to try diffrent parameters?

[cloneofsimo]

If you are using scripts on the root of the repo, yes. I think It just really boils down to how much tuning you need to get the initial latent right.
In the newest version, my experience was 2000 steps for inversion, 1000 steps for LoRA,
1e-4,1e-5,1e-3 for lrunet,lrclip,lrti, zero initialization + 0.1 weight decay works very well.

[cloneofsimo]

I'm still working on improving the robustness of the performance, and will release 0.0.8 soon

[andykaufseo]

this looks great, if you have any tips for getting those faces just right (and photorealistic), that would be great! also, how many images did you use in your training dataset and how many steps / epochs?

[cloneofsimo]

I used two images, and total of 1500 steps. 500 for training textual inversion, 1000 for LoRA.

[andykaufseo]

and is there a way to fine tune with captions?

[cloneofsimo]

Yes. You can use lora_pti command in the dev branch. like this one:

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export INSTANCE_DIR="./data_example_captioned_small"
export OUTPUT_DIR="./exps/output_example_krk_w_mask_rank1"

lora_pti \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --train_text_encoder \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate_unet=3e-4 \
  --learning_rate_text=3e-5 \
  --learning_rate_ti=1e-3 \
  --color_jitter \
  --lr_scheduler="constant" \
  --lr_warmup_steps=100 \
  --placeholder_tokens="<s1>|<s2>" \
  --placeholder_token_at_data="<s>|<s1><s2>"\
  --initializer_tokens="game|girl" \
  --save_steps=100 \
  --max_train_steps_ti=5000 \
  --max_train_steps_tuning=1000 \
  --perform_inversion=True \
  --weight_decay_ti=0.001 \
  --weight_decay_lora=0.001\
  --continue_inversion_lr=1e-4\
  --device="cuda:0"\
  --lora_rank=1\
  --use_face_segmentation_condition\

By default, you will use the filename as your caption.
You can define the number of tokens by setting placeholder_tokens argument with | splitter : in above case, you are going to use 2 new tokens. (that is two-vector). But for your convenience, i've also added placeholder_token_at_data argument as well. this will map caption token in your caption to token that you have defined in placeholder_tokens.

Example:
If your filename is "<s> looking outside the window.jpg", then your caption during training will be
"<s1><s2> looking outside the window".