Clear Instructions for Finetuning Faces

[lazy-nurd]

Hey thanks for amazing work,
I was trying to implement finetuning faces like dreambooth works really well in finetuning model on specific face subject. But when I tried finetuning faces there is very less guidelines for it. But I tried training through diffusers and the results were very odd and the identity loss was very high. Training through scripts is difficult to understand since there is no specific guideline for what is instance prompt and what is class prompt.
It would be great if you can provide us better guidelines on finetuning faces easily with custom diffusion.
Also the papers say the model is ~6x faster than dreambooth, however this does not seems to be the case, since training custom diffusion through diffusers is taking almost the same time and gpu memory, please guide us on this as well.
Really looking forward to your reply

[nupurkmr9]

Hi,
Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.

class_prompt: the category of the training images, e.g., person for human faces.
instance_prompt: the text prompt used during training on the target training images, i.e., photo of a <new1> person
training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or using deepspeed library with accelarate. Can you mention which GPU configuration you are training the model on?
regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt.
Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.

For training, using these configurations should be good.

accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4"  \
          --instance_data_dir=./data/cat  \
          --class_data_dir=./real_reg/samples_cat/ \
          --output_dir=./logs/cat  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=750 \
          --num_class_images=200 \
          --scale_lr \
          --modifier_token "<new1>"

[yizhangliu]

@nupurkmr9 If you check regularization images in real_reg/samples_person, you will find that those images are very bad. Most of them are not real faces.

[nupurkmr9]

Hi,
Thanks for pointing this out. Can you try increasing the aesthetic weight in this line? Increasing the weight leads to more time in retrieving the images.
I will update the readme/code to allow that with a flag easily.

Also, our method works with generated images as regularization as well which can be enabled by removing the --real_prior flag from the command.

[SaimAli420]

Hi, Thanks for your interest in our work. I have listed below the answers to your questions. Let me know if you have any other questions.

class_prompt: the category of the training images, e.g., person for human faces. instance_prompt: the text prompt used during training on the target training images, i.e., photo of a <new1> person training time: all our experiments in the paper were with stable-diffusion-1.4 and 2A100 GPUs, on which it takes ~6 minutes and 30GB VRAM. For human faces, a lower learning rate and more training steps are better, which can then take ~18 minutes of training. VRAM requirement can be reduced by using less batch size but more training time, or using deepspeed library with accelarate. Can you mention which GPU configuration you are training the model on? regularization images: we collect real images from the LAION dataset using clip-retrieval with similar captions as class_prompt before starting the training. This can take some time, depending on the class prompt. Number of images for human faces: I haven't benchmarked the performance on human faces extensively. But in my experiments, 15-20 high-resolution images should be good, more is better. An example result is shown here.

For training, using these configurations should be good.

accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4"  \
          --instance_data_dir=./data/cat  \
          --class_data_dir=./real_reg/samples_cat/ \
          --output_dir=./logs/cat  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=750 \
          --num_class_images=200 \
          --scale_lr \
          --modifier_token "<new1>"

Hi,

I appreciate your work. I used same command as you mentioned here.
Model takes at least 3000 steps to recover the identity loss.
Not produce good results on too long prompts.
Finetune with "runwayml/stable-diffusion-v1-5" producing good results than 2.1 version but not too good.

Some of the results comparison between 1.5 vs 2.1 versions.
1.5 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.

Best Results:

2.1 prompts:
Ultra HD quality of styriodio woman as god of war character with beautiful open eyes
beautiful styriodio woman as viking.

Best Results:

On long prompts:
Portrait painting of styriodio as a pretty goth girl, gorgeous tattoos, beautiful hair, symmetrical! intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha

Total identity loss on Emma Watson Images. But dreambooth working fine on these long prompts.

Can you please guide me how I can improve the model on long prompts.
Also guide me about how I can improve the results. Because after 3000 steps still there are some identity loss.
Looking forward to your reply thanks.

[dawei03896]

@nupurkmr9 Thanks for amazing wonderful work, I used the these configurations, but the result is bad.

accelerate launch src/diffuser_training.py
--pretrained_model_name_or_path=$MODEL_NAME
--instance_data_dir=$INSTANCE_DIR
--class_data_dir=$CLASS_DIR
--output_dir=$OUTPUT_DIR
--with_prior_preservation --real_prior --prior_loss_weight=1.0
--instance_prompt="photo of a V* person"
--class_prompt="person"
--resolution=512
--train_batch_size=2
--learning_rate=5e-6
--lr_warmup_steps=0
--max_train_steps=3000
--num_class_images=200
--scale_lr --hflip
--mixed_precision "no"
--modifier_token " V*"

[nupurkmr9]

Hi,

I'm sorry for the delayed response in this thread. I trained a model on 16 face photos with the following configurations on 2 GPUs. Training on faces with the --freeze_model crossattn configuration is usually better than the default setting.

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
CUDA_VISIBLE_DEVICES=1,2 accelerate launch src/diffuser_training.py \
          --pretrained_model_name_or_path=$MODEL_NAME  \
          --instance_data_dir=./face_photos  \
          --class_data_dir=./real_reg/samples_person  \
          --output_dir=./face_model  \
          --with_prior_preservation --real_prior --prior_loss_weight=1.0 \
          --instance_prompt="photo of a <new1> person"  \
          --class_prompt="person" \
          --resolution=512  \
          --train_batch_size=2  \
          --learning_rate=5e-6  \
          --lr_warmup_steps=0 \
          --max_train_steps=2000 \
          --num_class_images=200 \
          --scale_lr  --hflip \
          --freeze_model crossattn \
          --modifier_token "\<new1\>"

And these are the results with sampling done as python src/sample_diffuser.py --prompt <text-prompt> --delta_ckpt <path-to/delta.bin> --ckpt "CompVis/stable-diffusion-v1-4" --freeze_model crossattn --batch_size 10. I selected the best 3 out of 10 generations. I used two prompts from the above post.