Add option --sample_at_first

[shirayu]

• ✨ Generate samples before training
  • This is useful for comparison before and during the training

🗒️ Note

There is a possibility that the images generated by this and the images generated by sdxl_gen_img.py (or other generation scripts) are slightly different, even with the same parameters. Whether this is true or not has not yet been investigated.

[shirayu]

I have noticed that the reason for the difference in results may be due to --full_bf16 or --full_bf16.
It would be better to add is_full argument to sdxl_train_util.load_target_model and convert the unet, text_encoder1, text_encoder2 in it.

Related to #628

[araleza]

This seems like a good feature. I found myself wanting it just last week.

[kohya-ss]

Thank you for this. I think the difference in results may be acceptable.

In addition, I have another issue in generating samples during training (I got strange sepia images), so I cannot confirm the difference yet.

[kohya-ss]

I found that if --sample_every_n_epochs is set, the script generates the images two times every N epochs after merging. I think this is because the following lines are in the training loop (for epoch ...).

        # For --sample_at_first
        self.sample_images(accelerator, args, epoch, global_step, accelerator.device, vae, tokenizer, text_encoder, unet)

To solve this, I will move these two lines just before the loop.