EDMEulerScheduler fails when fineturning CosXL_edit model.

[leeruibin]

When I try to fineturning CosXL model. As they stated in CosXL, CosXL employ Cosine-Continuous EDM VPred scheduler to turn the model. I follow the instruction of train_dreambooth_lora_sdxl to fineturn CosXL model as there are official implementation of edm_style_training by diffuser. However, the model perfermance will become terrible after I try to fineturn it.

I have tried EDMEulerScheduler and DDPMScheduler but none of them seems work well. I have on idea whether the EDMEulerScheduler is exactly the Cosine-Continuous EDM VPred scheduler mentioned by StablilityAI or not. They only release the model parameters and give a very simple statement.

I would be grateful if you could give me some suggestions to make the fineturninig progress work normally.

In my setting, I can edit image successfully when I use instruction like "change the cup to red" before fineturning. However, when I try to fineturn the CosXL_edit model with fusing/instructpix2pix-1000-samples dataset as a toy experiment, the model becomes weird. The loss can decrease normally but the editing fails and the output image contains many artifacts.

My training code is in here. Most of the codes are copied from train_dreambooth_lora_sdxl and cosxl and here are my fineturning results.

I think this is caused by the wrong setting of the training scheduler. Currently, I follow the setting of train_dreambooth_lora_sdxl to set the training scheduler to EDMEulerScheduler, I am not sure whether this is the Cosine-Continuous EDM VPred schedule mentioned by stabilityAI.

The inference setting of the fineturned model is the same as the first line. I use inference step 20, CFG=7, no negative prompt.

The earlier checkpoints also fail. This is the evaluation result after 3K steps. The training batch size is 12.

Thanks a lot!

Originally posted by @leeruibin in #8281 (comment)

[sayakpaul]

Yeah the script doesn't support the scheduling and training objective used by the CosXL model. I think you will have to modify it accordingly so that it fits the same objective. Sadly, our team won't have the bandwidth to look into it in the coming days.

To me, this also feels more like a discussion rather than an issue in the first place.

[leeruibin]

Thanks for you kind reply. You are right, I should have open a discussion session rather than an issue session.

If you or other viewers have any idea about how to set the right scheduling and trainig objective for fineturning CosXL model later on. I would appreciate it.

Thanks!!