Some wrong in sd3's lora training script

[CuddleSabe]

Describe the bug

diffusers/examples/research_projects/sd3_lora_colab/train_dreambooth_lora_sd3_miniature.py

Line 717 in e8aacda

transformer.to(accelerator.device, dtype=weight_dtype)

the transformer before accelerator.prepare should not convert to fp16 in mix precision;
it will broken the grad precision and result in worse training result

Reproduction

see the "/examples/research_projects/sd3_lora_colab/train_dreambooth_lora_sd3_miniature.py"

Logs

System Info

diffusers 0.31.0

Who can help?

No response

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Accelerate wraps the model such that outputs are in fp32, while also using gradient scaling when training in fp16, so it should be okay. accelerator.prepare only distributes the model across GPUs, so whether we do it before/after it should not really affect training. Have you faced any particular bad runs that you can share examples of? If this can be improved, could you open a PR?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.