Use upload_folder in training scripts

[Wauplin]

Related to #2860.

Updated some training scripts to use upload_folder instead of Repository.push_to_hub. The two advantages are upload speed (x1.5?) and UX (progress bars).

As said in #2860 (comment), upload_folder does not support blocking=False (see huggingface/huggingface_hub#939). This means I did not update the scripts that are pushing weights while training (train_dreambooth_flax.py, ./research_projects/onnxruntime/unconditional_image_generation/train_unconditional.py and ./unconditional_image_generation/train_unconditional.py).

Worth mentioning that this is not ISO-feature with the previous scripts. In particular, upload_folder does not take into account if there is a .gitignore file in the output_dir already.

(Personal opinion: I think this PR makes the training script a bit cleaner (less duplicated code) but if you prefer to stick to the existing ones for consistency, I completely get it as well)

[yiyixuxu]

Can we merge this one in?
I tested the Pytorch controlnet here https://huggingface.co/YiYiXu/test_uploadfolder
is it sufficient? or do we have to test all the scripts?

[sayakpaul]

Thanks a lot for handling this one!

Also, I love the little cleaning you have done to remove the bloating of code.

Could we also ensure the related tests pass the CI?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sayakpaul]

@yiyixuxu could you give this a quick review as well?

I think we're good to merge this one.

[yiyixuxu]

thanks!

[sayakpaul]

@patrickvonplaten could you give it a quick look as well?

[Wauplin]

Some tests are still failing but I'm not sure they are related to my changes.

Otherwise, I wrote a few warnings in the description but that's it from my side.

[sayakpaul]

Yes, sorry @Wauplin!

https://github.com/huggingface/diffusers/actions/runs/4599394852/jobs/8124713689?pr=2934 should be fixed with #2959. MPS test failure is unrelated.

[patrickvonplaten]

Super cool PR - much more concise thanks a lot!

[patrickvonplaten]

Merging, taking:

I think we're good to merge this one.

as a "approved"