Push sharded checkpoint to hub when push_to_hub=True in TrainingArguments

[SunMarc]

What does this PR do ?

This PR make sure that sharded checkpoints are pushed to the hub when we set push_to_hub=True in TrainingArguments.
Fixes #30724.
Thanks @alvarobartt for the detailed issue !

I didn't add a test as it requires a big model to test it and we can't change the shard size when saving from Trainer.

To reproduce:

I was able to successfully push the checkpoints here : https://huggingface.co/marcsun13/sft_openassistant-guanaco/tree/main

python trl/examples/scripts/sft.py     --model_name_or_path="mistralai/Mistral-7B-v0.1"     --dataset_text_field="text"     --report_to="wandb"     --learning_rate=1.41e-5     --per_device_train_batch_size=2     --gradient_accumulation_steps=2     --output_dir="sft_openassistant-guanaco"   --torch_dtype="bfloat16"  --optim=adamw_bnb_8bit   --logging_steps=1     --num_train_epochs=3     --hub_strategy="every_save" --save_strategy="steps"  --save_steps=10    --push_to_hub

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[muellerzr]

Thanks for the fix! 🥳

[amyeroberts]

Thanks for enabling this!

Regarding testing, I'm happy to have merged as you've been able to run. However, not being able to test is a symptom of something needing to change on our code side to enable this.