improve train loss metric logged in examples

[pacman100]

What does this PR do?

1. Train loss being logged wasn't normalized and as such wasn't intuitive to understand. This also made it difficult to compare train loss between different tools such as comparing train loss from Trainer with that of Accelerate. This PR normalizes the train_loss per epoch to make is more intuitive and comparable.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for fixing!

[benjpau]

@pacman100 @sgugger In a multi-gpu environment, does the total_loss here represent the loss of all training data in the entire epoch or just the loss of training data on the main process?

I tried to print the result of total_loss on each process, and it seems that accelerator.log() only records the total_loss on the main process. So, please, do I need to use accelerator.gather() to get the training loss on the whole epoch? I hope I can get your confirmation, many thanks!

[pacman100]

Hello @benjpau , yes, you are correct, you would need to gather to get the total epoch loss. Will raise a PR to fix the examples to reflect the same.