Removing the modification of loss value due to rounding off to 4 digits

[harish6696]

System Info

transformers version: 4.50.2
python version: 3.13.1

Who can help?

@zach-huggingface @SunMarc

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Inside the Trainer class, why is the loss rounded to 4 digits? I have applications where I am interested to see the loss go below 4 significant digits, but they all get rounded to 0. Please let the user set this rounding number, or let the loss be displayed in scientific notation like 1.xxxe^y

this is set inside def _maybe_log_save_evaluate()
logs["loss"] = round(tr_loss_scalar / (self.state.global_step - self._globalstep_last_logged), 4)

Expected behavior

It would be great if this hardcoding of rounding off the "loss" is removed. Best output would be to remove the round function and log the loss value in scientific notation which is much clearer.

[Rocketknight1]

I think we could accept a PR for that! Ideally we'd keep the old format for losses > 1e-4 and then switch to scientific notation with 4 (decimal) digits of the significand below that? Please wait for @SunMarc to confirm, though!

[harish6696]

Thanks for the quick reply, @Rocketknight1. Yes, exactly, as soon as the loss <1e-4, we can switch to scientific notation with 4 significant digits in the mantissa and introduce an exponent. Looking forward to hearing the thoughts from @SunMarc as well

[SunMarc]

Happy to have this feature ! Would you like to give it a try @harish6696 ? This rounding was mainly introduced to have cleaner logs #9491. Let's also check that these can be correctly interpreted by third party libraries like wandb.

[harish6696]

@Rocketknight1 and @SunMarc . Yes, I would definitely like to try it out. To be honest, I have never done a pull request to such a big repository, but i am willing to try it. Let me know if there is anything special that i should be careful about.. any hints/suggestions are also welcome.

[SunMarc]

Let me know if there is anything special that i should be careful about.. any hints/suggestions are also welcome.

Since this is a small change, just check that it logs the loss as you expected on wandb for example !

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[marcndo]

Hi @harish6696, are you still working on this? If not, i'd be happy to pick it up, if that's oK by you and @SunMarc .

[harish6696]

Hi @marcndo, I have not yet started with that implementation. Please feel free to do it if you have time. No problem from my side.

[marcndo]

OK, thank you.

[marcndo]

Hi @SunMarc, I’ve linked a PR for this issue. Please review it and let me know if any changes are needed. Also, I decided to log both cases - loss values greater than 1e-4 and less than or equal to 1e-4 — as strings for consistency.