self.log in ddp model change `reduce_fx` to "mean", "min" or "max", but always got sum result

[randydl]

Dear Team:
As the title I said, the default behavior of self.log when sync_dist=True in ddp model is average all GPUs' results, but in the csv report I always got sum result.

[ananthsub]

Hi @randydl , are you using torchmetrics for calculating these metrics? for example, to track the average: https://github.com/PyTorchLightning/metrics/blob/master/torchmetrics/average.py

This way your metric updates, computation, and syncing is all encapsulated in the torchmetrics class, and the logging (transferring data from the LightningModule to the trainer) is handled when calling self.log

Would you find this more convenient?

[jordan-benjamin]

@randydl I've had similar issues, when I upgraded my PyTorch Lightning version yesterday and ran identical code from before I updated versions, my logged values were 4x larger because they were not divided by the number of GPUs. Just as you said, this behaviour didn't change even if I explicitly specified reduce_fx="mean".

I observed this behaviour on package versions 1.4.3 and 1.4.0, unfortunately I don't recall which version I had installed when the averaging was working properly, just that that it was earlier than 1.4.0.

[tchaton]

Dear @jordan-benjamin @randydl,

Thanks for raising this issue. I am increasing priority to P0.

Best,
T.C