`LightningModule.log(..., on_epoch=True)` logs with `global_step` instead of `current_epoch`

[quinor]

🐛 Bug

When logging using:
self.log(f"some_metric", value, on_step=False, on_epoch=True)
in ie. training_step, the data is logged to the tensorboard with X axis in steps instead of epochs:

Expected behavior is for the x axis to be in epochs:

To Reproduce

(I'll try to work on reproduction example once I find some free cycles this week)

Environment

Pytorch 1.7 and Lightning 1.1rc

Additional context

@tchaton

[awaelchli]

This is intended behaviour :) Certainly not a bug

[quinor]

Hmm, I sense some miscommunication somewhere @awaelchli , please take a look at this slack conversation: https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1605528802405600 @tchaton told me explicitly it should work the way I described it.

[awaelchli]

The on_epoch argument refers to "when" w.r.t. global step the metric should be logged. An epoch with 10 batches and logging on epoch should create datapoints at steps 9, 19, 29, ... (indexed from 0)

[quinor]

Thanks for the clarification! Then it could be an improvement (or at least there could be an option) to have the labels be epoch numbers instead. I don't know then what tchaton meant on slack though.

[awaelchli]

I would argue basically never would you want that as it makes it even more harder to compare runs :) Making correct comparisons between runs is crucial for tuning hyperparameters and ablations. But besides that, we should not forget that loggers are not plotting tools. There are some advanced loggers like wandb or comet where you can change the plots in the GUI (x axis, labels etc).

[quinor]

I would argue the other way - in the models I'm training I've got epoch-count scheduling on more than one occasion and the batch size is not always constant, so the step-number logging actually works against me there. I can see I'm the only one (or one of the few) needing that option though. Back in Lightning 0.7 or so I was able to just pass {(...), "step": <epoch>} as ie. training_epoch_end return value, but I wasn't able to figure that out with the new logging API.

[chidniu]

This is intended behaviour :) Certainly not a bug

It doesn't sound like that is intended behaviour in the docs https://pytorch-lightning.readthedocs.io/en/stable/logging.html#automatic-logging

Setting on_epoch=True will cache all your logged values during the full training epoch and perform a reduction on_epoch_end. We recommend using the Metrics API when working with custom reduction.

A reduction which just dumps all the cached values to the logger is not exactly very useful and kind of defeats the purpose of these flags imho. The Metric API docs say

If on_epoch is True, the logger automatically logs the end of epoch metric value by calling .compute()

Then looking at e.g. compute() from the Accuracy-Metric-Class

def compute(self):
        """
        Computes accuracy over state.
        """
        return self.correct.float() / self.total

suggests a single accumulated value should be returned, not some kind of list of per batch values. But using the accuracy metric with the on_epoch flag logs per batch values, as described in the original issue (pl 1.0.8).

Either the documentation is very unclear concerning this or the actual behaviour is different from the documentation

[awaelchli]

@chidniu This discussion is about the display of the abscissa in plots of the logger GUI. It has nothing to do with metric computation. If something isn't working for you or the docs are unclear, let's look at it in a separate issue?

@quinor Have you tried self.log("step", self.current_epoch)?

[chidniu]

@awaelchli Thanks for your reply. I think there is a misunderstanding. I am by no means an expert on Tensorboard but it seems to me the problem that is described in this issue is caused by the on_epoch flag not working in the log() call (which I was able to reproduce). If the metrics were aggregated and submitted on-epoch Tensorboard would not display the metrics batch-wise in the GUI.