No training loss logged + val metrics logged incorrectly when using lightning as a backend

[NickleDave]

Hi 👋
I develop a domain-specific library and recently we switched to using lightning as a backend.
We have realized that our logging is not working correctly though (we're using the default Tensorboard logger).
vocalpy/vak#726

Our calls to self.log with the loss in the training_step do not get saved in the events file, and all the metrics we log on validation_step appear to be incorrect--see first attached image.

However, if I run the exact same model in a script then everything gets logged correctly (tested with this notebook)--see second + third attached image.

Things that might matter:

• We subclass the lightning module, the class hierarchy for the model I'm looking at is: lightning.LightningModule <- vak.models.Base <- vak.models.FrameClassificationModel <- vak.models.TweetyNet. Some of this sub-classing happens by calling type directly to make a new sub-class. I know this system is a bit convoluted--we're trying to make it as easy as possible for someone to declare a model--but I'm not sure why it would break logging specifically? I can tell from the progress bar that training is working as expected and the model converges, even though the logs are incorrect
• The broken logging happens when we run through our library's CLI, which runs a __main__ function (i.e., vak train calls vak.__main__.main). But according to the Trainer docstring I would guess this is a good practice? It also means that the DDP strategy doesn't work for us, but that's another issue...
• This is for a model with a non-standard training loop--we train on every possible window from a set of spectrograms, so models are done training before one "epoch". See the trainer we use here. But I replicated that training loop in the notebook and logging seems to work there.

Any guesses as to why running with our library as the main process messes up lightning's logging?
My best guess is that it's something related to the sub-classing, since the script above that works correctly removes that sub-classing, and then logging seems to work correctly