How to log metrics during training

[PaulScemama]

Hi,

Sorry to ask so many questions! And thanks again for creating such a great library.

I'd like to log the loss / accuracy and other metrics during training...for example with Tensorboard or Weights & Biases. I've looked at Callbacks but it appears they interact with TrainerState which seems to only contain the parameters of the model state.

Do you know if there is any easy way to construct a Callback function to retrieve predictions and ground truth for a particular epoch? Then I could compute whatever sort of metrics I'd want.

Thank you!

[gianlucadetommaso]

If it is only accuracy, loss, or other metrics that only depend on the predictions and data, you can do this in FitMonitor. Something like this:

from fotuna.prob_model import FitConfig, FitMonitor
from fortuna.metric.classification import accuracy
fit_config = FitConfig(monitor=FitMonitor(metrics=(accuracy,)))

You can also pass arbitrary user-defined metrics, as long as it has the same signature as accuracy. The training loss should be logged automatically. The validation loss will be logged if you pass validation data and set the logger to INFO level.

About the callback, yes, look at FitCallback. This is also part of the FitConfig objects. You can do something like

fit_config = FitConfig(
    monitor=FitMonitor(metrics=(accuracy,)),
    callbacks=[FitCallback(...)]
)

Look here to check how FitCallback works. You can set a custom callback either at train_epoch_end, or train_epoch_start, or train_step_end. Anyway, if all you want is accuracy and loss, you won't need this.

[PaulScemama]

Okay great, thank you! I see that the loss/accuracy are printed during training, but I'm unsure how I can go inside the training loop and, for every epoch, do something like wandb.log({"val_loss": val_loss"}), etc.

Any ideas?

[gianlucadetommaso]

The training and validation metrics and loss values are all saved into the status when you do

status = prob_model.train(...)

Perhaps is this what you need? I am not sure if I understand the use case of going inside the trainer yet.

[PaulScemama]

@gianlucadetommaso yes this is good!