[FAQ] On-training custom metrics

[LordGedelicious]

I'm wondering if there's a way to customize the metrics logged per step and per epoch of the model training process. Right now, I'm training a binary classifier model (MambularClassifier) and it's only showing the train_loss_step, val_loss, and train_loss_epoch for the metrics. Is it possible to add other metrics such as MSE, RMSE, accuracy, val_accuracy? Other than extending the class implementation myself of course.

[AnFreTh]

Great Suggestion! We will add it in one of the next versions.
I will transform this to an "enhancement"

[LordGedelicious]

Thank you for your hard work! Quick follow up question if you don't mind, since the models are implemented based on Sklearn's base models, can mapping learning curves and/or validation curves be performed with the from sklearn.model_selection import learning_curve functionality? Right now, I'm noting the values manually since it takes 3-4 minutes per epoch to be mapped later on with matplotlib

[AnFreTh]

Since all models are ultimately training via lightning, this is unfortunately not possible. Once, logging custom metrics is possible, everything can logged via the trainer kwargs in the fit method

[AnFreTh]

Included in 850a5cc -> Will be published in next release.
TODO: adapt documentation

[AnFreTh]

Thanks again for raising this. The newest release #222 includes this now. We are happy if you have further suggestions to raise issues similar to this one. If you have further questions regarding the metrics, feel free to reopen this issue or raise another one.