Add a model "calibration" step.

[coforfe]

Hello Max,

Firstly, thanks to both of you for this excellent reference material about tidymodels.

I have just checked if there is any section about model calibration but I could not find any. Since you covered that in caret, are you considering to include it as another possible step in a model workflow ?.

Also, it would be fantastic if you could include some kind of suggestion to calibrate multi-class models, in the same was as you already covered performance metrics for this kind of models.

Thanks again!
Carlos Ortega

[topepo]

We don't have anything implemented yet but this is on our roadmap. If you have any examples or references that would be helpful, please send them along.

[coforfe]

Hi Max,

Thanks!.
Yes, these are some of the reference I usually use:

In the R's ecosystem:

• Your own entry from your book/blog "Applied Machine Learning": http://appliedpredictivemodeling.com/blog/2014/1/4/1vhzbkjxf1bt757xymsiucqmunj3m4
• There is another R package that covers this issue, there are not many (MachineShop): https://cran.r-project.org/web/packages/MachineShop/vignettes/UsersGuide.html#calibration-curves
• And also, H2O: https://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/calibrate_model.html
• And recently John Mount has blogged about this: https://win-vector.com/2020/10/28/an-example-of-a-calibrated-model-that-is-not-fully-calibrated/

In Python:

• Scikit-Learn has a method for this: https://scikit-learn.org/stable/modules/calibration.html
• With the same orientation as in your blog: https://machinelearningmastery.com/probability-calibration-for-imbalanced-classification/

And how this affects to DeepLearning models:

• https://medium.com/analytics-vidhya/more-about-model-calibration-353dafff9f76
• https://arxiv.org/abs/1706.04599

But for multi-class calibration, I have only seen this very good reference:

• https://jmetzen.github.io/2015-04-14/calibration.html

Finally, some papers about how to calibrate multi-class models:

• https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.7457&rep=rep1&type=pdf

[topepo]

Thanks! I'm going to transfer this to the probably repo. That's where these function will eventually reside.

[topepo]

Calibration is in the latest version of probably. Here is an article about some of them.

[coforfe]

Thanks a lot Max (and your team) for all of this!.

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.