-
Notifications
You must be signed in to change notification settings - Fork 155
Description
Is your feature request related to a problem? Please describe.
This is related to this issue and this is more of a proposal than an issue. ConformalPrediction.jl is a new package for uncertainty quantification through Conformal Prediction (CP) for machine learning models trained in MLJ.
Describe the solution you'd like
Conformal classifiers produce set-valued predictions. This is currently not something that is supported for downstream tasks by MLJ. To add conformal classifiers for general use, we would need LearnAPI.jl to support a ProbabilisticSet type (and maybe Set for deterministic classifiers). Of particular interest here is to implement proper evaluation schemes for these types of predictions. A good starting point to this end could be this tutorial from page 11 onwards.
Describe alternatives you've considered
Currently, conformal classifiers in ConformalPrediction.jl are simply subtypes of Supervised (though I might change this to Probabilisitic). They should be more or less ready for general use (except for evaluation, of course, and perhaps other downstream tasks I'm not familiar with).
As I won't get to this myself in the near future, I thought I'd open this up here for discussion. Any thought/comments/contribution very much welcome 🙏🏽
cc @ablaom