Skip to content

Support for ProbabilisticSet type in MLJModelInterface.jl #978

@pat-alt

Description

@pat-alt

Is your feature request related to a problem? Please describe.
This is related to this issue and this is more of a proposal than an issue. ConformalPrediction.jl is a new package for uncertainty quantification through Conformal Prediction (CP) for machine learning models trained in MLJ.

Describe the solution you'd like
Conformal classifiers produce set-valued predictions. This is currently not something that is supported for downstream tasks by MLJ. To add conformal classifiers for general use, we would need LearnAPI.jl to support a ProbabilisticSet type (and maybe Set for deterministic classifiers). Of particular interest here is to implement proper evaluation schemes for these types of predictions. A good starting point to this end could be this tutorial from page 11 onwards.

Describe alternatives you've considered
Currently, conformal classifiers in ConformalPrediction.jl are simply subtypes of Supervised (though I might change this to Probabilisitic). They should be more or less ready for general use (except for evaluation, of course, and perhaps other downstream tasks I'm not familiar with).

As I won't get to this myself in the near future, I thought I'd open this up here for discussion. Any thought/comments/contribution very much welcome 🙏🏽

cc @ablaom

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions