Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap models not machines? #5

Closed
ablaom opened this issue Oct 4, 2022 · 7 comments
Closed

Wrap models not machines? #5

ablaom opened this issue Oct 4, 2022 · 7 comments
Labels
enhancement New feature or request

Comments

@ablaom
Copy link

ablaom commented Oct 4, 2022

@pat-alt

Congratualations on the launch of this new package 🎉 Great to have the integration with MLJ!

I'm not familiar with conformal prediction, but I nevertheless wonder why this package wraps MLJ machines rather than models. If you wrap models, then you will buy into MLJ's model composition. So, a "conformally wrapped model" will behave like any other model: you can insert in pipeline, can wrap in tuning strategy, and so forth.

New models in MLJ generally implement the "basement level" model API. Machines are a higher level abstraction for: (i) user interaction; and (ii) syntax for building learning networks which are ultimately "exported" as standalone model types.

Here are other examples of model wrapping in MLJ: EnsembleModel (docs), BinaryThresholdPredictor, TunedModel, IteratedModel. What makes things a little complicated is the model hierarchy: the model supertype for the wrapped model depends on the supertype of the atomic model. So for example, we don't just have EnsembleModel we have DeterministicEnsembleModel (for ordinary point predictors) and ProbabilisticEnsembleModel (for probabilistic predictors) but the user only sees a single constructor EnsembleModel; see here. (A longer term goal is to drop the hierarchy in favour of pure trait interface, which will simplify things, but that's a little ways off yet.)

Happy to provide further guidance.

cc @azev77

@azev77
Copy link
Collaborator

azev77 commented Oct 4, 2022

@ablaom currently MLJ models can give deterministic & probabilistic predictions (GLM predicts an entire distribution).
If methods for conformal predictive inference become more developed, can users automatically combine their favorite (appropriate) deterministic predictor w/ their favorite predictive interval method, to make probabilistic predictions?

For example:
currently model = LGBMRegressor() gives deterministic predictions
would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

Note: at some point it would be great to compare these w/ predictions/Prediction Intervals from NGBoost.py etc

@ablaom
Copy link
Author

ablaom commented Oct 5, 2022

For example:
currently model = LGBMRegressor() gives deterministic predictions
would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

Yes, I guess that's the design I am suggesting. So, similar to the way BinaryThresholdPredictor wraps any probabilistic predictor and makes it deterministic.

@pat-alt
Copy link
Member

pat-alt commented Oct 7, 2022

Hi @ablaom 👋🏽 Thanks very much for this suggestion and sorry for the delayed response (been battling Covid this week while also trying to finish my JuliaCon proceedings submission 😅 ). I will implement this first thing once I turn back to working on this package some time next week 👍🏽

@pat-alt
Copy link
Member

pat-alt commented Oct 15, 2022

Implemented in #10, but will keep this open for now, because I still want to iron out a few things (some related questions here).

@pat-alt
Copy link
Member

pat-alt commented Oct 19, 2022

#18 should now be strictly in line with MLJ @ablaom, so I will close this. Will still need to figure out how to handle downstream tasks like evaluate in the future.

@pat-alt pat-alt closed this as completed Oct 19, 2022
@ablaom
Copy link
Author

ablaom commented Oct 19, 2022

Did you not need new abstract model subtype(s) at MLJModelInterface? For set-predictions (we already have Interval).

@pat-alt
Copy link
Member

pat-alt commented Oct 20, 2022

Yup, you're right. Have done that now in #20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

3 participants