Using a prefit model as a Transformer

[nxorable]

My goal is model stacking, where I wish to use an external, prefit model in an sklearn pipeline as an input without refitting it. I tried subclassing a TransformerMixin in order to use the predictions of this externally-fitted sklearn model (this model is related as it uses the same features, but it is trained on disjoint data). This external model's outputs would become inputs to my new estimator, the same way that we would utilize any unsupervised transformer. However, the sklearn API will reset any attached estimator to an unfitted state when cloning. Refitting is not an option, due to both fit time and the fact that the external model is trained on a disjoint dataset.

Is there a good way to accomplish this without giving up sklearn pipelines? Without using pipelines to encapsulate all transformations, including the upstream model's predictions, it's difficult to accomplish feature importance analyses.

Here's my attempt that doesn't work as intended - it still results in the (expensive) joblib reload on every clone function call - e.g. every CV iteration.

class UpstreamModelTransformer(TransformerMixin, BaseEstimator):
    
    def __init__(self, base_model):
        self.base_model = base_model
        # expensive load
        self._base_model_object = joblib.load(self.base_model)
        
    def fit(self, X, y=None):
        return self
    
    def transform(self, X, y=None):
        base_preds = self._base_model_object.predict(X)
        base_preds = np.expand_dims(base_preds, axis=1)
        return base_preds

[glemaitre]

This is a known limitation and I think that we are looking for a consensus to tackle the problem. You can have a look at the following PR: #8370 and the related issues and PRs. I think that we should make probably a SLEP in order to tackle this problem.

[nxorable]

Thank you, @glemaitre . Appreciate the great work on my favorite library.

[nxorable]

Note that if going the direction of the StaticTransformer route, performance of unserialization will be important:

recommending users serialise the models with transfer parameters, and advise the use of some kind of StaticTransformer which takes the serialised model (or a path thereto), decodes it when fitting and transforms using it.

Potentially large prefit models will be unserialized many times in a GridSearch, for instance