Just fitting the meta classifier in the StackingClassifier / Freezing some pre-trained models

[kingychiu]

Is there any way to freeze some models? It is useful to freeze some pre-trained model. For example, training each model on different machines or training serval models and comparing the performance of different base model combinations.

[rasbt]

Hi there,

there's no mlxtend-specific way to save/freeze pre-trained models. However, afaik, the typical scikit-learn pickle approach should work for most: http://scikit-learn.org/stable/modules/model_persistence.html

It would be nice to some solution that allows saving a model state in non-binary form. Some time ago, I had planned to build some utility for serializing scikit-learn estimators via JSON. E.g. see here https://cmry.github.io/notes/serialize . This still hovers around on my "someday" to-do list, but I don't think I will be able to get to this in near future. I would welcome PRs though if you like to build something like this -- actually, building it is not the challenge, I guess, but thoroughly checking and verifying it :)

[kingychiu]

Thanks for your reply and sharing your other cool projects! What I faced was every base model takes more than 10 hours to train and they were trained on different machines. I saved them with the sklearn.externals.joblib package. Then I can train the meta-classifier without retraining those base models. I have already achieved it (the current StackingClassifier API doesn't support training meta_clf only).

I am kind of interested in your model to JSON project, but from my previous experience loading a JSON object is really memory intensive (RAM used >>> the actual file size). I am not sure how large can a saved model be.

[rasbt]

Yeah, I think allowing this sort of behavior (fitting the meta-classifier on the existing models but not refitting the base models) would make sense for the StackingClassifier and the StackingRegressor (not for the CV methods, though, because of the cross-val).

It would also be super easy to implement. I actually renamed the old refit parameters to use_clones for all Stacking classes in version 0.12 because refit was misleading (I just see that somehow the online documentation didn't update it properly for some reason). refit could be used in this case now -- hoping that it doesn't cause any confusion because of its "old" use

I am kind of interested in your model to JSON project, but from my previous experience loading a JSON object is really memory intensive (RAM used >>> the actual file size). I am not sure how large can a saved model be.

Yeah, that would be a limitation. But theoretically, it should not occupy much more memory than the loaded model itself, I think. Regarding the loading process, yeah, I think the default JSON reader is not the most efficient one. I remember a benchmark article from a few years ago. I think there were pretty good alternatives though that could then be chosen as a backend, for example.

[kingychiu]

Thanks, @rasbt , sorry for the late reply. I read into the code I found the base learner will be trained even the use_clones is Flase.

In stacking_classification.py
`

    for clf in self.clfs_:
        if self.verbose > 0:
            i = self.clfs_.index(clf) + 1
            print("Fitting classifier%d: %s (%d/%d)" %
                  (i, _name_estimators((clf,))[0][0], i, len(self.clfs_)))

        if self.verbose > 2:
            if hasattr(clf, 'verbose'):
                clf.set_params(verbose=self.verbose - 2)

        if self.verbose > 1:
            print(_name_estimators((clf,))[0][1])

        clf.fit(X, y)

`

I achevied what I want by commenting the code shown above.

[rasbt]

Thanks, @rasbt , sorry for the late reply. I read into the code I found the base learner will be trained even the use_clones is Flase.

Thanks for the feedback. I think that's not correct, because of the following two lines that come before the lines you referenced:

    if self.use_clones:
        self.clfs_ = [clone(clf) for clf in self.classifiers]

So, it will fit clones of the base learners but not the baselearners themselves.

To allow what you had initially in mind, I think the best way would be adding a refit_base_estimators parameter to allow using the original estimators without refitting.