Support for state of art hyperparameter optimization packages

[oldrichsmejkal]

It would be nice to have options to select some state of art technique for hyperparameter optimization.
Such as:
https://scikit-optimize.github.io/stable/
https://github.com/optuna/optuna
or maybe the best (should be drop in replacement for scikit Grid/Random search, but support advanced techniques from packages above)
https://github.com/ray-project/tune-sklearn

[cerlymarco]

Hi, thanks for your feedback

I don't plan to add these as default addons in the library...

If you are looking for only parameters tuning, using only the cited hyperparameter optimization techniques/library may be optimal.

In parameters tuning + feature selection, the feature selection process is applied for each combination of parameters iteratively. In other words, for each parameter's config, we search for the best features and store the results. In the end, the best configuration is retrieved. The best option I consider is to use tune-sklearn with BoostRFE or BoostBoruta as estimator. To make this possible it seems I have to add for sure set_params/get_params methods to BoostRFE and BoostBoruta. I think I will do that and if the solution will be stable I'll create a new version of the library.

If you support the project don't forget to leave a star ;-)
Thanks again

[cerlymarco]

@oldrichsmejkal you can use tune-sklearn (and other sklearn objects) passing shap-hypetune estimators as wrapper. Below a full example:

from shaphypetune import BoostRFE
from lightgbm import LGBMClassifier
from tune_sklearn import TuneGridSearchCV

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split


from sklearn.utils.metaestimators import if_delegate_has_method
from sklearn.base import BaseEstimator

class BoostRFEWrap(BaseEstimator, BoostRFE):

  @if_delegate_has_method(delegate='estimator')
  def predict_proba(self, X):
    return self.predict(X, method='predict_proba')

model = BoostRFEWrap(LGBMClassifier(n_estimators= 50),
                     min_features_to_select=10, step=3)

parameters = {
            'estimator__learning_rate': [0.2, 0.1],
            'estimator__num_leaves': [25, 30, 35],
            }

tune_search = TuneGridSearchCV(
    model, parameters)
tune_search.fit(X_train, y_train)