Framework for Model Selection

[juliuskittler]

What I would you like to be added:

It would be great if NNI had an extension/framework for model selection.

Why is this needed:

Model selection is a very relevant decision for an ML project, arguably more important than hyperparameter tuning.

Without this feature, how does current nni work：

Currently, NNI seems to require you to decide on a model type in advance and then you optimise this model type. For example, you decide to use XGBoost and then you optimise hyperparameters. But you cannot compare / select from all the different possible model types that NNI supports in principle (e.g. models from scikit-learn, Keras, XGBoost etc.). In summary, it seems that NNI currently covers a little bit of feature selection and a lot of hyperparameter tuning; but it does not cover model selection at all.

I could not find any examples where NNI is used for model selection. If I missed something, I'd be thankful for your reply.

[QuanluZhang]

@juliuskittler thanks for your good suggestion. NNI allows user to do model selection by using hyper-parameter tuning. Here is a very simple example, and users can further leverage nested search space to express candidate models along with their own hyper-parameters.

I agree, a user-friendly interface and extension for model selection is indeed appealing. We will put it in our plan. We highly encourage design proposal and implementation from external contributors.

[juliuskittler]

@QuanluZhang Thank you for getting back. It seems to me that the example you shared covers only sklearn. What if I want to do model selection and hyperparameter tuning, allowing for models from sklearn, Keras, and LightGBM? If this is possible already, an example would be highly appreciated!