[ENH] Allow list of hyperparameter options for tuning

[fraimondo]

This is from a scikit-learn example:

tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
                     'C': [1, 10, 100, 1000]},
                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]

clf = GridSearchCV(
    SVC(), tuned_parameters, scoring='%s_macro' % score
)

This example creates two grids, one for the rbf kernel, exploring gamma, and one for the linear kernel, without testing gamma, because it does not make sense.

In order to do this, we need to set the following model_params:

model_params = {
    'svm__C': [1, 10, 100, 1000],
    'svm__kernel': ['rbf', linear'],
    'svm__gamma': [1e-3, 1e-4]
}

However, we will be testing different gamma options for the linear kernel (useless).

We need to find a way of specifying sets of parameters to be tested.

[fraimondo]

What about checking if model_params is a list?

[fraimondo]

I've been thinking on this issue for quite some time now. My goal is to allow for "lists of dicts" as scikit-learn does, but with the current user-friendly API.

The solution I come up with, so far, is to repeat the name parameter.

For example, the following scikit-learn hyperparameters definition

model = Pipeline([("svm", SVC())])
model_params = [
    {
        'svm__C': [1, 10, 100, 1000],
        'svm__kernel': 'linear',
    },
    {
        'svm__C': [1, 10, 100, 1000],
        'svm__kernel': 'rbf',
        'svm__gamma': [1e-3, 1e-4]
    },

Could be created using julearn's PipelineCreator:

creator = PipelineCreator(problem_type="classification")
creator.add("svm", name="svm", C=[1, 10, 100, 1000], kernel="linear")
creator.add("svm", name="svm", C=[1, 10, 100, 1000], kernel="rbf", gamma=[1e-3, 1e-4])

This could also allow to tune different learning algorithms:

creator = PipelineCreator(problem_type="classification")
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

And even allow to do so at more than one step at the time:

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", name="preprocess")
creator.add("robustscaler", name="preprocess")
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

This should test combination of 2 different preprocessing with the 2 different learning algorithms.

However, one potential downside happens when more than one step has more than one hyperparameter set. For example, lets try two different preprocessing, one for the RF and one for the SVM

model = Pipeline([("preprocess", StandardScaler()), ("model", SVC())])
model_params = [
    {
        'preprocess': StandardScaler(),
        'model': SVC(),
        'model__C': [1, 10, 100, 1000],
        'model__kernel': 'linear',
    },
    {
        'preprocess': RobustScaler(),
        'model': RandomForestClassifier(),
        'model__n_estimators': 1000,
        'model__max_depth': [4, 5, 6, 7]
    },

Using the proposed method, this is not possible. In this special case, my solution would be (given that it is a completely deterministic approach), to allow to pick elements in the list of dicts that we want to try, as part of the search params parameter.

For example, this:

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", name="preprocess")
creator.add("robustscaler", name="preprocess")
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

would generate:

model = Pipeline([("preprocess", StandardScaler()), ("model", SVC())])
model_params = [
    {
        'preprocess': StandardScaler(),
        'model': SVC(),
        'model__C': [1, 10, 100, 1000],
        'model__kernel': 'linear',
    },
    {
        'preprocess': RobustScaler(),
        'model': SVC(),
        'model__C': [1, 10, 100, 1000],
        'model__kernel': 'linear',
    },
    {
        'preprocess': StandardScaler(),
        'model': RandomForestClassifier(),
        'model__n_estimators': 1000,
        'model__max_depth': [4, 5, 6, 7]
    },
    {
        'preprocess': RobustScaler(),
        'model': RandomForestClassifier(),
        'model__n_estimators': 1000,
        'model__max_depth': [4, 5, 6, 7]
    },

So we could do something like:

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", name="preprocess")
creator.add("robustscaler", name="preprocess")
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

search_params = {"picks": [0, 3], search="grid"}

which should create a GridSearchCV but constrain the list of dicts only to the first and last elements on the list.

@samihamdan @LeSasse any thoughts?

[LeSasse]

I quite like this idea. I think that this would cover at least my use cases well enough. Another idea I could imagine (but tell me if that makes sense) is to simply give a list of PipelineCreators to run_cross_validation. Then for each PipelineCreator a separate grid is built.

[fraimondo]

give a list of PipelineCreators to run_cross_validation. Then for each PipelineCreator a separate grid is built.

Something like this?

creator1 = PipelineCreator(problem_type="classification")
creator1.add("zscore", name="preprocess")
creator1.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

creator2 = PipelineCreator(problem_type="classification")
creator2.add("robustscaler", name="preprocess")
creator2.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])

and then model=[creator1, creator2]

I thought about this option too. There is a restriction though: all the pipeline creators must have the same step names in the same order. Because in scikit-learn, the pipeline is only one, it's the "parameters" that change (including the model). Can you, in scikit-learn, tune to use or not a preprocessing step?

Also, while this solution simplifies the "hyperparameters of multiple steps in pairs issue", it make simpler cases more complex.

For example, the "SVM RBF with gamma issue" would look like:

creator1 = PipelineCreator(problem_type="classification")
creator1.add("zscore", name="preprocess")
creator1.add("svm", name="svm", C=[1, 10, 100, 1000], kernel="linear")

creator2 = PipelineCreator(problem_type="classification")
creator2.add("zscore", name="preprocess")
creator2.add("svm", name="svm", C=[1, 10, 100, 1000], kernel="rbf", gamma=[1e-3, 1e-4])

And this will be even more complicated if more preprocessing steps are added, as the user will be forced to copy/paste the preprocessing.

But this is just my opinion, maybe there's something I'm not seeing.

[LeSasse]

tune to use or not a preprocessing step? I guess you can using the "passthrough" string. But for sure I can see a lot of code replication as an issue when building these type of models.

I think your proposed solution is better though, especially since those simple cases will make up most of the use cases.

[fraimondo]

I'll try to come up with an implementation to test this.

[samihamdan]

I like the first option but I am not 100% sure about the picks argument.
Maybe I misunderstand it, but what I would do to create max flexibility would be the following:
How about making it more explicit to what dict they should be added.
Lets say we add a param called tuning_set_level (tmp name, we will find a better one) to step. This defines what combinations belong together and makes accidental usage less probable. It can be either 'all' which is default or a number starting from 0. Everything with all can only have this one step with this one name. Else these are number of the dict they should be added to.
Your use-case would still work. Also this should prevent that people forget to change the name of a step and then have a step optional as an accident.

e.g. this

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", name="preprocess", tuning_set_level=0)
creator.add("robustscaler", name="preprocess",tuning_set_level=1)
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7], tuning_set_level=0)
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear", tuning_set_level=1)

This also means that I can add multiple preprocessing steps only for one model but not the other. If a parameter at one level is not specified its 'passthrough'

e.g. the following will only add a pca to the svm but not the rf

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", name="preprocess", tuning_set_level=0)
creator.add("robustscaler", name="preprocess",tuning_set_level=1)
creator.add("pca" ,tuning_set_level=1)
creator.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7], tuning_set_level=0)
creator.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear", tuning_set_level=1)

[fraimondo]

I see your point. Indeed the "picks" is not the clearest idea, but it still works.

The immediate problem that arises from your approach is that, as stated before, the pipeline steps (names) must be constant. That means, that all the steps must be defined for all "tuning levels".

Your first example is fine, but your second is not.

in this case, PCA is only defined for level 1. The default would be, as you say, to set it as "passthrough". For me, this creates both a complication in the internal logic as well as the creation of the pipeline.

First, because the pipeline will have as many steps added, with several passthrough. So then inspecting models and debugging issues will not be clear/easy to the user. I mean, there is no reason to set the name anymore, your second example could be written as:

creator = PipelineCreator(problem_type="classification")
creator.add("zscore", tuning_set_level=0)
creator.add("robustscaler", tuning_set_level=1)
creator.add("pca", tuning_set_level=1)
creator.add("rf",  n_estimators=1000, max_depth=[4, 5, 6, 7], tuning_set_level=0)
creator.add("svm", C=[1, 10, 100, 1000], kernel="linear", tuning_set_level=1)

Second, because it will not be straightforward to read these 6 lines of code and understanding what the model is (think not only of 2 levels, but 3, or more, with some steps present in more than one level).

In this sense, I do prefer @LeSasse version, in which two or more pipeline creators are passed:

creator1 = PipelineCreator(problem_type="classification")
creator1.add("zscore", name="preprocess")
creator1.add("rf", name="model", n_estimators=1000, max_depth=[4, 5, 6, 7])

creator2 = PipelineCreator(problem_type="classification")
creator2.add("robustscaler", name="preprocess")
creator2.add("pca" )
creator2.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

creator3 = PipelineCreator(problem_type="classification")
creator3.add("robustscaler", name="preprocess")
creator3.add("svm", name="model", C=[1, 10, 100, 1000], kernel="linear")

The internal logic is the same: build one common pipeline with a list of N dictionaries, setting "passthrough" to the non-specified steps, where N is the number of levels/creators passed.

They both still have the "name" problem with the model inspection, as the "joint pipeline" could have as many steps as the sum of the number of steps from all the pipelines.

One alternative could be to not set "passthrough" as default, but ask the user to specify pipelines in which all the named steps match and are defined (explicit "passthrough" instead of implicit).

I'm not married to any implementation, I'm just brainstorming on this issue.

We can, easily, support both implementations (@LeSasse and mine). How? Simply by splitting creators with repeated names into N creators and then following @LeSasse idea. These way it:
A) Allows for simple definition of parameter sets like in the RBF vs linear kernel issue (my idea)
B) Allows to split complex definitions into explicit sets (@LeSasse), while keeping code readability and model interpretation easy.

[fraimondo]

So I solved the issue:

1. Now the model parameter can be a list of pipeline creators.
2. The pipeline creator, when having the same explicit name on two or more steps will be considered as "for hyperparameter tunning".
3. The pipeline creator knows how to split themselves into several pipeline creators in case one or more steps have repeated names.
4. There is a merger component that knows how to merge scikit-learn Pipelines and (GridSearchCV or RandomizedSearchCV) objects into one searcher with a list of grids.

So now in the API, all the creators are split. Then each creator is converted into a pipeline. If there are more than 1 pipeline, then they are merged.

[codecov]

Codecov Report

Merging #47 (b6e93b2) into julearn_sk_pandas (19af973) will increase coverage by 0.69%.
The diff coverage is 98.21%.

@@                  Coverage Diff                  @@
##           julearn_sk_pandas      #47      +/-   ##
=====================================================
+ Coverage              92.75%   93.44%   +0.69%     
=====================================================
  Files                     53       55       +2     
  Lines                   2304     2579     +275     
  Branches                 424      474      +50     
=====================================================
+ Hits                    2137     2410     +273     
  Misses                    97       97              
- Partials                  70       72       +2     

Flag	Coverage Δ
docs	100.00% <ø> (ø)
julearn	93.44% <98.21%> (+0.69%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
julearn/pipeline/merger.py	94.44% <94.44%> (ø)
julearn/pipeline/pipeline_creator.py	91.15% <96.10%> (+1.62%)	⬆️
julearn/api.py	98.42% <100.00%> (+0.19%)	⬆️
julearn/pipeline/test/test_merger.py	100.00% <100.00%> (ø)
julearn/pipeline/test/test_pipeline_creator.py	100.00% <100.00%> (ø)
julearn/prepare.py	97.10% <100.00%> (+0.20%)	⬆️
julearn/transformers/dataframe/set_column_types.py	100.00% <100.00%> (ø)

[github-actions]

PR Preview Action v1.4.4
Preview removed because the pull request was closed.
2023-05-25 14:37 UTC

[samihamdan]

I like the approach and have nothing big to comment. As long as coverage is increased and this solves the todos I would approve this.