Take hyperparameter ranges from components #516

jeremyliweishih · 2020-03-20T19:34:19Z

Fixes #359.

codecov · 2020-03-20T19:42:04Z

Codecov Report

Merging #516 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #516   +/-   ##
=======================================
  Coverage   98.68%   98.69%           
=======================================
  Files         114      114           
  Lines        4026     4056   +30     
=======================================
+ Hits         3973     4003   +30     
  Misses         53       53

Impacted Files	Coverage Δ
...ml/pipelines/classification/logistic_regression.py	`100.00% <ø> (ø)`
evalml/pipelines/classification/random_forest.py	`100.00% <ø> (ø)`
evalml/pipelines/classification/xgboost.py	`100.00% <ø> (ø)`
evalml/pipelines/regression/linear_regression.py	`100.00% <ø> (ø)`
evalml/pipelines/regression/random_forest.py	`100.00% <ø> (ø)`
evalml/pipelines/classification/catboost.py	`100.00% <100.00%> (ø)`
evalml/pipelines/pipeline_base.py	`98.52% <100.00%> (+0.07%)`	⬆️
evalml/pipelines/regression/catboost.py	`100.00% <100.00%> (ø)`
evalml/tests/conftest.py	`100.00% <100.00%> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`99.62% <100.00%> (+0.04%)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ba98f3...c87e4ba. Read the comment docs.

dsherry · 2020-03-24T14:11:49Z

evalml/pipelines/pipeline_base.py

@@ -32,6 +32,8 @@ def component_graph(cls):
    def problem_types(cls):
        return NotImplementedError("This pipeline must have `problem_types` as a class variable.")

+    _hyperparameters = None


@jeremyliweishih why the underscore here? I see you included an example in the custom pipelines doc of how to override hyperparams. If we support that, we should make this public, right?

If we include PipelineBase.hyperparameters as a class variable we can't name this field hyperparameters as well as they share the same namespace. If it makes more sense I can make this hyperparameters and PipelineBase._hyperparameters. I went with this as it was along the same setup as _name as a class variable and name as a property.

Oh! I had forgotten this is what we did for name. Cool, makes sense to me.

I bet there's a way to wrangle this using python magic methods/decorators. But it's not worth it. What you have here is great 👍

dsherry · 2020-03-24T14:12:03Z

evalml/pipelines/classification/random_forest.py

-        "max_depth": Integer(1, 32),
-        "impute_strategy": ["mean", "median", "most_frequent"],
-        "percent_features": Real(.01, 1)
-    }


dsherry · 2020-03-24T14:13:24Z

evalml/pipelines/pipeline_base.py

        return handle_component(cls.component_graph[-1]).model_family

+    @classproperty
+    def hyperparameters(cls):
+        "Returns hyperparameter ranges from components"


Can you please add a quick description of the return structure, specifically that it returns hyperparameters as a flat list across all components?

dsherry · 2020-03-24T14:13:52Z

evalml/pipelines/pipeline_base.py

+            hyperparameter_ranges.update(component.hyperparameter_ranges)
+
+        if cls._hyperparameters:
+            hyperparameter_ranges.update(cls._hyperparameters)


So this is how we support pipeline-level overriding?

Yeah this was the easiest way I thought of doing it.

dsherry · 2020-03-24T14:14:46Z

evalml/pipelines/regression/catboost.py

        "impute_strategy": ["most_frequent"],
-        "n_estimators": Integer(10, 1000),
-        "eta": Real(0, 1),
-        "max_depth": Integer(1, 8),


Oh great. So after your changes, this pipeline inherits its hyperparams from its components, but also fixes impute_strategy to "most_frequent", because that's all that's supported by catboost. Right?

yeah thats how the overriding works!

can you explain why the other settings for impute_strategy don't matter here? I would think you still want to explore the other options for the numeric features handled by simple imputer

CatBoost handles categorical encoding natively and since One Hot Encoder is not a component of this pipeline, Simple Imputer needs to handle categorical data as well.

if we set the simple imputer like this it is also using most_frequent for the numeric columns as well. what if the user wanted to also search for imputing with mean?

based on the catboost documentation, it appears catboost can handle nan's as well, at least for numeric

https://catboost.ai/docs/concepts/algorithm-missing-values-processing.html

I can put up an issue for this and discuss more there! This PR doesn't actually alter how the hyperparameters are set for the catboost pipelines.

ya, let's do that. sorry for the tangent. thanks

I filed #519. It was great you brought it up @kmax12

evalml/tests/pipeline_tests/test_pipelines.py

dsherry · 2020-03-24T14:23:28Z

evalml/tests/pipeline_tests/test_pipelines.py

+    }
+
+    assert MockPipeline.hyperparameters == hyperparameters
+    assert MockPipeline(parameters={}, objective='precision').hyperparameters == hyperparameters


Suggestion for more tests:

Create a pipeline which just uses one component, which has no hyperparameters. Make sure the hyperparams list comes back empty with no error

Try setting invalid hyperparam values. This may be outside the scope of this PR... but it would be nice to have hyperparameter validation somewhere... although it's probably best if that's a separate PR, so that's ok

What happens if you set "impute_strategy": None in the pipeline hyperparams? I'd hope that would mean the tuner would ignore that parameter. We should decide what that should do

Try setting invalid hyperparam values. This may be outside the scope of this PR... but it would be nice to have hyperparameter validation somewhere... although it's probably best if that's a separate PR, so that's ok

I'll add an issue to validate overridden hyperparameters.

What happens if you set "impute_strategy": None in the pipeline hyperparams? I'd hope that would mean the tuner would ignore that parameter. We should decide what that should do

When testing this skopt gives me a ValueError: Dimension has to be a list or tuple.. Should we handle this case in validation as well?

Yeah, good call, can add to #518 . Maybe we need code which ignores args set to None. We can discuss elsewhere

dsherry

LGTM! Left some testing suggestions. Also started a discussion of whether _hyperparameters should be marked private or not.

evalml/pipelines/regression/catboost.py

jeremyliweishih · 2020-03-27T16:19:10Z

evalml/tuners/grid_search_tuner.py

@@ -31,10 +31,10 @@ def __init__(self, space, n_points=10, random_state=None):
        raw_dimensions = list()
        for dimension in space:
            # Categorical dimension
-            if isinstance(dimension, list) and all(isinstance(s, (str, bool)) for s in dimension):
+            if isinstance(dimension, list):


@dsherry @christopherbunn

I put this patch in because RFClassifierSelectFromModel and RFRegressorSelectFromModel includes a list with a string and a number as its values: "threshold": ['mean', -np.inf]. I like this change as we should standardize using Real and Integer for ranges and lists for specific values. Let me know what you guys think.

Makes sense. So our "categorical" hyperparameter option can support not only string values, but also numbers or even python objects. That's cool.

I have two suggestions:

Add unit test coverage for this type of input, for all our tuners

Where should we document this? Perhaps we can add this to the list for Documentation for Component and Pipeline API #385 ? This should go into a guide we write about how to define custom pipelines.

…s_359_hyper

dsherry

Approved, pending a) update to _hyperparameters name, and b) unit test coverage of the GridSearch patch. Good stuff!

…s_359_hyper

jeremyliweishih added 3 commits March 20, 2020 14:23

Add hyperparameter classproperty

3c73af2

Remove hyperparemeters, add overriding feature

7afbb65

Add overriding to docs

1da2fbe

jeremyliweishih changed the title ~~Take hyperparameter ranges from components~~ [WIP] Take hyperparameter ranges from components Mar 20, 2020

changelog

a67df5b

jeremyliweishih self-assigned this Mar 20, 2020

Add testcase for hyperpameters

24e75c0

jeremyliweishih changed the title ~~[WIP] Take hyperparameter ranges from components~~ Take hyperparameter ranges from components Mar 20, 2020

jeremyliweishih requested a review from dsherry March 20, 2020 20:55

jeremyliweishih mentioned this pull request Mar 24, 2020

Add supported_problem_types as required field for Pipelines and Components #515

Merged

dsherry reviewed Mar 24, 2020

View reviewed changes

evalml/tests/pipeline_tests/test_pipelines.py Show resolved Hide resolved

dsherry reviewed Mar 24, 2020

View reviewed changes

dsherry approved these changes Mar 24, 2020

View reviewed changes

jeremyliweishih added 3 commits March 24, 2020 10:51

Update comment

3902254

Separate tests'

61964c5

Add no hyperparams test'

46f569c

kmax12 reviewed Mar 24, 2020

View reviewed changes

evalml/pipelines/regression/catboost.py Outdated Show resolved Hide resolved

jeremyliweishih added 2 commits March 27, 2020 10:52

Merge branch 'master' into js_359_hyper

2da2b43

Make gridsearch take list as only values accepted

6dc83c8

jeremyliweishih commented Mar 27, 2020

View reviewed changes

jeremyliweishih added 2 commits March 27, 2020 16:33

Merge branch 'master' of https://github.com/FeatureLabs/evalml into j…

ea321f4

…s_359_hyper

Fix tests'

7127365

dsherry approved these changes Mar 27, 2020

View reviewed changes

Merge branch 'master' of https://github.com/FeatureLabs/evalml into j…

a7aee44

…s_359_hyper

jeremyliweishih added 5 commits March 30, 2020 10:31

Change _hyperparameters to custom_hyperparameters

ec7db26

Add testcases for mixed dtypes list for tuners

c037036

Fix indexing:

945e104

Add custom_hyperparameters to docs since vscode doesn't like to save

95308fb

Revert SPT

c87e4ba

jeremyliweishih merged commit d9cd7e0 into master Mar 30, 2020

dsherry mentioned this pull request Mar 30, 2020

Lots of warnings printing in docs build job #539

Closed

angela97lin mentioned this pull request Apr 1, 2020

Make release for v0.8.0 #565

Merged

dsherry mentioned this pull request Apr 14, 2020

Rename PipelineBase._name to custom_name #533

Closed

dsherry deleted the js_359_hyper branch October 29, 2020 23:49

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Take hyperparameter ranges from components #516

Take hyperparameter ranges from components #516

jeremyliweishih commented Mar 20, 2020

codecov bot commented Mar 20, 2020 •

edited

Loading

dsherry Mar 24, 2020

jeremyliweishih Mar 24, 2020

dsherry Mar 24, 2020

dsherry Mar 24, 2020

dsherry Mar 24, 2020

dsherry Mar 24, 2020

jeremyliweishih Mar 24, 2020

dsherry Mar 24, 2020

jeremyliweishih Mar 24, 2020

kmax12 Mar 24, 2020

jeremyliweishih Mar 24, 2020

kmax12 Mar 24, 2020

kmax12 Mar 24, 2020

jeremyliweishih Mar 24, 2020

kmax12 Mar 24, 2020

jeremyliweishih Mar 24, 2020

dsherry Mar 24, 2020

jeremyliweishih Mar 24, 2020

dsherry Mar 27, 2020 •

edited

Loading

dsherry left a comment

jeremyliweishih Mar 27, 2020

dsherry Mar 27, 2020

dsherry left a comment

Take hyperparameter ranges from components #516

Take hyperparameter ranges from components #516

Conversation

jeremyliweishih commented Mar 20, 2020

codecov bot commented Mar 20, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry Mar 27, 2020 • edited Loading

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 20, 2020 •

edited

Loading

dsherry Mar 27, 2020 •

edited

Loading