Replace allowed_pipelines with allowed_component_graphs #2364

ParthivNaresh · 2021-06-10T05:23:19Z

Here are the docs

Here's an example of the new implementation:

from pprint import pp

from evalml.automl import AutoMLSearch
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=100, n_features=20, n_informative=2, n_redundant=2, random_state=0
    )

allowed_component_graph = {
        "Logistic Regression Binary Pipeline": [
            "Imputer",
            "One Hot Encoder",
            "Standard Scaler",
            "Logistic Regression Classifier",
        ]
    }
automl = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    max_iterations=3,
    allowed_component_graphs=[allowed_component_graph],
    ensembling=True,
)
automl.search()

for pipe in automl.full_rankings['parameters']:
    pp(pipe)

================= OUTPUT ==================
{'Imputer': {'categorical_impute_strategy': 'most_frequent',
             'numeric_impute_strategy': 'mean',
             'categorical_fill_value': None,
             'numeric_fill_value': None},
 'One Hot Encoder': {'top_n': 10,
                     'features_to_encode': None,
                     'categories': None,
                     'drop': 'if_binary',
                     'handle_unknown': 'ignore',
                     'handle_missing': 'error'},
 'Logistic Regression Classifier': {'penalty': 'l2',
                                    'C': 1.0,
                                    'n_jobs': -1,
                                    'multi_class': 'auto',
                                    'solver': 'lbfgs'}}
{'Imputer': {'categorical_impute_strategy': 'most_frequent',
             'numeric_impute_strategy': 'most_frequent',
             'categorical_fill_value': None,
             'numeric_fill_value': None},
 'One Hot Encoder': {'top_n': 10,
                     'features_to_encode': None,
                     'categories': None,
                     'drop': 'if_binary',
                     'handle_unknown': 'ignore',
                     'handle_missing': 'error'},
 'Logistic Regression Classifier': {'penalty': 'l2',
                                    'C': 8.474044870453413,
                                    'n_jobs': -1,
                                    'multi_class': 'auto',
                                    'solver': 'lbfgs'}}
{'Baseline Classifier': {'strategy': 'mode'}}

…yperparameter ranges

codecov · 2021-06-10T16:07:03Z

Codecov Report

Merging #2364 (81a0067) into main (8c51031) will decrease coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2364     +/-   ##
=======================================
- Coverage   99.7%   99.7%   -0.0%     
=======================================
  Files        281     281             
  Lines      25014   25057     +43     
=======================================
+ Hits       24917   24957     +40     
- Misses        97     100      +3

Impacted Files	Coverage Δ
evalml/tests/automl_tests/test_engine_base.py	`100.0% <ø> (ø)`
...lml/automl/automl_algorithm/iterative_algorithm.py	`99.2% <100.0%> (-<0.1%)`	⬇️
evalml/automl/automl_search.py	`99.4% <100.0%> (-<0.1%)`	⬇️
evalml/automl/utils.py	`100.0% <100.0%> (ø)`
...valml/pipelines/time_series_regression_pipeline.py	`100.0% <100.0%> (ø)`
.../tests/automl_tests/dask_tests/test_automl_dask.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`99.8% <100.0%> (+0.2%)`	⬆️
.../automl_tests/test_automl_search_classification.py	`100.0% <100.0%> (ø)`
...ests/automl_tests/test_automl_search_regression.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl_utils.py	`100.0% <100.0%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c51031...81a0067. Read the comment docs.

bchen1116

Didn't yet finish, but left a few questions that would help me understand! Looking good so far!

evalml/automl/automl_algorithm/iterative_algorithm.py

evalml/automl/automl_search.py

docs/source/user_guide/automl.ipynb

evalml/automl/utils.py

bchen1116 · 2021-06-11T21:01:13Z

evalml/tests/automl_tests/test_automl.py

            optimize_thresholds=False,
            n_jobs=1,
        )
+    automl._automl_algorithm = IterativeAlgorithm(


Did you need to define this since we're patching IterativeAlgo? Why did we not need to do it before?

evalml/tests/automl_tests/test_automl.py

bchen1116 · 2021-06-11T21:07:10Z

evalml/tests/automl_tests/test_automl.py

        objective="log loss binary",
        additional_objectives=["f1"],
    )
+    automl._automl_algorithm = IterativeAlgorithm(


Why is this needed?

Because the DummyPipeline score is being mocked for the purpose of this test (which is something ComponentGraph doesn't have), I have to specify that pipeline being passed into the IterativeAlgorithm since we can't pass it through AutoMLSearch anymore. This is the same reason it was added for the other tests as well! Luckily it doesn't look like this increases the execution time

freddyaboulton

@ParthivNaresh Looks good! I have some minor comments I'd like to discuss before merge though.

evalml/automl/utils.py

docs/source/user_guide/automl.ipynb

evalml/automl/automl_search.py

freddyaboulton · 2021-06-14T19:17:30Z

evalml/automl/automl_search.py

+                    )
+            unique_names = set()
+            for graph in allowed_component_graphs:
+                unique_names.add(list(graph.keys())[0])


I'm wondering if we should change the expected type of allowed_component_graphs from a list of dicts to just a dict.

So from

allowed_component_graphs=[ {"Name_0": [dummy_classifier_estimator_class]}, {"Name_1": [dummy_classifier_estimator_class]}, {"Name_2": [dummy_classifier_estimator_class]}, ],

to

allowed_component_graphs={ "Name_0": [dummy_classifier_estimator_class], "Name_1": [dummy_classifier_estimator_class], "Name_2": [dummy_classifier_estimator_class] },

I see there's some duplicated code in this pr to unpack the list, e.g list(graph.keys())[0] and next(iter(component_graph)) making this change may make things clearer/reduce code?

I'll go a step further...is there any reason to not just expect a list or dictionary of ComponentGraph objects?

Originally the format was set as such to support random_seed but since we're not passing that anymore, this would definitely be a better alternative.

evalml/automl/automl_algorithm/iterative_algorithm.py

freddyaboulton · 2021-06-14T19:43:45Z

evalml/tests/automl_tests/test_automl.py

-                "mean",
-            }
+        if row["pipeline_name"] == "Name_linear":
+            assert row["parameters"]["Imputer"]["numeric_impute_strategy"] == "mean"


What's causing this diff? Is it the deletion of _frozen_pipeline_parameters?

Yes I think so, since the custom_hyperparameter is set to a constant value, all pipelines will return mean for the Imputer

evalml/tests/conftest.py

chukarsten

Wow, that was a lot of work. Good job. I think we should discuss whether we want to preserve the two different ways of passing in the component graphs that are allowed. To me, it would seem the way to do it would be to reduce the complexity and only pass in a dict of component graphs (if they need to be named). The handfulls of instances where IterativeAlgorithm is set in the tests is also weird, but I don't think these are blocking to me.

evalml/automl/automl_algorithm/iterative_algorithm.py

evalml/automl/automl_search.py

chukarsten · 2021-06-14T22:02:46Z

evalml/automl/automl_search.py

+                    )
+            unique_names = set()
+            for graph in allowed_component_graphs:
+                unique_names.add(list(graph.keys())[0])


I'll go a step further...is there any reason to not just expect a list or dictionary of ComponentGraph objects?

chukarsten · 2021-06-15T15:16:37Z

evalml/tests/automl_tests/test_automl_dask.py

+        automl._automl_algorithm = IterativeAlgorithm(
+            max_iterations=4,
+            allowed_pipelines=pipelines,
+            tuner_class=SKOptTuner,
+            random_seed=0,
+            n_jobs=-1,
+            number_features=X.shape[1],
+            pipelines_per_batch=5,
+            ensembling=False,
+            text_in_ensembling=False,
+            pipeline_params={},
+            custom_hyperparameters=None,
+        )


All these instances of having to set the IterativeAlgorithm are kinda weird.

# Conflicts: # docs/source/release_notes.rst # evalml/tests/automl_tests/test_automl.py # evalml/tests/automl_tests/test_automl_search_classification.py # evalml/tests/automl_tests/test_engine_base.py

…ub.com/alteryx/evalml into Replace-AllowedPipelines-ComponentGraphs

ParthivNaresh added 30 commits May 20, 2021 14:07

add custom_hyperparameters

1f4d7c2

comment

33df646

print

cc2bb32

print

0ea03de

print

d3323e0

print statements

a45a187

remove custom hyperparameters from pipelines

8b31f5c

more changes

f2a2917

print update

e749141

Set component_parameters to include passed pipeline parameters

358532c

prints

d87aeb3

print

38559e8

update get_hyperparameter_ranges

416af96

print

c4c3d98

prevent batch 0 parameter selection from being out of bounds of the h…

8f8dad8

…yperparameter ranges

api updates

4b93ea6

test updates

0e968e5

add changes to pipelines passed in

27f048d

practice tests

7bcec95

test updates

a62361c

print update

4937680

print update

4b28e59

pass frozen parameters

98b4870

upadte tests

1698b9b

test update

8f7be8c

test update

da01d26

test update

8c84504

print statements

21cc485

test update

4adaef7

test updates

b7aae18

ParthivNaresh requested review from bchen1116, chukarsten and jeremyliweishih June 10, 2021 15:46

ParthivNaresh added 4 commits June 10, 2021 12:21

docs update

85c296e

conf test clean up

f4bd833

test update

5fd09c4

lint and doc fix

e0d54e5

bchen1116 reviewed Jun 11, 2021

View reviewed changes

ParthivNaresh added 5 commits June 14, 2021 17:59

test updates

5a3d5a3

add component graph check

a2aa4b0

lint fix

9cef3a5

lint fix

a4450a1

lint fix

ab05f3f

freddyaboulton approved these changes Jun 14, 2021

View reviewed changes

chukarsten approved these changes Jun 15, 2021

View reviewed changes

ParthivNaresh added 11 commits June 17, 2021 14:38

Merge branch 'main' into Replace-AllowedPipelines-ComponentGraphs

aa9ff36

# Conflicts: # docs/source/release_notes.rst # evalml/tests/automl_tests/test_automl.py # evalml/tests/automl_tests/test_automl_search_classification.py # evalml/tests/automl_tests/test_engine_base.py

Update allowed_component_graphs format, update tests

969dffa

lint updates

2e18c65

docs update

352f8d0

lint fixes and updates

da5bd00

Merge branch 'main' into Replace-AllowedPipelines-ComponentGraphs

3674fdb

test fix

a9e22f6

Merge branch 'Replace-AllowedPipelines-ComponentGraphs' of ssh://gith…

99447d5

…ub.com/alteryx/evalml into Replace-AllowedPipelines-ComponentGraphs

remove unused fixtures

2422aaf

docstring fix

86e788c

Merge branch 'main' into Replace-AllowedPipelines-ComponentGraphs

81a0067

ParthivNaresh merged commit cbc2799 into main Jun 18, 2021

chukarsten mentioned this pull request Jun 22, 2021

Release v0.27.0 #2428

Merged

freddyaboulton deleted the Replace-AllowedPipelines-ComponentGraphs branch May 13, 2022 15:35

Replace allowed_pipelines with allowed_component_graphs #2364

Replace allowed_pipelines with allowed_component_graphs #2364

Uh oh!

Conversation

ParthivNaresh commented Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh Jun 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh Jun 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ParthivNaresh commented Jun 10, 2021 •

edited

Loading

codecov bot commented Jun 10, 2021 •

edited

Loading

ParthivNaresh Jun 14, 2021 •

edited

Loading

ParthivNaresh Jun 17, 2021 •

edited

Loading