Clone Pipelines #842

eccabay · 2020-06-11T14:58:20Z

Closes #534

codecov · 2020-06-11T17:38:25Z

Codecov Report

Merging #842 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #842   +/-   ##
=======================================
  Coverage   99.69%   99.69%           
=======================================
  Files         195      195           
  Lines        7774     7846   +72     
=======================================
+ Hits         7750     7822   +72     
  Misses         24       24

Impacted Files	Coverage Δ
evalml/pipelines/components/component_base.py	`100.00% <100.00%> (ø)`
evalml/pipelines/pipeline_base.py	`100.00% <100.00%> (ø)`
evalml/tests/component_tests/test_components.py	`100.00% <100.00%> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`99.76% <100.00%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e8c859...119c6e1. Read the comment docs.

dsherry · 2020-06-11T19:34:52Z

evalml/pipelines/components/component_base.py

@@ -26,6 +28,22 @@ def name(cls):
    def model_family(cls):
        """Returns ModelFamily of this component"""

+    def clone(self, learned=True, random_state='match'):


Two thoughts:

Let's change learned=True to deep=False

I like that you thought about random_state here. Let's change this to be an int / RandomState with default 0, just like all other callsites in the repo

dsherry · 2020-06-11T19:35:11Z

evalml/pipelines/components/component_base.py

+        """
+        if learned:
+            return copy.deepcopy(self)
+        else:


Flattening: since you return on the previous line, no need for this else

dsherry · 2020-06-11T19:39:01Z

evalml/pipelines/components/component_base.py

+                cloned_component = component_class(parameters=self.parameters, random_state=self.random_state)
+            else:
+                cloned_component = component_class(parameters=self.parameters, random_state=random_state)
+            return cloned_component


Instead of adding parameters as an argument to every component, let's do

return self.__class__(**self.parameters, random_state=random_state)

This is how we instantiate components in PipelineBase.__init__.

evalml/pipelines/pipeline_base.py

dsherry

@eccabay looking good! You picked up a hard one... like I mentioned the other day, this touches on some real challenges with how our code is structured right now. It'll be great to have this merged.

I left comments about the impl. I see you have a test for each component and classifier -- wow! Will dig into that on the next pass.

Please revert the changes you made to the components, other than ComponentBase of course. We shouldn't add parameters as an input to the components. Not necessary when you can use python's ** syntax to pass the args along. See this explanation of how that works if you didn't know already.

dsherry · 2020-06-11T22:40:49Z

evalml/pipelines/components/estimators/classifiers/et_classifier.py

-                                               min_weight_fraction_leaf=min_weight_fraction_leaf,
+                      "max_depth": max_depth,
+                      "min_samples_split": min_samples_split,
+                      "min_weight_fraction_leaf": min_weight_fraction_leaf}


@jeremyliweishih pointed out you added some params here which were missing. That's great. We should keep changes like this, so that user-inputted values are always saved (except for random_state which we treat separately).

I started looking at #522 recently, which tracks fixing these sorts of issues. Heads up I may end up scooping these changes into my PR :)

Yeah, this and much more is now incorporated into #847

dsherry · 2020-06-15T16:22:00Z

evalml/pipelines/components/estimators/regressors/xgboost_regressor.py

+                                         eta=parameters['eta'],
+                                         max_depth=parameters['max_depth'],
+                                         n_estimators=parameters['n_estimators'],
+                                         min_child_weight=parameters['min_child_weight'])


@eccabay could you please back these changes out?

Why: if there's no reason to make them, we shouldn't make them :) perhaps this was left over from earlier?

dsherry · 2020-06-15T16:23:16Z

evalml/tests/component_tests/test_catboost_classifier.py

@@ -5,7 +5,7 @@
 from evalml.pipelines.components import CatBoostClassifier
 from evalml.utils import SEED_BOUNDS

-importorskip('catboost', reason='Skipping test because catboost not installed')
+catboost = importorskip('catboost', reason='Skipping test because catboost not installed')


Could you please back out this change?

dsherry · 2020-06-15T16:23:41Z

evalml/tests/component_tests/test_catboost_regressor.py

@@ -5,7 +5,16 @@
 from evalml.pipelines.components import CatBoostRegressor
 from evalml.utils import SEED_BOUNDS

-importorskip('catboost', reason='Skipping test because catboost not installed')
+catboost = importorskip('catboost', reason='Skipping test because catboost not installed')


Same for all changes in this file: please back them out since they're not necessary

dsherry · 2020-06-15T16:24:57Z

evalml/tests/pipeline_tests/classification_pipeline_tests/test_en_classification.py

@@ -40,6 +40,7 @@ def test_en_init(X_y):
        'Elastic Net Classifier': {
            "alpha": 0.5,
            "l1_ratio": 0.5,
+            "max_iter": 1000


Is this change important for this PR? Or is it left over from an earlier state of the PR?

Just an earlier state of the PR, will remove.

dsherry · 2020-06-15T16:25:05Z

evalml/tests/pipeline_tests/regression_pipeline_tests/test_xgboost_regression.py

@@ -9,7 +9,7 @@
 from evalml.pipelines.components import XGBoostRegressor
 from evalml.utils import get_random_seed, get_random_state, import_or_raise

-importorskip('xgboost', reason='Skipping test because xgboost not installed')
+xgboost = importorskip('xgboost', reason='Skipping test because xgboost not installed')


Same as others, please back this out

dsherry · 2020-06-15T16:26:33Z

evalml/tests/component_tests/test_components.py

+    clf_clone = clf.clone()
+    with pytest.raises(ValueError, match='Component is not fit'):
+        clf_clone.predict(X)
+    clf.parameters == clf_clone.parameters


I think you meant assert clf.parameters == clf_clone.parameters

dsherry · 2020-06-15T16:29:04Z

evalml/tests/component_tests/test_components.py

@@ -314,3 +314,42 @@ def test_component_parameters_all_saved():

        expected_parameters = {arg: default for (arg, default) in zip(args, defaults)}
        assert parameters == expected_parameters
+


@eccabay could you please add another test here which does the following:

def test_clone_init(): ... clf = MockFitComponent(**params) clf_clone = clf.clone() assert clf.parameters == clf_clone.parameters assert clf.random_state == clf_clone.random_state

You'll probably want to make MockFitComponent available via a test fixture in this file

Oh and I forgot to say: its valuable to add this because we need coverage of cloning a component which hasn't been fitted. Your main test test_clone covers the fitted case

I guess on reflection, you could just add assertions to your existing test. My key point was that we should have a test somewhere which checks the params and random state match before fitting. Your call on how to organize that.

dsherry · 2020-06-15T16:31:05Z

evalml/tests/component_tests/test_components.py

+    predicted = clf.predict(X)
+    assert isinstance(predicted, type(np.array([])))
+
+    clf_clone = clf.clone()


After the call to clone, let's also check assert clf_clone.random_state == clf.random_state

Helpful to have this to ensure that just because we're cloning a fitted pipeline doesn't mean the random_state won't get copied correctly

In order for the results to be consistent between the original component and its clone, the random_state is not copied from the original object, but is instead re-seeded with the same integer. Because of this, assert clf_clone.random_state == clf.random_state will actually fail if included in the tests, since they're different objects.

Ah, right, of course.

If I follow right, this wouldn't affect test_clone_init. And then in this unit test we could save the first value of the RandomState instance and use it to check the right seed was used in the clone:

clf = MockFitComponent(**params) random_state_first_value = clf.random_state.randint(2**30) ... clf.fit(...) ... clf_clone = clf.clone() assert clf_clone.random_state.randint(2**30) == random_state_first_value

dsherry · 2020-06-15T16:36:17Z

evalml/tests/component_tests/test_components.py

@@ -314,3 +314,42 @@ def test_component_parameters_all_saved():

        expected_parameters = {arg: default for (arg, default) in zip(args, defaults)}
        assert parameters == expected_parameters
+
+


@eccabay we also need test coverage for setting the random_state field, both with a) a non-default int and with b) a custom np.random.RandomState instance. We have some coverage in place for this at the automl-level I believe.

My suggestion: Define a separate test test_clone_random_state to do this. For the RandomState case, after cloning you can assert clf_clone.random_state.randint(2**30) == clf.random_state.randint(2**30) to ensure the RandomState are equivalent.

dsherry · 2020-06-15T16:38:16Z

evalml/tests/component_tests/test_components.py

@@ -314,3 +314,42 @@ def test_component_parameters_all_saved():

        expected_parameters = {arg: default for (arg, default) in zip(args, defaults)}
        assert parameters == expected_parameters
+
+
+def test_clone(X_y):


In light of adding the other tests, I suggest you call this test_clone_fitted since it focuses on cloning fitted components.

evalml/tests/component_tests/test_components.py

dsherry · 2020-06-15T16:39:54Z

evalml/tests/component_tests/test_components.py

+
+    clf.fit(X, y)
+    predicted = clf.predict(X)
+    assert isinstance(predicted, type(np.array([])))


I think you want assert isinstance(predicted, np.ndarray). But actually my suggestion is to not do this here. We have other tests which should do this. And I'm about to add more in my PR for #236

evalml/tests/component_tests/test_components.py

dsherry · 2020-06-15T16:43:31Z

evalml/tests/component_tests/test_components.py

+
+    clf_clone.fit(X, y)
+    predicted_clone = clf_clone.predict(X)
+    np.testing.assert_almost_equal(predicted, predicted_clone)


Awesome! Good thinking. We want clones to predict the same values when trained on the same data with the same random seed.

dsherry · 2020-06-15T16:50:04Z

evalml/tests/pipeline_tests/test_pipelines.py

@@ -658,3 +660,70 @@ class PipelineWithDropCol(BinaryClassificationPipeline):
    pipeline_with_drop_col.fit(X, y)
    pipeline_with_drop_col.score(X, y, ['auc'])
    assert list(pipeline_with_drop_col.feature_importances["feature"]) == ['other col']
+
+


I think many of my comments from test_components.py apply to this file too.

Is there any reason we need a separate test for each problem type? Unless I'm overlooking something here, I'm leaning towards no, and that we can just have one unit test. Why: pipeline cloning is implemented in PipelineBase, not in each pipeline subclass, and component cloning is also implemented in ComponentBase, not in each component subclass. So there should be no variation in behavior between the problem types.

dsherry

@eccabay this is super close! The impl looks great, no further comments there. I particularly like the component impl--so simple now! All my comments were on the tests. Let's get those resolved and we can get this merged!

dsherry · 2020-06-15T21:28:11Z

evalml/pipelines/pipeline_base.py

+        cloned_pipeline.component_graph = cloned_components
+        cloned_pipeline.estimator = cloned_components[-1] if isinstance(cloned_components[-1], Estimator) else None
+
+        return cloned_pipeline


One final thought here: I think what you have works, but that we should rely on the pipeline constructor to set up any necessary state:

def clone(self, random_state=0): return self.__class__(self.parameters, random_state=random_state)

Just like you did for ComponentBase.

Fine to do this in a separate PR if you prefer, since the impl you have passes the tests.

dsherry · 2020-06-15T21:30:22Z

evalml/tests/component_tests/test_components.py

+
+    clf = MockFitComponent(**params, random_state=np.random.RandomState(2))
+    clf_clone = clf.clone(random_state=np.random.RandomState(2))
+    assert clf_clone.random_state.randint(2**30) == clf.random_state.randint(2**30)


In one of the two code blocks in this test, did you mean to pass in an integer for random_state instead of a np.random.RandomState?

Oops, you're totally right 🤦

dsherry · 2020-06-15T21:32:17Z

evalml/tests/pipeline_tests/test_pipelines.py

+    assert pipeline_clone.random_state.randint(2**30) == pipeline.random_state.randint(2**30)
+
+    pipeline = LinearRegressionPipeline(parameters=parameters, random_state=np.random.RandomState(2))
+    pipeline_clone = pipeline.clone(random_state=np.random.RandomState(2))


Same comment as for components, should this be saying random_state=2 instead?

dsherry · 2020-06-15T21:33:06Z

evalml/tests/pipeline_tests/test_pipelines.py

+    with pytest.raises(RuntimeError):
+        pipeline_clone.predict(X)
+    pipeline_clone.fit(X, y)
+    X_t_clone = pipeline_clone.predict_proba(X)


Is there a reason to do predict_proba here instead of predict?

Leftover from when I was getting weird test failures with some of the pipelines, it gave me a more detailed look into what was going on.

dsherry

@eccabay I left a few more comments but I think this is ready to go! 🎊 In particular check out my suggestion on how to simplify PipelineBase.clone.

…s/evalml into 534_cloning_pipelines

eccabay added 15 commits June 4, 2020 08:48

Add empty-clone methods for sklearn classifier components

4adb3a5

Add empty-clone methods for sklearn regressor components

58a31c6

Add empty-clone methods for baseline components

1dc7a46

Add empty-clone methods for transformer components

f33f3da

Add pipeline empty-cloning (missing testing)

4ecb3d6

Add pipeline cloning tests (currently failing)

ea4f38b

Add xgboost and catboost empty-cloning (component and pipeline)

82832d0

Add learned cloning for components and pipelines (no tests)

c5e0a79

Add learned-cloning component tests

f5693f3

Add more pipeline cloning tests (still have failures)

2be61f7

Consolidate from two clone methods to one

097917e

Update components and tests to refelct reworked clone method

cfbcd02

Fix failing tests with random_seeds

b6fffac

Merge master into branch

ebbbf6c

Fix linting issues

c9ec784

eccabay marked this pull request as ready for review June 11, 2020 17:50

auto-assign bot assigned eccabay Jun 11, 2020

eccabay changed the title ~~534 cloning pipelines~~ Clone Pipelines Jun 11, 2020

eccabay requested review from dsherry and jeremyliweishih June 11, 2020 17:50

dsherry reviewed Jun 11, 2020

View reviewed changes

evalml/pipelines/pipeline_base.py Outdated Show resolved Hide resolved

dsherry suggested changes Jun 11, 2020

View reviewed changes

Address PR comments

d11ca58

dsherry reviewed Jun 11, 2020

View reviewed changes

dsherry mentioned this pull request Jun 12, 2020

Ability to clone components and pipelines #534

Closed

Merge master into branch for component parameters fix

6828175

dsherry reviewed Jun 15, 2020

View reviewed changes

evalml/tests/component_tests/test_components.py Show resolved Hide resolved

dsherry reviewed Jun 15, 2020

View reviewed changes

evalml/tests/component_tests/test_components.py Show resolved Hide resolved

dsherry reviewed Jun 15, 2020

View reviewed changes

dsherry suggested changes Jun 15, 2020

View reviewed changes

eccabay and others added 3 commits June 15, 2020 14:45

Refactor tests

46a5118

Update random_seed testing

34c6ac1

Merge branch 'master' into 534_cloning_pipelines

aca1b87

dsherry reviewed Jun 15, 2020

View reviewed changes

dsherry approved these changes Jun 15, 2020

View reviewed changes

eccabay added 3 commits June 16, 2020 08:27

Fix random_state tests and update pipeline impl

3c023cc

Merge branch '534_cloning_pipelines' of https://github.com/FeatureLab…

02ff841

…s/evalml into 534_cloning_pipelines

Remove commented out code

119c6e1

eccabay merged commit f6a9815 into master Jun 16, 2020

angela97lin mentioned this pull request Jun 30, 2020

Release v0.11.0 #901

Merged

dsherry deleted the 534_cloning_pipelines branch October 29, 2020 23:48

		@@ -314,3 +314,42 @@ def test_component_parameters_all_saved():

		expected_parameters = {arg: default for (arg, default) in zip(args, defaults)}
		assert parameters == expected_parameters

Clone Pipelines #842

Clone Pipelines #842

Conversation

eccabay commented Jun 11, 2020

codecov bot commented Jun 11, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

dsherry Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 11, 2020 •

edited

Loading

dsherry Jun 15, 2020 •

edited

Loading

dsherry Jun 15, 2020 •

edited

Loading

dsherry Jun 15, 2020 •

edited

Loading