Add universal error for predict/transform before fitting #969

jeremyliweishih · 2020-07-23T19:55:54Z

Fixes #851

Make new issues about pipelines/needs_fitting/mocking.

…851_error

codecov · 2020-07-24T18:30:09Z

Codecov Report

Merging #969 into main will increase coverage by 0.17%.
The diff coverage is 99.00%.

@@            Coverage Diff             @@
##             main     #969      +/-   ##
==========================================
+ Coverage   99.68%   99.85%   +0.17%     
==========================================
  Files         178      178              
  Lines        9163     9282     +119     
==========================================
+ Hits         9134     9269     +135     
+ Misses         29       13      -16

Impacted Files	Coverage Δ
...ents/estimators/classifiers/baseline_classifier.py	`100.00% <ø> (ø)`
...onents/estimators/regressors/baseline_regressor.py	`100.00% <ø> (ø)`
...components/transformers/encoders/onehot_encoder.py	`100.00% <ø> (ø)`
...components/transformers/imputers/simple_imputer.py	`100.00% <ø> (ø)`
.../transformers/preprocessing/datetime_featurizer.py	`100.00% <ø> (ø)`
...ts/transformers/preprocessing/drop_null_columns.py	`100.00% <ø> (ø)`
.../tests/component_tests/test_baseline_classifier.py	`100.00% <ø> (ø)`
...l/tests/component_tests/test_baseline_regressor.py	`100.00% <ø> (ø)`
.../tests/component_tests/test_datetime_featurizer.py	`100.00% <ø> (ø)`
...ponent_tests/test_drop_null_columns_transformer.py	`100.00% <ø> (ø)`
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b71cb22...4ca2d4d. Read the comment docs.

jeremyliweishih · 2020-07-24T19:20:24Z

@dsherry PR should be ready but there's this weird lint issue going on (where I'm not getting locally but failing on CircleCI).

jeremyliweishih · 2020-07-24T19:28:21Z

@dsherry I think we should give the same treatment for pipelines by metaclass wrappers for predict etc. as well but I'm pushing that out of the scope of this PR. However, as it currently stands if someone class predict before fit on a pipeline the error will surface from the component itself.

freddyaboulton

@jeremyliweishih This is so cool! Never seen metaclasses in action before. My only blocking comment is about whether there are better alternatives than defining a static NO_FITTING_REQUIRED list. I can see pros and cons so looking forward to hear what you think. Apart from that, nothing blocking.

evalml/pipelines/components/wrappers.py

evalml/tests/component_tests/test_datetime_featurization.py

evalml/pipelines/components/component_base.py

freddyaboulton · 2020-07-24T21:44:03Z

evalml/pipelines/components/wrappers.py

+
+from evalml.exceptions import UnfitComponentError
+
+NO_FITTING_REQUIRED = ['DropColumns', 'SelectColumns']


An alternative to this would be adding a _no_fitting_required flag to ComponentBase like you did with _has_fit. Might be nice to let developers turn the check_for_fit functionality off if they need to without having them modify this file. What do you think?

@freddyaboulton oh that's a neat idea. So like this?

class NonLearningComponent(Transformer): fitting_required = False ... # the call below won't error NonLearningComponent().predict(X)

I think that's cool. I think it would be good to do this in a separate PR since there's already a lot in here.

Yea that's what I was thinking and it makes sense to move it to another PR.

@freddyaboulton I like the idea! We used to have a field on components called needs_fitting and from what I recall we removed it and made the assumption that all components needs "fitting" even if fitting doesn't do anything. From the components I had to alter (Drop Columns etc.) it seems like there has been more components where it would be more clear to add a field back in denoting fitting.

I'll file another issue after this PR is merged to add that back in @dsherry!

evalml/tests/component_tests/test_components.py

evalml/pipelines/components/wrappers.py

dsherry

@jeremyliweishih wow, it's so cool that you got this working!! Especially for the properties. Very nice 🎉

I think we should keep all the metaclass code in the same file. I think its fine if that's component_base.py. I also think we should keep the wrapper-making functions as classmethods on the metaclass.

I left a few naming suggestions. I agree we should delete the old test coverage left over from individual components.

Will approve once we round off those conversations! I left other suggestions but nothing else which was blocking.

dsherry · 2020-07-24T22:08:14Z

evalml/tests/component_tests/test_components.py

        parameters = {'param_a': param_a, 'param_b': param_b}
        super().__init__(parameters=parameters,
                         component_obj=None,
                         random_state=0)

    def fit(self, X, y=None):
-        self.is_fitted = True
+        pass


I think this will cause issues with codecov. Try

def fit(self, X, y=None): """Docstring"""

I vaguely remember talking with @angela97lin a couple months back and realizing this would resolve a codecov issue we were seeing

But its weird because the exceptions use pass and are fine in codecov! Idk 🤷

Yeah... I remember using docstrings as a way to cover this case after reading this SO post: https://stackoverflow.com/questions/9202723/excluding-abstractproperties-from-coverage-reports

Not sure why exceptions are fine, but maybe a guess is the way python executes code? That is, when we raise an exception, what's within the class gets scanned? Idk, no idea 🤷‍♀️

evalml/exceptions/exceptions.py

evalml/pipelines/components/component_base.py

evalml/pipelines/components/wrappers.py

evalml/tests/component_tests/test_components.py

dsherry · 2020-07-24T22:43:22Z

evalml/tests/component_tests/test_components.py

+    trans.transform(X)
+
+
+def test_all_components_check_fit(X_y_binary):


@jeremyliweishih what's the runtime of this test? If its high, we could consider mocking the impl methods

Because we don't care what happens in the actual fit/transform/predict, just that the wrappers get called correctly

evalml/tests/component_tests/test_components.py

…851_error

angela97lin

👍 Cool stuff!!

Just left a blocking comment about updating tests :D

evalml/pipelines/components/transformers/preprocessing/datetime_featurizer.py

evalml/pipelines/components/component_base.py

evalml/tests/component_tests/test_components.py

evalml/pipelines/components/wrappers.py

…851_error

freddyaboulton

@jeremyliweishih This is awesome! Thanks for the hard work on this. My comments have been addressed. The only question I have is whether we need to check the X is None and y is None case in check_for_fit

freddyaboulton · 2020-07-27T20:13:37Z

evalml/pipelines/components/component_base.py

+            klass = type(self).__name__
+            if not self._is_fitted and klass not in cls.NO_FITTING_REQUIRED:
+                raise ComponentNotYetFittedError('This {klass} is not fitted yet. You must fit {klass} before calling {method.__name__}.')
+            elif X is None and y is None:


Do we need to check this case?

@freddyaboulton I consolidated the property version (that I had before) into one method so this is needed for properties.

Got it! Thanks for clarifying.

dsherry

LGTM! Nice going

jeremyliweishih added 4 commits July 23, 2020 14:21

Add for fit and predict

72ef686

Add transformer logic'

818d35b

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

4e4d59a

…851_error

fix component tests

b117020

jeremyliweishih changed the title ~~Js 851 error~~ Add universal error for predict/transform before fitting Jul 23, 2020

jeremyliweishih added 5 commits July 24, 2020 10:23

Unify errors'

84fcb4a

Add feature importances

1011f67

remove other checks

d900d07

fix tests'

d10edd7

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

ab97ecd

…851_error

jeremyliweishih added 2 commits July 24, 2020 14:42

Edit CL

2b71bb6

add iterating through all components test

f7da16c

add feature importance missing method check

62366d1

jeremyliweishih added 3 commits July 24, 2020 15:35

Fix new test

45d2a2a

lint

09d4fba

codecove

68b1183

jeremyliweishih marked this pull request as ready for review July 24, 2020 20:18

auto-assign bot assigned jeremyliweishih Jul 24, 2020

more codecov

9f87d8a

jeremyliweishih requested review from dsherry, angela97lin and freddyaboulton July 24, 2020 21:19

freddyaboulton suggested changes Jul 24, 2020

View reviewed changes

dsherry suggested changes Jul 24, 2020

View reviewed changes

jeremyliweishih added 3 commits July 27, 2020 10:05

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

2d07761

…851_error

fix tests

9a6a8c5

fix docs'

9cdb2da

jeremyliweishih added 3 commits July 27, 2020 10:45

Remove tests

37579cb

Fix freddy comments

7ec8066

Rename exceptions

0bae9ce

angela97lin suggested changes Jul 27, 2020

View reviewed changes

jeremyliweishih added 6 commits July 27, 2020 11:47

Move to component_base'

5f93a79

Add subsubclass test

5028c41

Add not fitting required test

5e08bf9

Rename to _is_fitted

4636787

Make error more descriptive

b3cb441

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

6d7f289

…851_error

jeremyliweishih requested review from dsherry, angela97lin and freddyaboulton July 27, 2020 19:42

freddyaboulton approved these changes Jul 27, 2020

View reviewed changes

Merge branch 'main' into js_851_error

4ca2d4d

jeremyliweishih merged commit ad5a695 into main Jul 28, 2020

dsherry reviewed Jul 28, 2020

View reviewed changes

angela97lin mentioned this pull request Jul 31, 2020

Release v0.12.0 #1007

Merged

dsherry deleted the js_851_error branch October 29, 2020 23:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add universal error for predict/transform before fitting #969

Add universal error for predict/transform before fitting #969

jeremyliweishih commented Jul 23, 2020 •

edited

Loading

codecov bot commented Jul 24, 2020 •

edited

Loading

jeremyliweishih commented Jul 24, 2020

jeremyliweishih commented Jul 24, 2020

freddyaboulton left a comment

freddyaboulton Jul 24, 2020

dsherry Jul 24, 2020

freddyaboulton Jul 27, 2020

jeremyliweishih Jul 27, 2020

dsherry left a comment

dsherry Jul 24, 2020

dsherry Jul 24, 2020

angela97lin Jul 25, 2020

dsherry Jul 24, 2020

dsherry Jul 24, 2020

angela97lin left a comment

freddyaboulton left a comment

freddyaboulton Jul 27, 2020

jeremyliweishih Jul 27, 2020

freddyaboulton Jul 27, 2020

dsherry left a comment


		from evalml.exceptions import UnfitComponentError

		NO_FITTING_REQUIRED = ['DropColumns', 'SelectColumns']

		trans.transform(X)


		def test_all_components_check_fit(X_y_binary):

Add universal error for predict/transform before fitting #969

Add universal error for predict/transform before fitting #969

Conversation

jeremyliweishih commented Jul 23, 2020 • edited Loading

codecov bot commented Jul 24, 2020 • edited Loading

Codecov Report

jeremyliweishih commented Jul 24, 2020

jeremyliweishih commented Jul 24, 2020

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

jeremyliweishih commented Jul 23, 2020 •

edited

Loading

codecov bot commented Jul 24, 2020 •

edited

Loading