363 common method for registering components #898

freddyaboulton · 2020-06-30T14:32:22Z

Pull Request Description

Addresses #363 by creating a function called get_importable_subclasses for creating a list of all non-base-class subclasses of Transformer, Estimator, and PipelineBase. This function has a flag for filtering for estimators/pipelines used for automl search. It should be removed in #899.

After creating the pull request: in order to pass the changelog_updated check you will need to update the "Future Release" section of docs/source/changelog.rst to include this pull request by adding :pr:123.

codecov · 2020-06-30T14:35:22Z

Codecov Report

Merging #898 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #898      +/-   ##
==========================================
- Coverage   99.83%   99.83%   -0.01%     
==========================================
  Files         168      168              
  Lines        8352     8331      -21     
==========================================
- Hits         8338     8317      -21     
  Misses         14       14

Impacted Files	Coverage Δ
evalml/pipelines/__init__.py	`100.00% <ø> (ø)`
evalml/pipelines/components/__init__.py	`100.00% <ø> (ø)`
...lml/automl/automl_algorithm/iterative_algorithm.py	`100.00% <100.00%> (ø)`
evalml/model_family/utils.py	`100.00% <100.00%> (ø)`
evalml/pipelines/components/utils.py	`100.00% <100.00%> (ø)`
evalml/pipelines/pipeline_base.py	`100.00% <100.00%> (ø)`
evalml/pipelines/utils.py	`100.00% <100.00%> (ø)`
evalml/tests/component_tests/test_components.py	`100.00% <100.00%> (ø)`
evalml/tests/component_tests/test_estimators.py	`100.00% <100.00%> (ø)`
evalml/tests/component_tests/test_utils.py	`100.00% <100.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b7afaa...52b6966. Read the comment docs.

dsherry

Just discussed with Freddy, agreed on plan for changes. Its close though!

dsherry · 2020-07-01T21:32:04Z

evalml/utils/gen_utils.py

+
+        return classes
+
+    return get_all_classes


Why the extra level of indirection here? Why not just return the list of classes?

evalml/utils/gen_utils.py

dsherry · 2020-07-01T21:51:46Z

evalml/utils/gen_utils.py

+        classes = []
+        for cls in all_classes:
+            try:
+                cls(*args)


Hmm not blocking but perhaps we should just update pipelines to have parameters default to an empty dict. Not sure there's a good reason why we require that for pipelines. What if you want a pipeline with default params? The way to get that would be to pass in an empty dict.

Could file separately

dsherry · 2020-07-02T16:17:44Z

evalml/utils/gen_utils.py

+    subclasses = []
+
+    while classes_to_check:
+        subclass = classes_to_check.pop()


What if we do this:

# __module__ == 'evalml.pipelines.components' # __name__ == 'MyComponent' if not 'evalml.pipelines' in subclass.__module__: continue

dsherry · 2020-07-02T16:31:29Z

evalml/pipelines/utils.py

-            logger.debug('Estimator {} failed import, withholding from all_estimators'.format(estimator_name))
-    return estimators
-
-
 def get_estimators(problem_type, model_families=None):


Let's also move this to evalml/pipelines/components/utils.py, to be alongside the all_estimators list

As we discussed offline, this leads to a circular dependency becausemodel_family depends on pipeline_base.py, which depends on pipelines/components/utils which depends on model_family

dsherry · 2020-07-02T16:33:39Z

evalml/pipelines/components/utils.py

+_estimator_message = 'Estimator {} failed import, withholding from all_estimators'
+_all_estimators = get_importable_subclasses(Estimator, args=[],
+                                            message=_estimator_message,
+                                            used_in_automl=False)


In discussion with Freddy. We are including this so that our tests can look up the components which we don't currently support in automl, e.g. extratrees, elasticnet, and baseline estimators. We hope to delete this in the future once those are supported for automl.

dsherry · 2020-07-02T16:36:10Z

evalml/pipelines/components/utils.py

+                                            used_in_automl=False)
+all_estimators_used_in_search = get_importable_subclasses(Estimator, args=[],
+                                                          message=_estimator_message,
+                                                          used_in_automl=True)


In discussion with Freddy. Let's keep all_estimators_used_in_search private for now. Once we add all components to automl, we won't need to draw a distinction between those in our codebase vs those in our codebase supported for automl. And at that point we'll want to rename this. So, keeping private.

dsherry · 2020-07-02T16:38:59Z

Use-cases to support:

User wants to run automl with stock evalml components/estimators, make_pipeline
User wants to run automl with custom estimators (we don't currently support this, can support later)
User wants to run automl with custom pipelines (allowed_pipelines)

Plan for this PR:

Define get_importable_subclasses, only looks at classes in the evalml module, excludes custom classes
Move evalml.pipelines.get_estimators to evalml.pipelines.components.get_estimators
Keep the all_estimators* lists private for now
Resolve other comments

This reverts commit 65de98b.

freddyaboulton

@dsherry This should be good for a second look now! The one thing I wanted to call out is that I saw another circular dependency issue with model_family and pipeline utils. I resolved this by not importing the pipeline utils in pipelines/__init__.py and I left a comment explaining my reasoning. This change had some minor impacts on the api_reference.rst file but apart from that, I kept pretty close to what we discussed earlier.

freddyaboulton · 2020-07-02T21:52:39Z

docs/source/automl/regression_example.ipynb

@@ -16,7 +16,7 @@
    "import evalml\n",
    "from evalml import AutoMLSearch\n",
    "from evalml.demos import load_diabetes\n",
-    "from evalml.pipelines import PipelineBase, get_pipelines\n",


Removed get_pipelines because it is not used.

freddyaboulton · 2020-07-02T22:56:44Z

evalml/pipelines/__init__.py

@@ -59,14 +59,7 @@
    BaselineRegressionPipeline,
    MeanBaselineRegressionPipeline
 )
-from .utils import (


Either the model family utils or the pipeline utils are exposed at the top level of their respective modules but not both.

If both are exposed at the top level, we will get circular dependency errors. The reason is as follows:

model_family is imported first. This needs to import pipelines/components/utils because of the line from evalml.pipelines.components.utils import _all_estimators_used_in_search

Before you can import components/utils, you need to run pipelines/__init__.py
If you expose, pipeline utils in pipelines/__init__.py, then you will need to import model_family which takes you full circle.

I decided to not expose the pipeline utils because they are only used by automl search internally and so users are not likely to do from evalml.pipelines.get_pipeline (at least for now) whereas model_family.list_model_families feels more intuitive.

Great, yeah sounds good to me. We can always change this later if there's a good reason to do so.

freddyaboulton · 2020-07-02T23:19:30Z

Also, there seems to be another issue with code cov like the one I brought up at stand up earliear this week. I reduced the hits and lines by the same amount and now I fail project coverage. I can't find any dark red lines in my diff though.

dsherry

@freddyaboulton looks good!

I think we should wait to merge this until @angela97lin mergeas #904 . Unfortunately it looks like there will be conflicts to resolve but hopefully not too bad

dsherry · 2020-07-07T20:22:45Z

docs/source/api_reference.rst

+    all_pipelines
+    get_pipelines
+    get_estimators
+    make_pipeline


@freddyaboulton why make this change?

Since pipelines.utils are no longer listed in pipelines/__init__.py, we need to change the current module so that sphinx can build the docs.

dsherry · 2020-07-07T20:23:41Z

evalml/pipelines/__init__.py

@@ -59,14 +59,7 @@
    BaselineRegressionPipeline,
    MeanBaselineRegressionPipeline
 )
-from .utils import (


Great, yeah sounds good to me. We can always change this later if there's a good reason to do so.

dsherry · 2020-07-07T20:34:25Z

evalml/pipelines/components/utils.py

+_all_estimators = get_importable_subclasses(Estimator, args=[], used_in_automl=False)
+_all_estimators_used_in_search = get_importable_subclasses(Estimator, args=[], used_in_automl=True)
+_all_transformers = get_importable_subclasses(Transformer, args=[], used_in_automl=False)
+all_components = _all_estimators + _all_transformers


👍 this is great.

dsherry · 2020-07-07T20:35:17Z

evalml/pipelines/utils.py

-            pipeline_name = pipeline_class.name
-            logger.debug('Pipeline {} failed import, withholding from all_pipelines'.format(pipeline_name))
-    return pipelines
+all_pipelines = get_importable_subclasses(PipelineBase, args=[{}], used_in_automl=True)


Once #904 is merged, we can delete this, and we can then also delete the args input.

dsherry · 2020-07-07T20:38:59Z

evalml/tests/component_tests/test_components.py

-    components = all_components()
-    for component_name, component_class in components.items():
+    components = all_components
+    for component_class in components:


Could make into one line

dsherry · 2020-07-07T20:40:20Z

evalml/tests/utils_tests/test_gen_utils.py

+    class ChildClass(ComponentBase):
+        pass
+
+    assert ChildClass not in get_importable_subclasses(ComponentBase, args=[])


freddyaboulton linked an issue Jun 30, 2020 that may be closed by this pull request

Common method for registering pipeline/component/other classes #363

Closed

freddyaboulton self-assigned this Jun 30, 2020

freddyaboulton force-pushed the 363-common-method-for-registering-components branch from 83cb4f2 to 0adc224 Compare June 30, 2020 17:44

freddyaboulton mentioned this pull request Jun 30, 2020

Include ExtraTrees, ElasticNet, and Baselines in all_estimators #899

Closed

freddyaboulton added this to the July 2020 milestone Jun 30, 2020

freddyaboulton marked this pull request as ready for review June 30, 2020 18:25

freddyaboulton requested review from dsherry, angela97lin and jeremyliweishih June 30, 2020 18:25

freddyaboulton force-pushed the 363-common-method-for-registering-components branch 2 times, most recently from 52370da to bb62700 Compare July 1, 2020 19:11

dsherry suggested changes Jul 2, 2020

View reviewed changes

freddyaboulton added 6 commits July 2, 2020 14:09

Refactoring all_components, all_pipelines, all_estimators.

53e2717

Adding ability to filter for classes used during automl search.

2681efb

Updating changelog for 898.

4862fe0

Fixing import statement in iterative_algorithm.

254584c

Making all_estimators, all_pipelines, all_transformers, static lists.

3a26dbc

Moving get_estimators back to pipelines/utils.py.

e475845

freddyaboulton force-pushed the 363-common-method-for-registering-components branch from bb62700 to e475845 Compare July 2, 2020 20:53

freddyaboulton added 3 commits July 2, 2020 17:35

Fixing docs. Making all_pipelines public.

ab0dd1e

Not importing model_family.utils at the top level.

65de98b

Revert "Not importing model_family.utils at the top level."

6e19993

This reverts commit 65de98b.

freddyaboulton commented Jul 2, 2020

View reviewed changes

freddyaboulton mentioned this pull request Jul 6, 2020

Consolidate utils to avoid circular imports #911

Closed

dsherry approved these changes Jul 7, 2020

View reviewed changes

freddyaboulton added 2 commits July 8, 2020 15:07

Merging latest master.

c4849d7

Condensing loop into one line.

52b6966

freddyaboulton merged commit d2f6c79 into master Jul 8, 2020

freddyaboulton deleted the 363-common-method-for-registering-components branch July 8, 2020 19:43

This was referenced Jul 16, 2020

Release v0.11.2 #939

Merged

Move list_model_families to avoid circular dependencies #959

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

363 common method for registering components #898

363 common method for registering components #898

freddyaboulton commented Jun 30, 2020 •

edited

Loading

codecov bot commented Jun 30, 2020 •

edited

Loading

dsherry left a comment

dsherry Jul 1, 2020

dsherry Jul 1, 2020

dsherry Jul 2, 2020

freddyaboulton Jul 2, 2020

dsherry Jul 2, 2020

freddyaboulton Jul 2, 2020

dsherry Jul 2, 2020

dsherry Jul 2, 2020

dsherry commented Jul 2, 2020

freddyaboulton left a comment

freddyaboulton Jul 2, 2020

freddyaboulton Jul 2, 2020

dsherry Jul 7, 2020

freddyaboulton commented Jul 2, 2020

dsherry left a comment

dsherry Jul 7, 2020

freddyaboulton Jul 7, 2020

dsherry Jul 7, 2020

dsherry Jul 7, 2020

dsherry Jul 7, 2020

dsherry Jul 7, 2020

dsherry Jul 7, 2020

363 common method for registering components #898

363 common method for registering components #898

Conversation

freddyaboulton commented Jun 30, 2020 • edited Loading

Pull Request Description

codecov bot commented Jun 30, 2020 • edited Loading

Codecov Report

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry commented Jul 2, 2020

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Jul 2, 2020

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Jun 30, 2020 •

edited

Loading

codecov bot commented Jun 30, 2020 •

edited

Loading