Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add skopt.space.Categorical documentation and tests for component hyperparameter range #1228

Merged
merged 14 commits into from Sep 29, 2020

Conversation

bchen1116
Copy link
Contributor

@bchen1116 bchen1116 commented Sep 25, 2020

fix #352

Update documentation to include option for either list or skopt.space.Categorical in the hyperparameter definition. Include unit tests.

@bchen1116 bchen1116 self-assigned this Sep 25, 2020
@codecov
Copy link

codecov bot commented Sep 25, 2020

Codecov Report

Merging #1228 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1228   +/-   ##
=======================================
  Coverage   99.92%   99.92%           
=======================================
  Files         200      200           
  Lines       12339    12365   +26     
=======================================
+ Hits        12330    12356   +26     
  Misses          9        9           
Impacted Files Coverage Δ
.../automl_tests/test_automl_search_classification.py 100.00% <100.00%> (ø)
evalml/tests/component_tests/test_components.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbdb8bc...0b9414e. Read the comment docs.

@bchen1116 bchen1116 marked this pull request as ready for review Sep 25, 2020
Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

@bchen1116 This looks good! I think it would be good to write a test where AutoMLSearch searches over pipelines that use a component that defines some hyperparameter with Categorical before merging. (I think we could subclass LogisticRegressionClassifier and change the range for penalty for example)

docs/source/user_guide/pipelines.ipynb Outdated Show resolved Hide resolved
evalml/tests/component_tests/test_components.py Outdated Show resolved Hide resolved
Copy link
Contributor

@angela97lin angela97lin left a comment

Looks good! I agree with @freddyaboulton's comments about testing / clarification but otherwise 👍

docs/source/user_guide/components.ipynb Outdated Show resolved Hide resolved
@bchen1116 bchen1116 requested a review from freddyaboulton Sep 29, 2020
Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

@bchen1116 Thanks for making these changes!

@@ -852,3 +853,36 @@ def test_component_equality_all_components(component_class):
parameters = component.parameters
equal_component = component_class(**parameters)
assert component == equal_component


@pytest.mark.parametrize("categorical", [{
Copy link
Contributor

@freddyaboulton freddyaboulton Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@bchen1116 bchen1116 merged commit 4387fe0 into main Sep 29, 2020
}

automl = AutoMLSearch(problem_type="multiclass", allowed_pipelines=[CustomPipeline])
automl.search(X, y)
Copy link
Collaborator

@dsherry dsherry Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bchen1116 just a tip, for tests which don't check scoring, we usually mock fit/score so that we save time in the unit test. See some of the other tests in this file

@patch('evalml.pipelines.BinaryClassificationPipeline.score')
@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
def test_categorical_hyperparam(mock_fit, mock_score, X_y_multi):

Copy link
Contributor Author

@bchen1116 bchen1116 Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. So patching fit/score without calling mock_fit or mock_score still makes it run faster?

random_state=0)

assert MockComponent(agg_type="mean").fit(X, y)
assert MockComponent(agg_type="moat", category="blue").fit(X, y)
Copy link
Collaborator

@dsherry dsherry Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bchen1116 why call fit here? What is the goal of this test?

Copy link
Contributor Author

@bchen1116 bchen1116 Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal was to make sure the component works, but since the component uses an estimator, I called fit to satisfy codecov

Copy link
Collaborator

@dsherry dsherry left a comment

@bchen1116 thanks, great! I left a couple minor test suggestions.

@angela97lin angela97lin mentioned this pull request Sep 29, 2020
@freddyaboulton freddyaboulton deleted the bc_352_categorical branch May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pipeline/component doc: we accept list or Categorical as hyperparameter defs
4 participants