Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate DefaultAlgorithm into AutoMLSearch #2634

Merged
merged 42 commits into from
Aug 31, 2021
Merged

Conversation

jeremyliweishih
Copy link
Collaborator

@jeremyliweishih jeremyliweishih commented Aug 16, 2021

Fixes #2526.

  • Add top level search methods for fast/long mode
  • Add algorithm parameter to AutoMLSearch
  • Add coverage for custom_hyperparameters and pipeline params in EvalML algo
  • Fix docs
  • File issue for seperating iterative algorithm and AutoMLSearch
  • File issue for evalml algorithm and time series

@codecov
Copy link

codecov bot commented Aug 16, 2021

Codecov Report

Merging #2634 (4c2c471) into main (d3ae705) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #2634     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        300     301      +1     
  Lines      27439   27600    +161     
=======================================
+ Hits       27395   27556    +161     
  Misses        44      44             
Impacted Files Coverage Δ
evalml/__init__.py 100.0% <100.0%> (ø)
evalml/automl/__init__.py 100.0% <100.0%> (ø)
evalml/automl/automl_algorithm/__init__.py 100.0% <100.0%> (ø)
...valml/automl/automl_algorithm/default_algorithm.py 100.0% <100.0%> (ø)
evalml/automl/automl_search.py 99.9% <100.0%> (+0.1%) ⬆️
.../automl_tests/test_automl_search_classification.py 100.0% <100.0%> (ø)
...ests/automl_tests/test_automl_search_regression.py 100.0% <100.0%> (ø)
...valml/tests/automl_tests/test_default_algorithm.py 100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_search.py 100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_search_iterative.py 100.0% <100.0%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3ae705...4c2c471. Read the comment docs.

@jeremyliweishih jeremyliweishih marked this pull request as ready for review August 18, 2021 19:35
@@ -235,8 +235,9 @@ def test_multi_objective(X_y_multi):
assert automl.problem_type == ProblemTypes.BINARY


def test_categorical_classification(X_y_categorical_classification):
def test_categorical_classification(AutoMLTestEnv, X_y_categorical_classification):
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By product of testing with EvalMLAlgorithm. I can remove this but I think these changes are good.

@jeremyliweishih jeremyliweishih requested a review from a team August 18, 2021 20:43
Copy link
Contributor

@ParthivNaresh ParthivNaresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work with this, I'm going to take a deeper look at the tests in a bit, just left some other comments. Also do we have plans to update our docs? We briefly mention the standalone search method in the AutoML section but it might be helpful to have a separate section describing our algorithms. It could be a lot of updates so feel free to file a separate issue for this if you don't want to lump it all into one PR.

evalml/automl/automl_search.py Outdated Show resolved Hide resolved
@jeremyliweishih
Copy link
Collaborator Author

Awesome work with this, I'm going to take a deeper look at the tests in a bit, just left some other comments. Also do we have plans to update our docs? We briefly mention the standalone search method in the AutoML section but it might be helpful to have a separate section describing our algorithms. It could be a lot of updates so feel free to file a separate issue for this if you don't want to lump it all into one PR.

@ParthivNaresh, I'll make an issue to track this!

evalml/automl/automl_search.py Outdated Show resolved Hide resolved
evalml/automl/automl_search.py Outdated Show resolved Hide resolved
Copy link
Contributor

@chukarsten chukarsten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jeremy, great work. This looks good to me. I think we should figure out (not necessarily in this PR) a new name instead of EvalMLAlgorithm, and same for the resulting string. But yea, would like to see this puppy in action and see how it compares to iterative.

evalml/automl/automl_search.py Outdated Show resolved Hide resolved
@@ -254,6 +357,8 @@ class AutoMLSearch:
_pipelines_per_batch (int): The number of pipelines to train for every batch after the first one.
The first batch will train a baseline pipline + one of each pipeline family allowed in the search.

_automl_algorithm (str): The automl algorithm to use. Currently the two choices are 'iterative' and 'evalml'. Defaults to `iterative`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to come up with a better alternative to "evalml", haha.

Copy link
Contributor

@ParthivNaresh ParthivNaresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests look great, nice work with this!

@jeremyliweishih jeremyliweishih changed the title Integrate EvalMLAlgorithm into AutoMLSearch Integrate DefaultAlgorithm into AutoMLSearch Aug 31, 2021
@jeremyliweishih jeremyliweishih merged commit 3ae1500 into main Aug 31, 2021
@chukarsten chukarsten mentioned this pull request Sep 1, 2021
@freddyaboulton freddyaboulton deleted the js_2526_algo branch May 13, 2022 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integrate new algorithm into AutoMLSearch
4 participants