Skip to content

Integrate DefaultAlgorithm into AutoMLSearch - #2634

Merged
jeremyliweishih merged 42 commits into
mainfrom
js_2526_algo
Aug 31, 2021
Merged

jeremyliweishih merged 42 commits into
mainfrom
js_2526_algo

Conversation

@jeremyliweishih

@jeremyliweishih jeremyliweishih commented Aug 16, 2021

Copy link
Copy Markdown
Collaborator

Fixes #2526.

  • Add top level search methods for fast/long mode
  • Add algorithm parameter to AutoMLSearch
  • Add coverage for custom_hyperparameters and pipeline params in EvalML algo
  • Fix docs
  • File issue for seperating iterative algorithm and AutoMLSearch
  • File issue for evalml algorithm and time series

@codecov

codecov Bot commented Aug 16, 2021

Copy link
Copy Markdown

Codecov Report

Merging #2634 (4c2c471) into main (d3ae705) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #2634     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        300     301      +1     
  Lines      27439   27600    +161     
=======================================
+ Hits       27395   27556    +161     
  Misses        44      44             
Impacted Files Coverage Δ
evalml/__init__.py 100.0% <100.0%> (ø)
evalml/automl/__init__.py 100.0% <100.0%> (ø)
evalml/automl/automl_algorithm/__init__.py 100.0% <100.0%> (ø)
...valml/automl/automl_algorithm/default_algorithm.py 100.0% <100.0%> (ø)
evalml/automl/automl_search.py 99.9% <100.0%> (+0.1%) ⬆️
.../automl_tests/test_automl_search_classification.py 100.0% <100.0%> (ø)
...ests/automl_tests/test_automl_search_regression.py 100.0% <100.0%> (ø)
...valml/tests/automl_tests/test_default_algorithm.py 100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_search.py 100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_search_iterative.py 100.0% <100.0%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3ae705...4c2c471. Read the comment docs.

@jeremyliweishih
jeremyliweishih marked this pull request as ready for review August 18, 2021 19:35


def test_categorical_classification(X_y_categorical_classification):
def test_categorical_classification(AutoMLTestEnv, X_y_categorical_classification):

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By product of testing with EvalMLAlgorithm. I can remove this but I think these changes are good.

@jeremyliweishih
jeremyliweishih requested a review from a team August 18, 2021 20:43

@ParthivNaresh ParthivNaresh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work with this, I'm going to take a deeper look at the tests in a bit, just left some other comments. Also do we have plans to update our docs? We briefly mention the standalone search method in the AutoML section but it might be helpful to have a separate section describing our algorithms. It could be a lot of updates so feel free to file a separate issue for this if you don't want to lump it all into one PR.

Comment thread evalml/pipelines/components/transformers/column_selectors.py Outdated
Comment thread evalml/automl/automl_search.py Outdated
@jeremyliweishih

Copy link
Copy Markdown
Collaborator Author

Awesome work with this, I'm going to take a deeper look at the tests in a bit, just left some other comments. Also do we have plans to update our docs? We briefly mention the standalone search method in the AutoML section but it might be helpful to have a separate section describing our algorithms. It could be a lot of updates so feel free to file a separate issue for this if you don't want to lump it all into one PR.

@ParthivNaresh, I'll make an issue to track this!

Comment thread evalml/automl/automl_search.py Outdated
Comment thread evalml/automl/automl_search.py Outdated

@chukarsten chukarsten left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jeremy, great work. This looks good to me. I think we should figure out (not necessarily in this PR) a new name instead of EvalMLAlgorithm, and same for the resulting string. But yea, would like to see this puppy in action and see how it compares to iterative.

Comment thread evalml/automl/automl_search.py Outdated
Comment thread evalml/automl/automl_search.py Outdated
_pipelines_per_batch (int): The number of pipelines to train for every batch after the first one.
The first batch will train a baseline pipline + one of each pipeline family allowed in the search.

_automl_algorithm (str): The automl algorithm to use. Currently the two choices are 'iterative' and 'evalml'. Defaults to `iterative`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to come up with a better alternative to "evalml", haha.

@ParthivNaresh ParthivNaresh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests look great, nice work with this!

@jeremyliweishih jeremyliweishih changed the title Integrate EvalMLAlgorithm into AutoMLSearch Integrate DefaultAlgorithm into AutoMLSearch Aug 31, 2021
@jeremyliweishih
jeremyliweishih merged commit 3ae1500 into main Aug 31, 2021
@chukarsten chukarsten mentioned this pull request Sep 1, 2021
@freddyaboulton
freddyaboulton deleted the js_2526_algo branch May 13, 2022 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate new algorithm into AutoMLSearch

4 participants