Make `DefaultAlgorithm` the default AutoML Algorithm #3261

jeremyliweishih · 2022-01-19T21:07:37Z

Fixes #2867. Since there is a lot in this PR, I will use the PR for #2693 to fix docstrings and think about changes to AutoMLSearch API.

I will be doing the following in this PR:

justify why a test deserves a _iterative tag
moving such _iterative tests to their own files
generalize any tests I can to be algorithm agnostic

I will also create a list of tests that we can improve in #2868 with a MockAlgorithm with these following criteria:

test logic hinges on algorithm implementation (i.e number of pipelines and such)
I cannot generalize the test to work with both current algorithm implementations

List of tests that we can use a mock algorithm with:

1. test_automl_algorithm
2. test_time_series_regression_with_parameters
3. test_max_batches_works
4. test_callback (there’s multiple)
5. test_automl_tuner_exception
6. test_results_getter
7. test_automl_pipeline_params_kwargs
8. test_search_with_text_and_ensembling_iterative
9. test_pipelines_per_batch_iterative
10. test_automl_respects_random_seed_iterative
11. test_automl_respects_pipeline_parameters_with_duplicate_components_iterative
12. test_automl_respects_pipeline_custom_hyperparameters_with_duplicate_components_iterative
13. test_automl_search_sampler_ratio
14. test_automl_search_sampler_method
15. test_time_series_pipeline_parameter_warnings
16. test_pipeline_count

List of tests we can maybe generalize with much larger lift:

any tests with component graphs or iterative specific parameters

Performance tests vs. using top level search method (results are identical):
default_swap_com.html.zip

codecov · 2022-01-19T21:11:43Z

Codecov Report

Merging #3261 (7ba7992) into main (e06a912) will decrease coverage by 0.2%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3261     +/-   ##
=======================================
- Coverage   99.8%   99.7%   -0.1%     
=======================================
  Files        324     327      +3     
  Lines      31764   31814     +50     
=======================================
+ Hits       31674   31689     +15     
- Misses        90     125     +35

Impacted Files	Coverage Δ
evalml/automl/automl_algorithm/automl_algorithm.py	`100.0% <100.0%> (ø)`
...valml/automl/automl_algorithm/default_algorithm.py	`100.0% <100.0%> (ø)`
evalml/automl/automl_search.py	`99.7% <100.0%> (-0.1%)`	⬇️
...ts/automl_tests/parallel_tests/test_automl_dask.py	`96.3% <100.0%> (-3.7%)`	⬇️
evalml/tests/automl_tests/test_automl.py	`99.5% <100.0%> (-0.1%)`	⬇️
...ts/automl_tests/test_automl_iterative_algorithm.py	`100.0% <100.0%> (ø)`
.../automl_tests/test_automl_search_classification.py	`96.4% <100.0%> (-3.6%)`	⬇️
...sts/test_automl_search_classification_iterative.py	`100.0% <100.0%> (ø)`
...ests/automl_tests/test_automl_search_regression.py	`94.7% <100.0%> (-5.3%)`	⬇️
...l_tests/test_automl_search_regression_iterative.py	`100.0% <100.0%> (ø)`
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e06a912...7ba7992. Read the comment docs.

…e created after the fact in default

…, must use iterative here

…d remove uncessary mock logic

…erative

…and remove extra call

…se iterative

…d use iterative

…test

…ents to test_automl_respects_pipeline_parameters_with_duplicate_components_iterative and use iterative

…ams algorithm agnostic

…line_parameter_warnings_component_graphs_iterative and use iterative

…generalize

…tive'

… and use iterative

…erative and use iterative

chukarsten

Jeremy - great job here. I like what you're doing with the test refactoring and I think you're really hinting at the need to split integration vs. unit testing, which I think will require more planning and thought. It would be nice if we could get some of those coverage gaps filled, but I won't block on that. I think if you want to hit the ones that are low hanging fruit, that would be great.

chukarsten · 2022-02-01T03:46:59Z

evalml/automl/automl_search.py

@@ -446,7 +446,7 @@ def __init__(
        allow_long_running_models=False,
        _ensembling_split_size=0.2,
        _pipelines_per_batch=5,
-        _automl_algorithm="iterative",
+        _automl_algorithm="default",


That seems reasonable.

chukarsten · 2022-02-01T15:51:00Z

evalml/tests/automl_tests/test_automl.py

-            "median",
-        ]
-        assert 10 <= row["parameters"]["One Hot Encoder"]["top_n"] <= 12
+    assert automl._automl_algorithm._custom_hyperparameters == custom_hyperparameters


Agreed. So begins the work of disentangling unit tests from integration tests.

chukarsten · 2022-02-01T15:52:43Z

evalml/tests/automl_tests/test_automl.py

            verbose=verbose,
        )
+        env = AutoMLTestEnv("binary")


evalml/tests/automl_tests/test_automl_iterative_algorithm.py

chukarsten · 2022-02-01T16:29:48Z

evalml/tests/automl_tests/test_automl_iterative_algorithm.py

+        if "Mode Baseline Binary" in row["pipeline_name"]:
+            continue
+        if row["pipeline_name"] == "Name_linear":
+            assert row["parameters"]["Imputer"]["numeric_impute_strategy"] == "mean"


I think this will require some investigation. We probably want to clear these up before we merge this PR. Maybe run these tests and see what is contained in row["pipeline_name"] to try and verify the intent of this line. I suspect this is the product of some naming rot as "Name_linear" doesn't look like anything I've seen, stylistically, since being here.

jeremyliweishih · 2022-02-01T20:35:03Z

@chukarsten I originally filed #3292 to fix the coverage issues but I'll see if I can get a quick win here!

…867_default

…_with_duplicate_components

bchen1116

This looks good to me! Great work on pushing this forward, I just left a few nits/questions, but nothing blocking.

I know you've run some perf tests with the Default Algo before, but I think it could be useful to run perf tests on this PR versus current main, just to ensure there is no change in performance for peace of mind.

docs/source/release_notes.rst

bchen1116 · 2022-02-03T16:30:50Z

evalml/tests/utils_tests/test_logger.py

@@ -108,7 +108,13 @@ def test_pipeline_count(
    else:
        X, y = X_y_regression
    if not allowed_families:
-        _ = AutoMLSearch(X_train=X, y_train=y, problem_type=type_, verbose=verbose)
+        _ = AutoMLSearch(


Is this going to be addressed for default algo in a future PR?

* Change default max batches to fast mode * Make automl_algorithm parameter public * RL * Fix release notes * Add more logic to max_batches change * Fix tests to work with new max batches * Drop periods down to 100 * Add default max batches property to algos * Lint * Add time series logic to default max batches of default algo * Fix test_max_batches_num_pipelines

Make default default

79b7f31

jeremyliweishih added 28 commits January 20, 2022 16:56

Merge branch 'main' of github.com:alteryx/evalml into js_2867_default

a7e5350

Merge branch 'main' of github.com:alteryx/evalml into js_2867_default

c03a883

Remove batches case as the limit is algorithm dependent

ec25212

Ammend batches case to be greater than 0

287ed7f

Remove allowed model families and # of pipelines for test_rankings

5fa67fe

Either remove test_automl_tuner_exception or use iterative, tuners ar…

0351098

…e created after the fact in default

rename to test_automl_feature_selection_with_allowed_component_graphs…

c59cfc1

…, must use iterative here

Rename to test_automl_allowed_component_graphs_iterative_algorithm an…

7a351ce

…d remove uncessary mock logic

Change to use default and explicitely name algorithm

0a9159b

Rename to test_describe_pipeline_with_ensembling_iterative and use it…

f52befb

…erative

Rename to test_component_graph_with_incorrect_problem_type_iterative …

51e2529

…and remove extra call

Rename to test_jobs_cancelled_when_keyboard_interrupt_iterative and u…

77b3c89

…se iterative

Rename to test_max_iteration_works_with_stacked_ensemble_iterative an…

809f251

…d use iterative

Merge branch 'main' of github.com:alteryx/evalml into js_2867_default

7733400

Change max_batches tests to only test max batches and moved original …

2a3e1a3

…test

lint

2ef28a5

Change ts regression test to check algorithm pipeline parameters

375900f

Rename test_automl_respects_pipeline_parameters_with_duplicate_compon…

5e994ff

…ents to test_automl_respects_pipeline_parameters_with_duplicate_components_iterative and use iterative

make test_automl_adds_pipeline_parameters_to_custom_pipeline_hyperpar…

47ba0f1

…ams algorithm agnostic

Fix test_automl_drop_unknown_columns by adding verbose to default

5c19dd5

Rename test_pipeline_parameter_warnings_component_graphs to test_pipe…

1b4d2f0

…line_parameter_warnings_component_graphs_iterative and use iterative

Rename to test_component_and_pipeline_warnings_surface_in_search and …

0a3aa5f

…generalize

Rename to test_graph_automl_iterative and use iterative

b88388f

Rename to test_automl_respects_pipeline_order_iterative and use itera…

4512268

…tive'

Rename to test_get_ensembler_input_pipelines_iterative and use iterative

f41d70c

Rename to test_automl_one_allowed_component_graph_ensembling_disabled…

38deaa2

… and use iterative

Generalize test_max_batches_plays_nice_with_other_stopping_criteria

da7d1e5

Rename to test_pipeline_hyperparameters_make_pipeline_other_errors_it…

a92dc0e

…erative and use iterative

Merge branch 'main' into js_2867_default

19e5f13

jeremyliweishih requested review from freddyaboulton, eccabay, bchen1116, angela97lin, christopherbunn, chukarsten and ParthivNaresh January 31, 2022 15:39

chukarsten approved these changes Feb 1, 2022

View reviewed changes

jeremyliweishih added 11 commits February 1, 2022 15:35

Merge branch 'main' into js_2867_default

3146aa3

Merge branch 'main' of github.com:alteryx/evalml into js_2867_default

992a2a0

Merge branch 'js_2867_default' of github.com:alteryx/evalml into js_2…

27b9f32

…867_default

Rename tests

2c6549b

Fix test_describe_pipeline_with_ensembling

5e40768

Fix coverage in test_pipeline_custom_hyperparameters_make_pipeline

7f31a18

Fix coverage for test_automl_respects_pipeline_custom_hyperparameters…

76d5d53

…_with_duplicate_components

lint

5041b4c

Merge branch 'main' of github.com:alteryx/evalml into js_2867_default

c5791f0

Fix doc failures

717316a

Fix release notes

8e73bf4

bchen1116 approved these changes Feb 3, 2022

View reviewed changes

Fix release notes

df9f827

jeremyliweishih mentioned this pull request Feb 3, 2022

Change AutoMLSearch parameters for AutoML Algorithm Upgrade #3304

Merged

jeremyliweishih added 5 commits February 10, 2022 10:50

Merge branch 'main' into js_2867_default

103fae4

Merge branch 'main' into js_2867_default

e12c592

Merge branch 'main' into js_2867_default

f3cd3e1

Merge branch 'main' into js_2867_default

7ba7992

jeremyliweishih merged commit e67430c into main Feb 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `DefaultAlgorithm` the default AutoML Algorithm #3261

Make `DefaultAlgorithm` the default AutoML Algorithm #3261

jeremyliweishih commented Jan 19, 2022 •

edited

Loading

codecov bot commented Jan 19, 2022 •

edited

Loading

chukarsten left a comment

chukarsten Feb 1, 2022

chukarsten Feb 1, 2022

chukarsten Feb 1, 2022

chukarsten Feb 1, 2022

jeremyliweishih commented Feb 1, 2022

bchen1116 left a comment

bchen1116 Feb 3, 2022

Make DefaultAlgorithm the default AutoML Algorithm #3261

Make DefaultAlgorithm the default AutoML Algorithm #3261

Conversation

jeremyliweishih commented Jan 19, 2022 • edited Loading

codecov bot commented Jan 19, 2022 • edited Loading

Codecov Report

chukarsten left a comment

Choose a reason for hiding this comment

chukarsten Feb 1, 2022

Choose a reason for hiding this comment

chukarsten Feb 1, 2022

Choose a reason for hiding this comment

chukarsten Feb 1, 2022

Choose a reason for hiding this comment

chukarsten Feb 1, 2022

Choose a reason for hiding this comment

jeremyliweishih commented Feb 1, 2022

bchen1116 left a comment

Choose a reason for hiding this comment

bchen1116 Feb 3, 2022

Choose a reason for hiding this comment

Make `DefaultAlgorithm` the default AutoML Algorithm #3261

Make `DefaultAlgorithm` the default AutoML Algorithm #3261

jeremyliweishih commented Jan 19, 2022 •

edited

Loading

codecov bot commented Jan 19, 2022 •

edited

Loading