Engine API

[dsherry]

Fix #1296

Changes

• Add EngineBase base class to define engine interface, and SequentialEngine which preserves existing behavior, evaluates pipelines in series
• Define _pre_evaluation_callback for logging and _post_evaluation_callback to add evaluation results to AutoMLSearch
• Move AutoMLSearch. _compute_cv_scores to EngineBase.train_and_score_pipeline as a staticmethod
• Refactored the AutoMLSearch binary threshold tuning logic into tune_binary_threshold so it could be reused. Fixed bug where all bin class pipelines were getting a threshold of 0.5 -- now will get None as was the original intent.
• Tweak criterion for doing bin threshold tuning to apply to timeseries binary classification as well.
• Now that AutoMLSearch constructor expects data, move much of the search setup logic into the constructor to simplify the behavior.
• Have AutoMLAlgorithm.add_result raise error if pipeline is unrecognized
• Update and simplify keyboard interrupt code; no intended change in user behavior
• Rename _check_stopping_condition to _should_continue
• Add traceback to the end of log_error_callback, which I believe partially handles No stack trace visible in debug log #1778 (@bchen1116 heads up). This was helpful to me when debugging so I just went ahead and did it.
• Fixed some test bugs
  • In test_automl.py, CustomClassificationObjective wasn't subclassed from BinaryClassificationObjective, same for regression
  • Many tests were mocking pipeline score but not providing a return value, causing scores to be nan

Future:

• Pass a config object to engine classes' constructor instead of AutoMLSearch instance. I couldn't easily do this today because of the logging callback -- it expects a reference to AutoMLSearch to add the error to the results structure
• When we implement parallel evaluation, we'll have to decide what to do with the early stopping parameters, as those won't map over easily to the parallel batched evaluation format.
• The changes in this PR are not threadsafe. The calls to the should_continue/pre/post callbacks access and modify AutoMLSearch state without any sort of locking or atomicity. This is fine for this PR, but when we implement parallel evaluation, we'll need to address this.
• Add ability to accept an EngineBase subclass as input to AutoMLSearch, rather than creating internally. This would require thought about how to pass parameters and create the instance. We'd probably wanna switch to using a config object first for AutoMLSearch

Big credit to @christopherbunn for providing the initial implementation this PR is based on! 🙏

[codecov]

Codecov Report

Merging #1838 (e9eaea2) into main (99dd43e) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1838     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         255      260      +5     
  Lines       20644    20832    +188     
=========================================
+ Hits        20636    20826    +190     
+ Misses          8        6      -2     

Impacted Files	Coverage Δ
evalml/pipelines/pipeline_base.py	100.0% <ø> (ø)
evalml/automl/__init__.py	100.0% <100.0%> (ø)
evalml/automl/automl_algorithm/automl_algorithm.py	100.0% <100.0%> (ø)
evalml/automl/automl_search.py	100.0% <100.0%> (+0.4%)	⬆️
evalml/automl/callbacks.py	100.0% <100.0%> (ø)
evalml/automl/engine/__init__.py	100.0% <100.0%> (ø)
evalml/automl/engine/engine_base.py	100.0% <100.0%> (ø)
evalml/automl/engine/sequential_engine.py	100.0% <100.0%> (ø)
evalml/automl/utils.py	100.0% <100.0%> (ø)
evalml/objectives/objective_base.py	100.0% <100.0%> (ø)
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99dd43e...e9eaea2. Read the comment docs.

[dsherry]

A weakness of this code: if train_and_score_pipeline or one of the callbacks errors out, the whole while loop would halt.

We use the error callback inside train_and_score_pipeline to handle errors during pipeline evaluation, so this isn't a big concern. But if the data splitter errors, it would move up the stack. I think that behavior is acceptable for now.

[chukarsten]

dumb question: can we wrap train_score in a try/except and just continue/alert on exception?

[dsherry]

That's not dumb! You are correct, adding proper error handling now is the responsible thing to do. I was being lazy. Will do.

[dsherry]

I took a close look at this. I still agree that we should update the way we do error handling. But I'd like us to do it in a separate PR. There's a decent amount to consider. This PR doesn't change the existing error handling pattern; we will have to in order to complete the parallel work.

[chukarsten]

This looks good to me. Looks like a lot, but looks good.

[chukarsten]

dumb question: can we wrap train_score in a try/except and just continue/alert on exception?

[bchen1116]

Big PR! Tests and implementation looks good, but had a few cleanup suggestions. Had a question about add_to_rankings. It seems like the performance of it changed compared to main, because when I run

from evalml import AutoMLSearch
from evalml.demos import load_breast_cancer
from evalml.pipelines import BinaryClassificationPipeline

class NewPipeline(BinaryClassificationPipeline):
    component_graph = ['Imputer', 'LightGBM Classifier']

X, y = load_breast_cancer()
automl = AutoMLSearch(X_train=X, y_train=y, problem_type='binary', objective='f1')
automl.search()    
automl.add_to_rankings(NewPipeline({}))
automl.full_rankings

I get

whereas running on main doesn't give me this same error.

[dsherry]

@bchen1116 thanks for your comments!

That example you included is fantastic. You are correct. The way this code is laid out in this PR currently, it would prevent users from adding pipelines to the rankings leaderboard if they were not initially included in allowed_pipelines. After reflecting on your comments here and in the PR, I agree, there's no reason to add this limitation. I think I got confused, because its kinda weird that the existing code always tries to add pipelines to the automl algorithm tuners--it doesn't make sense to do that if no tuner exists!

I just updated the code to preserve the existing behavior for add_to_rankings (no exception thrown), and also tried to clean up the implementation. AutoMLAlgorithm.add_result will still raise an exception, because that seems like a helpful thing for that API to do, but the AutoMLSearch results callback swallows the exception and continues.

[bchen1116]

Thanks for addressing the comments! The implementation looks good to me, just left a few nit-picky comments but nothing too pressing here.

Do you think it'd be useful to run perf tests on this to ensure that the new search algo performances don't differ too much from what we currently have on main?

[bchen1116]

If we're checking that self.X_train, self.y_train aren't None, should we also do this for the AutoMLSearch object? If that's None, this call should also fail

[dsherry]

That's true. I suppose it can't hurt to add that. Since there's only one call site at the moment, and if we didn't set automl search there all the tests will fail, I will hold off on adding that right now.

[angela97lin]

Wow, this is an incredible PR with a lot of great refactoring 👁️ 👁️

[dsherry]

Perf test results here. Flat as expected!