Add callback for error handling in AutoMLSearch

[angela97lin]

Closes #1217

Questions:

• Should we subclass Protocol to enforce that kwargs is passed? (https://stackoverflow.com/questions/64071488/callback-protocol-for-functions-with-different-kwargs)

If all objectives fail and AutoMLException is raised, we will raise since it indicates that something is likely wrong. This is the same behavior as what is currently implemented.

[codecov]

Codecov Report

Merging #1403 (15404b2) into main (a5e6c50) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1403     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         213      214      +1     
  Lines       13975    14040     +65     
=========================================
+ Hits        13968    14033     +65     
  Misses          7        7             

Impacted Files	Coverage Δ
...ediction_explanations_tests/test_user_interface.py	100.0% <ø> (ø)
evalml/automl/automl_search.py	99.7% <100.0%> (-<0.1%)	⬇️
evalml/automl/callbacks.py	100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_automl.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5e6c50...15404b2. Read the comment docs.

[jeremyliweishih]

Looks good! I agree with @freddyaboulton that we should add the traceback as part of saving the errors. Could be done in this PR or another but would be great for looking glass.

[dsherry]

@angela97lin this looks great!

I left a few comments, but there was nothing major.

RE Protocol subclassing, I think we're ok to skip that for now, but its a good idea.

[dsherry]

Cool, so because we pass self back in the callback, someone could theoretically write a callback which responds to failure and calls automl.search again. So this is implementing one part of an asynchronous API (#1047 ). Neat!

[angela97lin]

Yes!!