Raise an error if all pipelines produce nan scores in automl

[angela97lin]

Closes #922 by checking if all pipelines in a batch return a score of NaN and stopping search if that is the case.

[codecov]

Codecov Report

Merging #1015 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1015   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         181      181           
  Lines        9597     9628   +31     
=======================================
+ Hits         9584     9615   +31     
  Misses         13       13           

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	99.55% <100.00%> (+<0.01%)	⬆️
evalml/exceptions/exceptions.py	100.00% <100.00%> (ø)
evalml/tests/automl_tests/test_automl.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21eaa34...5cd4e33. Read the comment docs.

[freddyaboulton]

Hey @angela97lin Looks great! I have a question about the implementation. I think we should be resetting current_batch_pipeline_scores after each batch?

[freddyaboulton]

@angela97lin Looks good to me!

[dsherry]

@angela97lin I think this "was-batch-all-nan" check would fit better in IterativeAlgorithm, right? You could basically adopt the same approach there as you took here, save the scores in a list and raise an error when next_batch is called. Happy to talk more about that if you'd like.

That said, your implementation works great and satisfies the requirements, so its fine to merge from my perspective.

[angela97lin]

@dsherry I think you're right! I tried it but realized that IterativeAlgorithm doesn't know about the main objective (for printing the exception). While this probably wouldn't be difficult to grab, I'll keep this implementation for now :)

[dsherry]

@angela97lin ok cool, thanks for looking into it! I suppose we could either not mention the objective in the exception or modify the iterative algo further, but sounds good to me for the time being 👍