Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raise an error if all pipelines produce nan scores in automl #1015

Merged
merged 8 commits into from Aug 5, 2020

Conversation

angela97lin
Copy link
Contributor

@angela97lin angela97lin commented Aug 3, 2020

Closes #922 by checking if all pipelines in a batch return a score of NaN and stopping search if that is the case.

@codecov
Copy link

codecov bot commented Aug 3, 2020

Codecov Report

Merging #1015 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1015   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         181      181           
  Lines        9597     9628   +31     
=======================================
+ Hits         9584     9615   +31     
  Misses         13       13           
Impacted Files Coverage Δ
evalml/automl/automl_search.py 99.55% <100.00%> (+<0.01%) ⬆️
evalml/exceptions/exceptions.py 100.00% <100.00%> (ø)
evalml/tests/automl_tests/test_automl.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21eaa34...5cd4e33. Read the comment docs.

@angela97lin angela97lin self-assigned this Aug 4, 2020
@angela97lin angela97lin marked this pull request as ready for review Aug 4, 2020
Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

Hey @angela97lin Looks great! I have a question about the implementation. I think we should be resetting current_batch_pipeline_scores after each batch?

evalml/tests/automl_tests/test_automl.py Outdated Show resolved Hide resolved
evalml/automl/automl_search.py Outdated Show resolved Hide resolved
@angela97lin angela97lin added this to the August 2020 milestone Aug 4, 2020
@angela97lin angela97lin requested a review from freddyaboulton Aug 4, 2020
Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

@angela97lin Looks good to me!

evalml/tests/automl_tests/test_automl.py Show resolved Hide resolved
dsherry
dsherry approved these changes Aug 4, 2020
Copy link
Collaborator

@dsherry dsherry left a comment

@angela97lin I think this "was-batch-all-nan" check would fit better in IterativeAlgorithm, right? You could basically adopt the same approach there as you took here, save the scores in a list and raise an error when next_batch is called. Happy to talk more about that if you'd like.

That said, your implementation works great and satisfies the requirements, so its fine to merge from my perspective.

evalml/automl/automl_search.py Outdated Show resolved Hide resolved
evalml/automl/automl_search.py Outdated Show resolved Hide resolved
evalml/automl/automl_search.py Outdated Show resolved Hide resolved
evalml/tests/automl_tests/test_automl.py Show resolved Hide resolved
@angela97lin
Copy link
Contributor Author

angela97lin commented Aug 5, 2020

@dsherry I think you're right! I tried it but realized that IterativeAlgorithm doesn't know about the main objective (for printing the exception). While this probably wouldn't be difficult to grab, I'll keep this implementation for now :)

@angela97lin angela97lin merged commit 6749c3c into main Aug 5, 2020
2 checks passed
@angela97lin angela97lin deleted the 922_err branch Aug 5, 2020
@dsherry
Copy link
Collaborator

dsherry commented Aug 5, 2020

@angela97lin ok cool, thanks for looking into it! I suppose we could either not mention the objective in the exception or modify the iterative algo further, but sounds good to me for the time being 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Raise an error if all pipelines produce nan scores in automl
4 participants