Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling KeyboardInterrupt in AutoMLSearch #915

Merged
merged 7 commits into from
Jul 17, 2020

Conversation

freddyaboulton
Copy link
Contributor

@freddyaboulton freddyaboulton commented Jul 8, 2020

Pull Request Description

Fixes #250 by catching KeyboardInterrupt exceptions in AutoML search. I think the functionality matches what's described in the original ticket (see demo below) but I wanted to call out the following limitation:

  1. When the user interrupts a pipeline and decides to continue the search, that pipeline will be trained from scratch again. I don't think we can continue from where we left off because of how fit is implemented in sklearn/catboost/xgboost libraries.

Demo

keyboard_interrupt_demo_updated


After creating the pull request: in order to pass the changelog_updated check you will need to update the "Future Release" section of docs/source/changelog.rst to include this pull request by adding :pr:123.

@freddyaboulton freddyaboulton changed the title Adding KeyboardInterrupt and tests. Handling KeyboardInterrupt in AutoMLSearch Jul 8, 2020
@freddyaboulton freddyaboulton added this to the July 2020 milestone Jul 8, 2020
@codecov
Copy link

codecov bot commented Jul 8, 2020

Codecov Report

Merging #915 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #915   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         170      170           
  Lines        8593     8645   +52     
=======================================
+ Hits         8581     8633   +52     
  Misses         12       12           
Impacted Files Coverage Δ
evalml/automl/automl_search.py 99.31% <100.00%> (+0.04%) ⬆️
evalml/tests/automl_tests/test_automl.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7771cf2...4d1a407. Read the comment docs.

@freddyaboulton freddyaboulton marked this pull request as draft July 13, 2020 16:03
@freddyaboulton freddyaboulton force-pushed the 250-better-keyboard-interrupt branch from 8f210d3 to bbae31f Compare July 13, 2020 23:46
@freddyaboulton freddyaboulton changed the base branch from master to main July 13, 2020 23:47
@freddyaboulton freddyaboulton marked this pull request as ready for review July 16, 2020 14:22
Copy link
Collaborator

@jeremyliweishih jeremyliweishih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good - I agree with the implementation. Still taking a closer look but one thing is to add to the docs with a tutorial or explanation of how this feature works (maybe under "AUTOMATED MACHINE LEARNING").

@freddyaboulton freddyaboulton force-pushed the 250-better-keyboard-interrupt branch from e969c7b to c0b01d1 Compare July 16, 2020 21:48
@freddyaboulton
Copy link
Contributor Author

@jeremyliweishih I added a section to the AutoML tutorial on how to stop the search. Let me know if you want me to make any edits!

Copy link
Collaborator

@jeremyliweishih jeremyliweishih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@freddyaboulton LGTM. What do you think about adding your GIF onto the docs as well? I think it would look good but not blocking!

@freddyaboulton
Copy link
Contributor Author

@jeremyliweishih Good idea!

@freddyaboulton freddyaboulton merged commit f9db83f into main Jul 17, 2020
@freddyaboulton freddyaboulton deleted the 250-better-keyboard-interrupt branch July 17, 2020 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve exit from Keyboard interupt
2 participants