-
Notifications
You must be signed in to change notification settings - Fork 89
Updated _evaluate_pipelines to consolidate side effects #1410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@christopherbunn a couple ideas for things to try right off the bat:
|
Could also try disabling the |
29a0f0a
to
22a7fe8
Compare
Codecov Report
@@ Coverage Diff @@
## main #1410 +/- ##
=========================================
- Coverage 100.0% 100.0% -0.0%
=========================================
Files 223 223
Lines 15139 15135 -4
=========================================
- Hits 15132 15128 -4
Misses 7 7
Continue to review full report at Codecov.
|
@christopherbunn and I spent a while this afternoon debugging the intermittent failures on this PR. Its still not 100% clear what's causing the problem here, but we have more evidence. For tl;dr see next steps at the bottom. UPDATE: filed Notes
Sidenote: we noticed around 500 tests are getting marked as skipped, filed as #1437 . Here's what the memory usage over time for the unit tests looked like on And here's what that looked like for @christopherbunn 's branch in this PR (rewound to commit Observations
Thoughts
Next steps
|
74d30cd
to
22a7fe8
Compare
@freddyaboulton @christopherbunn @rpeck and I have been looking into the memory issues in #1438 . Updates posted on that issue. Next steps for this PR:
|
49b81c8
to
817d522
Compare
7691f5d
to
c1b1720
Compare
After #1447 was merged into main, I was able to rerun the Python linux tests three times and have it successfully pass each time. All three runs should be grouped under the current checks. |
b9fdee7
to
7425573
Compare
evalml/automl/automl_search.py
Outdated
|
||
if search_iteration_plot: | ||
search_iteration_plot.update() | ||
if search_iteration_plot: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We call update()
here and in self._evaluate_pipelines
. I don't think we need both? I don't think this anything to do with the memory issues we've seen on this branch but I think it'd be good to only update the plot once if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Originally I wanted to be able to make sure that all of the results were plotted if the search was interrupted early. However, I did just test it and the one instance in self._evaluate_pipelines
was sufficient so I'll remove the redundant one in search()
.
7425573
to
284aad7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me @christopherbunn ! 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@christopherbunn thanks for sticking with this! It brought us down an interesting rabbit hole RE #1438 . Let's get it merged!
284aad7
to
9624243
Compare
Going off of #1337, but this will have fixes to parallel memory usage.
Resolves #1295