Stop AutoMLSearch from running multiple times - #1647
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1647 +/- ##
=========================================
+ Coverage 100.0% 100.0% +0.1%
=========================================
Files 240 240
Lines 18271 18287 +16
=========================================
+ Hits 18263 18279 +16
Misses 8 8
Continue to review full report at Codecov.
|
| # Baseline + first batch + each pipeline iteration + 1 | ||
| first_ensembling_iteration = (1 + len(self.allowed_pipelines) + len(self.allowed_pipelines) * self._pipelines_per_batch + 1) | ||
| if self.max_iterations < first_ensembling_iteration: | ||
| if not self._searched: |
There was a problem hiding this comment.
Unsure why it changed this much, but I added this if statement to determine if we need to do the search, then put the rest of the search within this if statement
There was a problem hiding this comment.
Have you considered instead just returning instead of true? Saves a little nesting of conditionals and might make the diff a little clearer.
There was a problem hiding this comment.
Agreed with @chukarsten's comment! Makes it easier to follow along
|
|
||
| best_pipeline = self.rankings.iloc[0] | ||
| best_pipeline_name = best_pipeline["pipeline_name"] | ||
| self._find_best_pipeline() |
There was a problem hiding this comment.
I find the best pipeline here
| self._find_best_pipeline() | ||
| logger.info(f"Best pipeline: {best_pipeline_name}") | ||
| logger.info(f"Best pipeline {self.objective.name}: {best_pipeline['score']:3f}") | ||
| self._searched = True |
There was a problem hiding this comment.
and I set searched to be true here so we won't redo the same search
| else: | ||
| logger.info("AutoMLSearch has already been run and will not run again on the same instance. Re-initialize AutoMLSearch to search again.") | ||
|
|
||
| def _find_best_pipeline(self): |
There was a problem hiding this comment.
New method to find the best pipeline and train only if the pipeline is new
| """ | ||
| if self._searched: | ||
| logger.info("AutoMLSearch has already been run and will not run again on the same instance. Re-initialize AutoMLSearch to search again.") | ||
| return |
There was a problem hiding this comment.
@bchen1116 hmm after our discussion yesterday I thought we were going with option 3 from the issue writeup, not option 2. Did I remember wrong?
There was a problem hiding this comment.
@dsherry, I voiced for option 2 since the X_train, y_train data are now passed through the init of AutoMLSearch, and during the discussion yesterday, I believe everyone agreed, which is why I proceeded with option 2. Do you think option 3 would be better? My main issue with option 3 is that rerunning the same AutoMLSearch class, with the same data and defaults, should lead to the same results, and therefore isn't very useful. Additionally, clearing the previous state could be tricky, especially if a user called add_to_rankings previously. Let me know your thoughts, and we can also discuss more after standup if needed!
There was a problem hiding this comment.
That's entirely fair. Unless we provided a way for people to reconfigure the automl settings on a single automl search instance, rerunning the search from scratch should produce highly similar results. 👍
Sounds good to me. Just wanted to make sure I wasn't missing something.
| * ``add_to_rankings`` updates ``AutoMLSearch.best_pipeline`` when necessary :pr:`1647` | ||
| * Fixed bug where time series baseline estimators were not receiving ``gap`` and ``max_delay`` in ``AutoMLSearch`` :pr:`1645` | ||
| * Changes | ||
| * Rerunning search for ``AutoMLSearch`` results in a message thrown rather than failing the search, and removed ``has_searched`` property :pr:`1647` |
There was a problem hiding this comment.
Please add the removal of has_searched to the breaking changes section as well.
| Disabled by default in non-Jupyter enviroments. | ||
| """ | ||
| if self._searched: | ||
| logger.info("AutoMLSearch has already been run and will not run again on the same instance. Re-initialize AutoMLSearch to search again.") |
There was a problem hiding this comment.
Nit-pick: "AutoMLSearch.search() has already been run..."
4 participants
fix #1424
has_searchedproperty fromAutoMLSearchbest_pipelineinadd_to_rankingsas wellAutoMLSearch.searchtwiceUpdated search call
The files changed look like I changed a lot, but I commented on the changes I made from the main branch and this branch in
automl_search.py