Implement RandomSearch and GridSearch Tuners #240

christopherbunn · 2019-11-22T22:00:10Z

Resolves #230

evalml/tuners/random_search_tuner.py

codecov · 2019-11-25T16:50:37Z

Codecov Report

Merging #240 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #240   +/-   ##
=======================================
  Coverage   98.58%   98.58%           
=======================================
  Files         111      111           
  Lines        3744     3744           
=======================================
  Hits         3691     3691           
  Misses         53       53

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b92ed5...5b92ed5. Read the comment docs.

jeremyliweishih

Looks like a good start! Can you also add in the AutoBase changes and some test cases for the next iteration?

evalml/tuners/grid_search_tuner.py

evalml/tuners/random_search_tuner.py

christopherbunn

Some additional questions:

evalml/tuners/grid_search_tuner.py

evalml/tuners/random_search_tuner.py

jeremyliweishih

Looking good, just need a little bit more on testing and a more graceful handling of the exception on the AutoML side. Adding a tutorial to the docs and API reference can come next as well.

evalml/models/auto_base.py

evalml/tests/tuners_tests/test_tuners.py

jeremyliweishih

Looking good, just need tests on AutoML side taking in new tuners and handling the exceptions.

christopherbunn · 2019-12-16T14:50:35Z

I discussed this issue with @jeremyliweishih last Friday, but the current structure as of now makes it difficult to pass in parameters for the tuner. For example, the GridSearchTuner has a parameter to determine how many points to sample from the search space. Here's two ways that we could pass in these parameters:

Rather than pass in the class type for the tuner, pass in a tuner object with the class defined
- This requires changing the way we pass in the space for each pipeline into the tuner
Create another parameter for AutoBase that takes in a dictionary of parameters for the tuner

The current implementation here hard codes these parameters for now. If we want to, we could merge this in to show that we have multiple tuners and then refactor them once we solidify #272.

jeremyliweishih · 2020-01-07T19:23:04Z

@dsherry aside of these issues, this PR should be ready to be merged. We should discuss how we want to proceed in the next planning meeting.

I discussed this issue with @jeremyliweishih last Friday, but the current structure as of now makes it difficult to pass in parameters for the tuner. For example, the GridSearchTuner has a parameter to determine how many points to sample from the search space. Here's two ways that we could pass in these parameters:

Rather than pass in the class type for the tuner, pass in a tuner object with the class defined

This requires changing the way we pass in the space for each pipeline into the tuner

Create another parameter for AutoBase that takes in a dictionary of parameters for the tuner

The current implementation here hard codes these parameters for now. If we want to, we could merge this in to show that we have multiple tuners and then refactor them once we solidify #272.

evalml/tuners/grid_search_tuner.py

dsherry · 2020-01-07T23:21:43Z

evalml/models/auto_base.py

+        except Exception as e:
+            self.parameter_exception = True
+            self.logger.log('\n✘ ' + str(e) + '\n')
+            return


Does this mean an error would show up in the terminal output, if we're using tqdm?

dsherry · 2020-01-31T22:30:18Z

evalml/automl/auto_base.py

+        try:
+            parameters = self._propose_parameters(pipeline_class)
+        except Exception as e:
+            raise e


Why do this?

We can't rethrow here. If the tuner says the search space has been exhausted, we need automl to handle that. I think we need to have that in this PR in order to merge.

I see two options to do that:

Update Tuner API. Add a Tuner.is_search_space_exhausted() member (name could be better). Have AutoBase._check_stopping_condition call this and return true if its true

Just update autobase code. Add an instance var _search_space_exhausted to AutoBase, and set that to true here when NoParamsException comes up the stack. Update AutoBase._check_stopping_condition() to return true if self._search_space_exhausted.

Option 1 feels cleaner. It does make each tuner's implementation more complex. But that's what I'd do. @christopherbunn lmk if you'd like to discuss this / talk about how to implement it.

I like option 1!

evalml/tuners/grid_search_tuner.py

dsherry · 2020-01-31T22:34:05Z

evalml/tuners/grid_search_tuner.py

+        Arguments:
+            points: The number of points to uniformly sample from \
+                Real dimensions.
+            random_state: Not used in grid search, kept for compatibility


This is missing a description of the space parameter

dsherry · 2020-01-31T22:38:10Z

evalml/tuners/grid_search_tuner.py

+class GridSearchTuner:
+    """Grid Search Optimizer"""
+
+    def __init__(self, space, points=10, random_state=None):


This is a nit-pick, but could we call points n_points instead? To me, the "n" prefix implies that this is a number, rather than a list of points or something

evalml/tuners/grid_search_tuner.py

dsherry · 2020-01-31T22:47:02Z

evalml/tuners/grid_search_tuner.py

+
+        Arguments:
+            points: The number of points to uniformly sample from \
+                Real dimensions.


What is meant by "Real dimensions"?

"Real dimensions" refers to a search space dimension that can take on any real value. Maybe it would be better to reword this to "The number of points to uniformly sample from Real search spaces"

Ah ok, thanks. Personally I find sklearn's docstring there to be unclear/cryptic. How about "the number of points to sample from along each dimension defined in the space argument" ?

Yeah I stand by what I said, would be nice to update this docstring a bit

dsherry · 2020-01-31T22:47:44Z

evalml/tuners/grid_search_tuner.py

+        Arguments:
+            points: The number of points to uniformly sample from \
+                Real dimensions.
+            random_state: Not used in grid search, kept for compatibility


I think you can just say "unused in this class"

dsherry · 2020-01-31T22:58:39Z

evalml/tuners/grid_search_tuner.py

+            else:
+                return Exception("Invalid dimension type in tuner")
+            raw_dimensions.append(range_values)
+        self.grid_points = list(itertools.product(*raw_dimensions))


By calling list() here, you're coercing the itertools.product generator into a list. That will cause all the combinations to be generated here, which could be harmful performance-wise as we increase n_points.

Fortunately, iterators are awesome! Here you can just say

self._grid_points = itertools.product(*raw_dimensions)

And then:

def propose(self): try: return next(self._grid_points) except StopIteration: raise NoParamsException("Grid search has exhausted all possible parameters.")

Here's a page from the python doc which shows the approximate code used to implement itertools.product. Note the use of yield, which adds an option to a generator rather than generating the output at that moment. Lazy computation ftw :)

dsherry · 2020-01-31T23:00:18Z

evalml/tuners/grid_search_tuner.py

+        self.grid_points = list(itertools.product(*raw_dimensions))
+
+    def add(self, parameters, score):
+        # Since this is a grid search, we don't need to store the results.


My personal preference is to delete this comment. In general I think code comments (distinct from docstrings) should only exist if they explain something which would be nearly impossible to discern from just reading the code. But that's just my opinion and I respect if you feel differently.

…arch

christopherbunn · 2020-03-20T19:47:28Z

Hey @dsherry and @jeremyliweishih, thanks for taking a look a few days ago. I incorporated most of the changes you all suggested. A few notes:

Re:

We can't rethrow here. If the tuner says the search space has been exhausted, we need automl to handle that. I think we need to have that in this PR in order to merge.

I see two options to do that:

Update Tuner API. Add a Tuner.is_search_space_exhausted() member (name could be better). Have AutoBase._check_stopping_condition call this and return true if its true

I gave updating the Tuner API a shot so that it handles this stopping condition more gracefully. I'm not sure if it's the most efficient method especially since it requires moving up getting the type of pipeline in AutoBase.check_stopping_condition() so I'm definitely looking for feedback on this section.

Re: pipeline mocking, I didn't really push much on this front. While it would definitely speed up performance I think it would be better to have a consistent mocking strategy by having a more broader implementation. The current tests are fairly minimal in terms of search & training. Imo, mocking fits better in a separate PR but again I'm open to thoughts/opinions.

jeremyliweishih

Looks good from the questions I had!

dsherry · 2020-03-24T17:44:52Z

evalml/automl/auto_base.py

+        # get new pipeline and check tuner
+        self._current_pipeline_class = self._select_pipeline()
+        if self.tuners[self._current_pipeline_class.name].is_search_space_exhausted():
+            return False


This looks great for now :) good going

dsherry · 2020-03-24T17:46:12Z

evalml/automl/auto_base.py

@@ -178,6 +178,11 @@ def search(self, X, y, feature_types=None, raise_errors=False, show_iteration_pl
        pbar.close()

    def _check_stopping_condition(self, start):
+        # get new pipeline and check tuner
+        self._current_pipeline_class = self._select_pipeline()


Great. Let's add this to __init__:

self._current_pipeline_class = None

dsherry · 2020-03-24T17:47:02Z

evalml/automl/auto_base.py

@@ -178,6 +178,11 @@ def search(self, X, y, feature_types=None, raise_errors=False, show_iteration_pl
        pbar.close()

    def _check_stopping_condition(self, start):
+        # get new pipeline and check tuner
+        self._current_pipeline_class = self._select_pipeline()


And can we rename this to _next_pipeline_class? I feel like that's a better description because it's the next class we'll be using in _do_iteration

But I acknowledge this is really just a nit-pick haha

dsherry

This looks great! Awesome work, particularly on updating auto_base.py

I agree with @jeremyliweishih 's suggestion about mocking in the tests. And I left a couple small comments. Otherwise, 🚢 :)

christopherbunn commented Nov 22, 2019

View reviewed changes

evalml/tuners/random_search_tuner.py Outdated Show resolved Hide resolved

christopherbunn requested a review from jeremyliweishih November 22, 2019 22:07

jeremyliweishih requested changes Nov 25, 2019

View reviewed changes

christopherbunn commented Nov 25, 2019

View reviewed changes

christopherbunn requested a review from jeremyliweishih November 26, 2019 16:40

jeremyliweishih requested changes Nov 26, 2019

View reviewed changes

christopherbunn requested a review from jeremyliweishih November 27, 2019 15:16

jeremyliweishih requested changes Nov 27, 2019

View reviewed changes

angela97lin mentioned this pull request Dec 10, 2019

AutoML Algorithm API #272

Closed

dsherry mentioned this pull request Jan 7, 2020

Add Random Search and Grid Search Tuners #230

Closed

dsherry reviewed Jan 7, 2020

View reviewed changes

evalml/tuners/grid_search_tuner.py Show resolved Hide resolved

dsherry reviewed Jan 7, 2020

View reviewed changes

christopherbunn requested a review from dsherry January 21, 2020 22:29

dsherry assigned christopherbunn Jan 22, 2020

christopherbunn changed the title ~~[WIP] Implement RandomSearch and GridSearch Tuners~~ Implement RandomSearch and GridSearch Tuners Jan 28, 2020

christopherbunn force-pushed the tuners branch 3 times, most recently from 7e635e7 to 3fdb7ac Compare January 31, 2020 16:49

dsherry reviewed Jan 31, 2020

View reviewed changes

evalml/tuners/grid_search_tuner.py Outdated Show resolved Hide resolved

dsherry reviewed Jan 31, 2020

View reviewed changes

evalml/tuners/grid_search_tuner.py Show resolved Hide resolved

dsherry reviewed Jan 31, 2020

View reviewed changes

christopherbunn added 16 commits March 20, 2020 14:46

Updated docstring, renamed n_points, and improved _grid_points access

4f5d448

Added tuners to API documentation

b4d2e36

Rebased tuners from Tuner class and updated docstrings

5590b05

Updated tuner tests

53b2c41

Fixed import order

622f391

Fixed docstring test errors

2bf6c63

Updated docstring with correct params

c2924f6

Updated docstring to show proper result

9e05379

Updated grid search docstrings

2d861ff

Restructured random search, including propose()

08aa541

Initial tuner test cleanup and changed type checking logic in grid se…

cb55b43

…arch

More test cleanup and fixed bug with discrete tuner types in grid search

0491302

Removed extraneous print statement

038595a

Separated param comparison into test_utils.py and updated docstrings

98e9e1a

Added search space check to tuners and tuner API

aaafe3c

Fixed lint errors

9b6844d

christopherbunn force-pushed the tuners branch from 6021d88 to 9b6844d Compare March 20, 2020 18:47

Fixed no_param test to use new parameter names

196794c

christopherbunn requested a review from jeremyliweishih March 20, 2020 19:47

jeremyliweishih approved these changes Mar 20, 2020

View reviewed changes

dsherry reviewed Mar 24, 2020

View reviewed changes

dsherry approved these changes Mar 24, 2020

View reviewed changes

Renamed to _next_pipeline_class and added to __init__

5b92ed5

christopherbunn merged commit 99a8ffd into master Mar 24, 2020

kmax12 unassigned christopherbunn May 20, 2020

dsherry deleted the tuners branch October 29, 2020 23:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement RandomSearch and GridSearch Tuners #240

Implement RandomSearch and GridSearch Tuners #240

christopherbunn commented Nov 22, 2019

codecov bot commented Nov 25, 2019 •

edited

Loading

jeremyliweishih left a comment

christopherbunn left a comment

jeremyliweishih left a comment

jeremyliweishih left a comment

christopherbunn commented Dec 16, 2019

jeremyliweishih commented Jan 7, 2020

dsherry Jan 7, 2020

dsherry Jan 31, 2020

dsherry Mar 11, 2020

jeremyliweishih Mar 16, 2020

dsherry Jan 31, 2020

dsherry Jan 31, 2020

dsherry Jan 31, 2020

christopherbunn Feb 7, 2020

dsherry Feb 7, 2020

dsherry Mar 10, 2020

dsherry Jan 31, 2020

dsherry Jan 31, 2020

dsherry Jan 31, 2020

christopherbunn commented Mar 20, 2020

jeremyliweishih left a comment

dsherry Mar 24, 2020

dsherry Mar 24, 2020

dsherry Mar 24, 2020

dsherry Mar 24, 2020

dsherry left a comment

Implement RandomSearch and GridSearch Tuners #240

Implement RandomSearch and GridSearch Tuners #240

Conversation

christopherbunn commented Nov 22, 2019

codecov bot commented Nov 25, 2019 • edited Loading

Codecov Report

jeremyliweishih left a comment

Choose a reason for hiding this comment

christopherbunn left a comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

christopherbunn commented Dec 16, 2019

jeremyliweishih commented Jan 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christopherbunn commented Mar 20, 2020

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 25, 2019 •

edited

Loading