Added string representation to AutoBase #675

christopherbunn · 2020-04-17T22:38:55Z

Before search is ran:

After search is ran:

Resolves #481

codecov · 2020-04-17T22:40:33Z

Codecov Report

Merging #675 into master will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #675      +/-   ##
==========================================
+ Coverage   99.29%   99.32%   +0.02%     
==========================================
  Files         140      140              
  Lines        4981     5018      +37     
==========================================
+ Hits         4946     4984      +38     
+ Misses         35       34       -1

Impacted Files	Coverage Δ
evalml/automl/auto_base.py	`97.50% <100.00%> (+0.44%)`	⬆️
evalml/tests/automl_tests/test_autobase.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 610e168...bcbf817. Read the comment docs.

christopherbunn · 2020-04-17T22:44:38Z

Notes:

I decided to add all parameters, even if they are not changed by the user.
The search results table drops the parameters column. Due to it's length, keeping this column would cause each row to look weird.
Not all of the parameters available were included in the test.

jeremyliweishih

Looking good! Some comments and needs more test coverage. Maybe removing the uncalled function will do the trick.

jeremyliweishih · 2020-04-24T16:43:30Z

docs/source/changelog.rst

@@ -34,6 +34,7 @@ Changelog
        * Changed requirements-parser to be a core dependency :pr:`673`
        * Replace `supported_problem_types` field on pipelines with `problem_type` attribute on base classes :pr:`678`
        * Update `ModelFamily` values: don't list xgboost/catboost as classifiers now that we have regression pipelines for them :pr:`677`
+        * Added __repr__ for AutoSearch object :pr:`675`


__str__ 😄

jeremyliweishih · 2020-04-24T17:33:52Z

evalml/automl/auto_base.py

@@ -100,6 +100,63 @@ def __init__(self, problem_type, tuner, cv, objective, max_pipelines, max_time,
            logger.log("Warning: unable to import plotly; skipping pipeline search plotting\n")
            self.plot = None

+    def __str__(self):
+        def _pipeline_names():


Kind of confused how this is different from _allowed_model_families() or how they are named. Is this for pipelines? or is _allowed_model_families

Actually seems like this isn't used.

jeremyliweishih · 2020-04-24T17:35:14Z

evalml/automl/auto_base.py

+                all_objectives += '\n\t'
+            return all_objectives[:-2]
+
+        def _allowed_model_families():


Should this just be Possible Pipelines? Same thing in searc_desc

angela97lin · 2020-05-01T22:13:53Z

evalml/automl/auto_base.py

+                all_objectives += objective.name
+                all_objectives += '\n\t'
+            return all_objectives[:-2]
+


Hmmmm... I think this could be simplified? To:

def _obj_names(): all_objectives = "" for objective in self.additional_objectives: all_objectives += "\n\t" + objective.name return all_objectives

Or I think using .join could also work here (with some check for last element)?

angela97lin · 2020-05-01T22:14:24Z

evalml/automl/auto_base.py

+                all_objectives += '\n\t'
+            return all_objectives[:-2]
+
+        def _possible_pipelines():


Same comment as for _obj_names :D

angela97lin · 2020-05-01T22:16:22Z

evalml/tests/automl_tests/test_autobase.py

+
+    automl.search(X, y, raise_errors=False)
+    str_rep = str(automl)
+    # str_rep = str_rep.lower()


Leftover comment here :o?

angela97lin

Nice! I just left a comment about simplifying some functions, but otherwise LGTM :)

jeremyliweishih

LGTM if you address Angela's comments!

Lint fixes

dsherry · 2020-05-04T22:45:32Z

evalml/automl/auto_base.py

+            f"Detect Label Leakage: {self.detect_label_leakage}\n"
+            f"Start Iteration Callback: {_get_funct_name(self.start_iteration_callback)}\n"
+            f"Add Result Callback: {_get_funct_name(self.add_result_callback)}\n"
+            f"Additional Objectives: {_print_list(self.additional_objectives)}\n"


What will this print out?

This will print out all of the additional objectives to score on in an indented list format. Here's an example:

dsherry · 2020-05-04T22:49:35Z

evalml/automl/auto_base.py

+            f"Objective: {self.objective.name}\n"
+            f"Max Time: {self.max_time}\n"
+            f"Max Pipelines: {self.max_pipelines}\n"
+            f"Possible Pipelines: {_print_list(self.possible_pipelines)}\n"


What will this print out?

In general, the _print_list function will go through any iterable object with a name attribute and print out a list of their names.

This call will print out all of the possible pipelines in an indented list format. Here's an example:

dsherry · 2020-05-04T22:53:51Z

evalml/automl/auto_base.py

+        try:
+            rankings_str = self.rankings.drop(['parameters'], axis='columns').to_string()
+            rankings_desc = f"\nSearch Results: \n{'='*20}\n{rankings_str}"
+        except KeyError:


@christopherbunn when would this happen? And is there any way to restructure this code to avoid having a try/except? Its nice to avoid these whenever possible because they make the flow harder to follow.

Could we do something like this:

rankings = self.rankings if 'parameters' not in rankings.columns: return search_desc return search_desc + "\nSearch Results: \n{'='*20}\n{rankings_str}".format(rankings['parameters'])

@dsherry The goal of this snippet was to only include the rankings after the search process has been ran. The reason I had the try-catch is that rankings is a @property and this KeyError is what I get when I try to call automl.rankings before any searches is ran. Is there a better way to check if the search process has ran yet?

@christopherbunn nice find. Could you please file this as a bug? :)

I suppose we could add a @property which is something like AutoBase.has_run. That way we could have whatever implementation we want for that method. Not critical though.

dsherry · 2020-05-04T22:56:15Z

evalml/tests/automl_tests/test_autobase.py

+
+
+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_automl_str_search(mock_fit, X_y):


@christopherbunn are there any edge cases which this test doesn't currently cover? For instance, is there any way a None value could be provided as the input to one of the fields you apply your _print_list and _get_funct_name helpers to?

christopherbunn mentioned this pull request Apr 17, 2020

[WIP] Added __repr__ for AutoSearch #563

Closed

christopherbunn changed the title ~~Added string representation to AutoBase~~ [WIP] Added string representation to AutoBase Apr 17, 2020

auto-assign bot assigned christopherbunn Apr 17, 2020

christopherbunn requested a review from jeremyliweishih April 17, 2020 22:46

christopherbunn force-pushed the 481_str_rep_search branch from 9a78d24 to 4b230e4 Compare April 21, 2020 13:35

jeremyliweishih requested changes Apr 24, 2020

View reviewed changes

christopherbunn force-pushed the 481_str_rep_search branch 3 times, most recently from 56565df to e4cc0e0 Compare April 29, 2020 16:53

christopherbunn requested review from jeremyliweishih, angela97lin and dsherry May 1, 2020 20:34

angela97lin reviewed May 1, 2020

View reviewed changes

angela97lin approved these changes May 1, 2020

View reviewed changes

jeremyliweishih approved these changes May 4, 2020

View reviewed changes

christopherbunn added 9 commits May 4, 2020 11:06

Added string representation to AutoBase

57d4f31

Updated changelog

d67a3d7

Lint fixes

Removed extra code and updated name to "Possible Pipelines"

0892bd4

Added additional parameters for str rep test

110e495

Resorted imports

800fdc3

Changed to random forest in autobase test to satisfy minimum tests

6624b02

Added function callback test

d75194a

Changed result parameter to not be covered

6e70fac

Simpiflied list printing logic

da93f2a

christopherbunn force-pushed the 481_str_rep_search branch from dd1ecd9 to da93f2a Compare May 4, 2020 15:26

Removed extra comment in test

bcbf817

christopherbunn changed the title ~~[WIP] Added string representation to AutoBase~~ Added string representation to AutoBase May 4, 2020

christopherbunn merged commit c75417d into master May 4, 2020

dsherry reviewed May 4, 2020

View reviewed changes

christopherbunn mentioned this pull request May 5, 2020

KeyError for automl.rankings when no search is ran #744

Closed

christopherbunn mentioned this pull request May 18, 2020

Added None test case to AutoBase __str__ representation #783

Merged

kmax12 unassigned christopherbunn May 20, 2020

angela97lin mentioned this pull request May 29, 2020

Release v0.10.0 May 29, 2020 #822

Merged

dsherry deleted the 481_str_rep_search branch October 29, 2020 23:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added string representation to AutoBase #675

Added string representation to AutoBase #675

christopherbunn commented Apr 17, 2020 •

edited

Loading

codecov bot commented Apr 17, 2020 •

edited

Loading

christopherbunn commented Apr 17, 2020

jeremyliweishih left a comment •

edited

Loading

jeremyliweishih Apr 24, 2020

jeremyliweishih Apr 24, 2020

jeremyliweishih Apr 24, 2020

jeremyliweishih Apr 24, 2020

angela97lin May 1, 2020

angela97lin May 1, 2020

angela97lin May 1, 2020

angela97lin left a comment

jeremyliweishih left a comment

dsherry May 4, 2020

christopherbunn May 5, 2020

dsherry May 4, 2020

christopherbunn May 5, 2020

dsherry May 4, 2020

christopherbunn May 5, 2020

dsherry May 5, 2020

dsherry May 4, 2020 •

edited

Loading



		@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
		def test_automl_str_search(mock_fit, X_y):

Added string representation to AutoBase #675

Added string representation to AutoBase #675

Conversation

christopherbunn commented Apr 17, 2020 • edited Loading

codecov bot commented Apr 17, 2020 • edited Loading

Codecov Report

christopherbunn commented Apr 17, 2020

jeremyliweishih left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry May 4, 2020 • edited Loading

Choose a reason for hiding this comment

christopherbunn commented Apr 17, 2020 •

edited

Loading

codecov bot commented Apr 17, 2020 •

edited

Loading

jeremyliweishih left a comment •

edited

Loading

dsherry May 4, 2020 •

edited

Loading