Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AutoML: add "search order" to rankings and full_rankings #2345

Merged
merged 12 commits into from Jun 16, 2021

Conversation

angela97lin
Copy link
Contributor

Closes #1841

@angela97lin angela97lin self-assigned this Jun 7, 2021
@codecov
Copy link

codecov bot commented Jun 8, 2021

Codecov Report

Merging #2345 (122e6c5) into main (861870e) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #2345     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        281     281             
  Lines      24923   24925      +2     
=======================================
+ Hits       24826   24828      +2     
  Misses        97      97             
Impacted Files Coverage Δ
evalml/tests/automl_tests/test_automl.py 99.7% <ø> (ø)
evalml/automl/automl_search.py 99.4% <100.0%> (+0.1%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 861870e...122e6c5. Read the comment docs.

Copy link
Contributor

@jeremyliweishih jeremyliweishih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but just have one testing question. If that case isn't covered we should add it.

@@ -160,6 +160,7 @@ def test_search_results(X_y_regression, X_y_binary, X_y_multi, automl_type, obje
np.dtype("float64"),
np.dtype("bool"),
np.dtype("O"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do any of these tests check if "search_order" has the correct values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeremyliweishih Line 109 checks: assert automl.results["search_order"] == [0, 1] :)

Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angela97lin Looks good!

if not self._results["pipeline_results"]:
return pd.DataFrame(columns=full_rankings_cols)

rankings_df = pd.DataFrame(self._results["pipeline_results"].values())
rankings_df = rankings_df[full_rankings_cols]
rankings_df = rankings_df[pipeline_results_cols]
rankings_df["search_order"] = pd.Series(self._results["search_order"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean search order is the last column? Might make more sense to put it next to the pipeline name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah :'( we get the other info from pipeline_results so I didn't want to break that up, but I think this makes more sense too

Copy link
Contributor

@bchen1116 bchen1116 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@angela97lin angela97lin merged commit 2e1d8ab into main Jun 16, 2021
@angela97lin angela97lin deleted the 1841_search_order branch June 16, 2021 00:59
@chukarsten chukarsten mentioned this pull request Jun 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AutoML: add "search order" to rankings and full_rankings
5 participants