WW Accessor: Updating objectives by freddyaboulton · Pull Request #2185 · alteryx/evalml

freddyaboulton · 2021-04-23T17:06:52Z

Pull Request Description

Updates objectives (and confusion_matrix implementation for the cost-benefit objective). Had to manually deselect some tests because they actually run automl.

See the roadmap here

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

freddyaboulton · 2021-04-23T17:08:40Z

 from evalml.objectives import LeadScoring


-def test_lead_scoring_objective(X_y_binary):


This test does two things: run automl with lead scoring and test it on small data not related to automl so I'm splitting it into two so we can cover the part that doesn't run automl.

Thanks for the explanation!

codecov · 2021-04-23T17:09:10Z

Codecov Report

Merging #2185 (7baf3dc) into 2035-use-ww-accessor (725320d) will increase coverage by 3.9%.
The diff coverage is 86.7%.

@@                  Coverage Diff                   @@
##           2035-use-ww-accessor   #2185     +/-   ##
======================================================
+ Coverage                  27.4%   31.3%   +3.9%     
======================================================
  Files                       295     295             
  Lines                     24336   24335      -1     
======================================================
+ Hits                       6657    7594    +937     
+ Misses                    17679   16741    -938

Impacted Files	Coverage Δ
.../tests/objective_tests/test_cost_benefit_matrix.py	`88.4% <33.4%> (+88.4%)`	⬆️
evalml/model_understanding/graphs.py	`14.9% <100.0%> (+3.9%)`	⬆️
evalml/objectives/objective_base.py	`100.0% <100.0%> (+50.0%)`	⬆️
...alml/tests/objective_tests/test_fraud_detection.py	`92.3% <100.0%> (+92.3%)`	⬆️
evalml/tests/objective_tests/test_lead_scoring.py	`87.7% <100.0%> (+87.7%)`	⬆️
evalml/tests/objective_tests/test_sla.py	`91.7% <100.0%> (+91.7%)`	⬆️
evalml/tests/conftest.py	`44.5% <0.0%> (+1.7%)`	⬆️
evalml/objectives/standard_metrics.py	`100.0% <0.0%> (+16.8%)`	⬆️
evalml/objectives/utils.py	`100.0% <0.0%> (+22.8%)`	⬆️
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 725320d...7baf3dc. Read the comment docs.

freddyaboulton · 2021-04-23T17:09:11Z

-	--ignore evalml/tests/model_understanding_tests --ignore evalml/tests/objective_tests \
-	-k "not test_save"
+	--ignore evalml/tests/model_understanding_tests \
+	-k "not test_save and not objective_test_uses_automl"


Needed to deselect the objective tests that run automl

jeremyliweishih

lgtm once tests pass!

dsherry

dsherry · 2021-04-23T19:19:01Z

-    y_true = _convert_woodwork_types_wrapper(y_true.to_series()).to_numpy()
-    y_predicted = _convert_woodwork_types_wrapper(y_predicted.to_series()).to_numpy()
+    y_true = y_true.to_numpy()
+    y_predicted = y_predicted.to_numpy()


👏 love it!!

dsherry · 2021-04-23T19:21:45Z

-    assert not np.isnan(predictions.to_series()).values.any()
-    assert not np.isnan(pipeline.predict_proba(X).to_dataframe()).values.any()
+    assert not np.isnan(predictions).values.any()
+    assert not np.isnan(pipeline.predict_proba(X)).values.any()


Lol go home codecov you're drunk 😆

Was this change required by lint? Or just seemed like a good idea? I understand you'll get this and the other objective_test_uses_automl tests green in a separate PR.

It just seemed like a good idea cause I'm now trained to delete all to_series() and to_dataframe() that I see 😂

dsherry · 2021-04-23T19:22:38Z


-    out = ww.DataColumn(fraud_cost.decision_function(y_predicted, 5, extra_columns))
-    pd.testing.assert_series_equal(out.to_series(), y_true, check_dtype=False, check_names=False)
+    out = pd.Series(fraud_cost.decision_function(y_predicted, 5, extra_columns))


Is this necessary? I would assume the output of decision_function is already a pandas series

It's not! Good catch

dsherry · 2021-04-23T19:22:53Z

 from evalml.objectives import LeadScoring


-def test_lead_scoring_objective(X_y_binary):


Thanks for the explanation!

* WW Accessor: Update demo datasets, preprocessing, and utils (#2172) * Update demos, utils, and preprocessing * Updating make test commands * Update ww requirement * Add test that infer_feature_types preserves schema * Add test that infer_feature_types raises errors with invalid schema * load_data always returns woodwork info. Deleted return_pandas * Updating docstrings * Release notes for first PR * Deleting redundant tests * update test name * WW 0.2.0: Update Data Checks (#2182) * Updating data checks * Undo accidental edit * Fixing typo where we assign to ww.init * WW Accessor: Updating objectives (#2185) * Updating objectives * Deleting superfluous pd.Series * WW Accessor: Updating Components (#2191) * Update components - first commit * Update delayed_features_transformer to not use assign * Fixing tests * Skipping Boolean with Nan test in imputers * Fixing base sampler _prepare_data * Fixing target imputer null bool test * Fix test skips * Addressing comments * Clean up sampler tests * Editing docstrings * Update to ww 0.3.0 * Fixing components that didn't have merge conflicts * WW 0.2.0 Update: Updating pipelines (#2205) * Updating pipelines * Fixing docstrings * Fix stacked test * Add tests to check input data not modified * Use rename in component graph * WW Accessor: Update model understanding (#2247) * Update model understanding module * Removing unused import * WW Accessor: Update Automl (#2243) * Updating pipelines * Fixing docstrings * Fix stacked test * Update automl * Update to preserve schema * Preserve schema in train_pipelines/score_pipelines * No ww drop * Update standard scaler * Use weak ref feture branch * Add tests to check input data not modified * Use weak-ref branch * Use weak ref branch * Fixing tests * Set ww back to 0.3.0 * Adding objective tests that use automl * Add back minimal-dependencies-flag * Fix test command * Updating docstrings adding dask test to check schema is sent * Fixing import order in test_dask_engine * Fixing last remaining docstrings * Fix typo in ignore command * Use ww init series * No doctest modules yet * Fix docstrings & use schemas stored in automl_config * WW Update: Update docs and docstrings (#2292) * Fixing docs and docstrings * Update docstring in highly_null_data_check * Delete warning from init * Remove print(X.ww) * Fixing notebook python version :( * Update release notes * Delete _convert_woodwork_types_wrapper completely * Fixing coverage * Add back standard scaler try * Remove _convert_woodwork_types_wrapper from docs' * Linting docs * Merging main * Fix release notes

Updating objectives

a55af10

freddyaboulton commented Apr 23, 2021

View reviewed changes

freddyaboulton marked this pull request as ready for review April 23, 2021 17:14

auto-assign Bot assigned freddyaboulton Apr 23, 2021

freddyaboulton requested review from ParthivNaresh, angela97lin, bchen1116, chukarsten, dsherry and jeremyliweishih April 23, 2021 17:14

jeremyliweishih approved these changes Apr 23, 2021

View reviewed changes

freddyaboulton changed the title ~~Updating objectives~~ WW Accessor: Updating objectives Apr 23, 2021

dsherry approved these changes Apr 23, 2021

View reviewed changes

Deleting superfluous pd.Series

7baf3dc

freddyaboulton merged commit 61b8443 into 2035-use-ww-accessor Apr 23, 2021

freddyaboulton deleted the 2035-update-objectives branch June 3, 2021 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WW Accessor: Updating objectives#2185

WW Accessor: Updating objectives#2185
freddyaboulton merged 2 commits into2035-use-ww-accessorfrom
2035-update-objectives

freddyaboulton commented Apr 23, 2021

Uh oh!

freddyaboulton Apr 23, 2021 •

edited

Loading

Uh oh!

dsherry Apr 23, 2021

Uh oh!

codecov Bot commented Apr 23, 2021 •

edited

Loading

Uh oh!

freddyaboulton Apr 23, 2021

Uh oh!

jeremyliweishih left a comment

Uh oh!

dsherry left a comment

Uh oh!

dsherry Apr 23, 2021

Uh oh!

dsherry Apr 23, 2021

Uh oh!

freddyaboulton Apr 23, 2021

Uh oh!

dsherry Apr 23, 2021

Uh oh!

freddyaboulton Apr 23, 2021

Uh oh!

dsherry Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		from evalml.objectives import LeadScoring


		def test_lead_scoring_objective(X_y_binary):

Conversation

freddyaboulton commented Apr 23, 2021

Pull Request Description

Uh oh!

freddyaboulton Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremyliweishih left a comment

Choose a reason for hiding this comment

Uh oh!

dsherry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

freddyaboulton Apr 23, 2021 •

edited

Loading

codecov Bot commented Apr 23, 2021 •

edited

Loading