Move SHAP explainers out of experimental #3596

dantegd · 2021-03-10T03:37:01Z

PR isolates tests improvements from general SHAP work, and is being used for debugging SHAP in CI.

…-enh-shap-plots

JohnZed

Looks great. I mostly have just small questions. Also, let's move this sucker out of experimental!

JohnZed · 2021-03-26T00:06:47Z

python/cuml/test/conftest.py

+    return create_synthetic_dataset(generator=skl_make_clas,
+                                    n_samples=101,
+                                    n_features=11,
+                                    test_size=1,


test_size=1 seems like it may hide some bugs potentially? I know you're trying to save on runtime but could we get to 2 or 3?

@dantegd ping on this one? Maybe I'm just misunderstanding

It was missing, just added a bigger test size to the exact tests of both perm and kernel (had to update the precalculated values because the seed changed when changing the sizes here)

JohnZed · 2021-03-26T00:10:13Z

python/cuml/test/experimental/test_explainer_kernel_shap.py

@@ -69,9 +53,13 @@ def experimental_test_and_log(cu_shap_values,
        print("golden_result_values")


Is this print meant to stay in the final version even when no assert hits?

python/cuml/test/experimental/test_explainer_kernel_shap.py

python/cuml/test/experimental/test_explainer_permutation_shap.py

python/cuml/test/experimental/test_shap_plotting.py

…19-enh-shap-plots

…ed to API docs

Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>

dantegd · 2021-04-01T21:34:30Z

rerun tests

JohnZed

One stray test print to remove but otherwise great!

JohnZed · 2021-04-01T23:07:55Z

python/cuml/test/explainer/test_explainer_permutation_shap.py

        npermutations=1,
        testing=True
    )

+    print(shap_values)


Stray print

mike-wendt

From the failed GPU builds:

CondaValueError: invalid package specification: shap>=0.37,

This will need to be updated

ci/gpu/build.sh

mike-wendt · 2021-04-02T15:20:43Z

ci/gpu/build.sh

@@ -58,7 +58,8 @@ gpuci_conda_retry install -c conda-forge -c rapidsai -c rapidsai-nightly -c nvid
      "xgboost=1.3.3dev.rapidsai${MINOR_VERSION}" \
      "rapids-build-env=${MINOR_VERSION}.*" \
      "rapids-notebook-env=${MINOR_VERSION}.*" \
-      "rapids-doc-env=${MINOR_VERSION}.*"
+      "rapids-doc-env=${MINOR_VERSION}.*" \
+      "shap>=0.37,<=0.39"


This will include version 0.39 as well. Did you want to exclude 0.39 or include it @dantegd

Include it, the original had the wrong blank space because I accidentally replaced the <= for < in a bulk replace when I was doing something else, thanks for the notice!

…19-enh-shap-plots

codecov-io · 2021-04-02T21:21:15Z

Codecov Report

Merging #3596 (4b10e49) into branch-0.19 (fd9ec89) will increase coverage by 2.31%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           branch-0.19    #3596      +/-   ##
===============================================
+ Coverage        80.70%   83.02%   +2.31%     
===============================================
  Files              227      227              
  Lines            17615    17712      +97     
===============================================
+ Hits             14217    14706     +489     
+ Misses            3398     3006     -392

Flag	Coverage Δ
dask	`45.62% <ø> (+0.63%)`	⬆️
non-dask	`75.11% <ø> (+2.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/cuml/__init__.py	`95.58% <ø> (+0.20%)`	⬆️
...cuml/_thirdparty/sklearn/preprocessing/__init__.py	`100.00% <ø> (ø)`
...on/cuml/_thirdparty/sklearn/preprocessing/_data.py	`64.40% <ø> (+1.29%)`	⬆️
...hirdparty/sklearn/preprocessing/_discretization.py	`83.33% <ø> (-0.88%)`	⬇️
...l/_thirdparty/sklearn/preprocessing/_imputation.py	`85.54% <ø> (+22.74%)`	⬆️
python/cuml/_thirdparty/sklearn/utils/extmath.py	`56.89% <ø> (ø)`
...cuml/_thirdparty/sklearn/utils/skl_dependencies.py	`80.00% <ø> (+26.07%)`	⬆️
...ython/cuml/_thirdparty/sklearn/utils/validation.py	`18.41% <ø> (-4.04%)`	⬇️
python/cuml/cluster/__init__.py	`100.00% <ø> (ø)`
python/cuml/cluster/agglomerative.pyx	`96.47% <ø> (ø)`
... and 181 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d59722...4b10e49. Read the comment docs.

dantegd · 2021-04-02T21:41:39Z

@gpucibot merge

dantegd added 2 commits March 9, 2021 21:34

ENH Plot tests and test cleanups

32487bc

DBG Add conda-forge shap to CI for testing

b61f6a0

dantegd added the 2 - In Progress Currenty a work in progress label Mar 10, 2021

dantegd added this to PR-WIP in v0.19 Release via automation Mar 10, 2021

github-actions bot added Cython / Python Cython or Python issue gpuCI gpuCI issue labels Mar 10, 2021

dantegd added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change and removed Cython / Python Cython or Python issue gpuCI gpuCI issue labels Mar 10, 2021

dantegd mentioned this pull request Mar 10, 2021

[REVIEW] Additional distance metrics for ANN #3533

Merged

FIX comment print to fix pep8

f5e188d

github-actions bot added Cython / Python Cython or Python issue gpuCI gpuCI issue labels Mar 10, 2021

dantegd added 4 commits March 10, 2021 14:53

DBG Add pip shap to CI for testing

a1b5583

Merge branch '019-enh-shap-plots' of github.com:dantegd/cuml into 019…

af3dc79

…-enh-shap-plots

Merge branch-0.19 into 019-enh-shap-plots

ed059e5

ENH Multiple enhancements

6c55212

dantegd added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currenty a work in progress labels Mar 25, 2021

JohnZed self-requested a review March 25, 2021 23:36

JohnZed reviewed Mar 26, 2021

View reviewed changes

dantegd added 4 - Waiting on Author Waiting for author to respond to review and removed 3 - Ready for Review Ready for review by team labels Mar 26, 2021

dantegd added 4 commits March 30, 2021 11:31

Merge branch 'branch-0.19' of https://github.com/rapidsai/cuml into 0…

0a377d5

…19-enh-shap-plots

ENH Move explainers out of experimental

b29eaed

FIX Remove not needed function

6f2dfaa

ENH Multiple enhancements, corrections to remove experimental and add…

d5761fe

…ed to API docs

dantegd changed the title ~~SHAP plotting tests and improvements~~ Move SHAP explainers out of experimental Apr 1, 2021

FIX The smallest copyright fix so far...

688b2a0

v0.19 Release automation moved this from PR-WIP to PR-Needs review Apr 1, 2021

dantegd changed the title ~~Move SHAP explainers out of experimental~~ Move SHAP explainers out of experimental [skip-ci] Apr 1, 2021

dantegd and others added 5 commits April 1, 2021 14:33

Update python/cuml/explainer/kernel_shap.pyx

68475ba

Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>

Update python/cuml/explainer/kernel_shap.pyx

8b467d2

Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>

Update python/cuml/explainer/kernel_shap.pyx

e48ddaf

Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>

Update python/cuml/explainer/permutation_shap.pyx

0b5a4b0

Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>

FIX Add more samples to pytests and fix a bug in permutation shap

8785f19

dantegd marked this pull request as ready for review April 1, 2021 21:14

dantegd requested review from a team as code owners April 1, 2021 21:14

dantegd added 4 - Waiting on Reviewer Waiting for reviewer to review or respond and removed 4 - Waiting on Author Waiting for author to respond to review labels Apr 1, 2021

dantegd changed the title ~~Move SHAP explainers out of experimental [skip-ci]~~ Move SHAP explainers out of experimental Apr 1, 2021

JohnZed approved these changes Apr 1, 2021

View reviewed changes

v0.19 Release automation moved this from PR-Needs review to PR-Reviewer approved Apr 2, 2021

mike-wendt approved these changes Apr 2, 2021

View reviewed changes

mike-wendt reviewed Apr 2, 2021

View reviewed changes

mike-wendt self-requested a review April 2, 2021 15:06

mike-wendt reviewed Apr 2, 2021

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

FIX gpu ci build script and stray print

3c51ae1

mike-wendt reviewed Apr 2, 2021

View reviewed changes

dantegd added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Waiting on Reviewer Waiting for reviewer to review or respond labels Apr 2, 2021

dantegd added 2 commits April 2, 2021 14:03

Merge branch 'branch-0.19' of https://github.com/rapidsai/cuml into 0…

d834754

…19-enh-shap-plots

FIX temporarily xfail hellinger pytest

4b10e49

rapids-bot bot merged commit 1554f14 into rapidsai:branch-0.19 Apr 2, 2021

v0.19 Release automation moved this from PR-Reviewer approved to Done Apr 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move SHAP explainers out of experimental #3596

Move SHAP explainers out of experimental #3596

dantegd commented Mar 10, 2021

JohnZed left a comment

JohnZed Mar 26, 2021

JohnZed Apr 1, 2021

dantegd Apr 1, 2021

JohnZed Mar 26, 2021

dantegd commented Apr 1, 2021

JohnZed left a comment

JohnZed Apr 1, 2021

mike-wendt left a comment

mike-wendt Apr 2, 2021

dantegd Apr 2, 2021

codecov-io commented Apr 2, 2021

dantegd commented Apr 2, 2021

		@@ -69,9 +53,13 @@ def experimental_test_and_log(cu_shap_values,
		print("golden_result_values")

Move SHAP explainers out of experimental #3596

Move SHAP explainers out of experimental #3596

Conversation

dantegd commented Mar 10, 2021

JohnZed left a comment

Choose a reason for hiding this comment

JohnZed Mar 26, 2021

Choose a reason for hiding this comment

JohnZed Apr 1, 2021

Choose a reason for hiding this comment

dantegd Apr 1, 2021

Choose a reason for hiding this comment

JohnZed Mar 26, 2021

Choose a reason for hiding this comment

dantegd commented Apr 1, 2021

JohnZed left a comment

Choose a reason for hiding this comment

JohnZed Apr 1, 2021

Choose a reason for hiding this comment

mike-wendt left a comment

Choose a reason for hiding this comment

mike-wendt Apr 2, 2021

Choose a reason for hiding this comment

dantegd Apr 2, 2021

Choose a reason for hiding this comment

codecov-io commented Apr 2, 2021

Codecov Report

dantegd commented Apr 2, 2021