BUG: Inconsistency and error in returned explainer objects with TreeExplainer #3187

NegatedObjectIdentity · 2023-08-11T10:00:40Z

Issue Description

I use shap to explain LightGBM models for different tasks. Depending on the explaination task (classification bin/multi or regression; with or without feature interactions), the TreeExplainer class returns ShapExplaination objects in various inconsistent shapes or in the case of task multiclass+interactions it crashes.

Here my testings:

Kind of task | Explainer object shape | My assessment

Regression+no_interactions | samples x features | OK
Regression+feature_interactions | samples x features x features | OK
Classification binary+no_interactions | samples x features x classes | inconsistent with 4.
Classification binary+feature_interactions | samples x features x features | inconsistent with 3. and 5.
Classification multiclass+no_interactions | samples x features x classes | inconsistent with 4.
Classification multiclass+feature_interactions | crashes with the traceback shown below | error

Minimal Reproducible Example

from sklearn.datasets import load_diabetes
from sklearn.datasets import load_digits
from sklearn.datasets import load_breast_cancer
from lightgbm import LGBMRegressor
from lightgbm import LGBMClassifier
from shap.explainers import Tree as TreeExplainer

# data | regression | binary | multi class
data_reg = load_diabetes(as_frame=True)
data_bin = load_breast_cancer(as_frame=True)
data_mult = load_digits(as_frame=True)

# train models | regression | binary | multi class
model_reg = LGBMRegressor(**{'verbosity': -1,}).fit(data_reg.data, data_reg.target)
model_bin = LGBMClassifier(**{'verbosity': -1,}).fit(data_bin.data, data_bin.target)
model_mult = LGBMClassifier(**{'verbosity': -1,}).fit(data_mult.data, data_mult.target)

# Explainer
explainer_reg = TreeExplainer(model_reg)
explainer_bin = TreeExplainer(model_bin)
explainer_mult = TreeExplainer(model_mult)

# Explainations
explainations_reg = explainer_reg(data_reg.data, interactions=False)
explainations_reg_inter = explainer_reg(data_reg.data, interactions=True)
explainations_bin = explainer_bin(data_bin.data, interactions=False)
explainations_bin_inter = explainer_bin(data_bin.data, interactions=True)
explainations_mult = explainer_mult(data_mult.data, interactions=False)
explainations_mult_inter = explainer_mult(data_mult.data, interactions=True)

Traceback

File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:238 in __call__
    v = self.shap_interaction_values(X)
File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:525 in shap_interaction_values
    X, y, X_missing, flat_output, tree_limit, _ = self._validate_inputs(X, y, tree_limit, False)
File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:265 in _validate_inputs
    tree_limit = self.model.values.shape[0]

AttributeError: 'TreeEnsemble' object has no attribute 'values'

Expected Behavior

No error in task case 6. (Classification multiclass+feature_interactions)
Consistent shape of the returned explainer objects

Bug report checklist

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest release of shap.
I have confirmed this bug exists on the master branch of shap.
I'd be interested in making a PR to fix this bug

Installed Versions

conda-forge shap 0.42.1
conda-forge lightgbm 4.0.0
conda-forge python 3.11.4
Windows 11

connortann · 2023-08-12T13:58:21Z

Thank you for the bug report, it would be great to investigate this and get it fixed.

As a first step, it would be helpful if you could update your example above to a complete minimal reproducible example, as described in this guide.

NegatedObjectIdentity · 2023-08-13T18:29:04Z

Thanks for your answer! I added a new MRE. I hope this helps to trace down the problem.

thatlittleboy · 2023-08-19T17:45:51Z

Haven't yet had time to look too deeply into this,

but on a surface level I agree that case 4 should return samples x features x features x num_classes.
case 6: will look into this.

Thanks for the bug report.

znacer · 2023-08-26T13:45:53Z

I tried to look a bit for case 6.
I didn't get the same traceback. I get the following :

Traceback (most recent call last):
  File "/home/zak/PycharmProjects/shap/data/sandbox.py", line 36, in <module>
    explainations_mult_inter = explainer_mult(data_mult.data, interactions=True)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zak/PycharmProjects/shap/shap/explainers/_tree.py", line 242, in __call__
    ev_tiled = np.tile(self.expected_value, (v.shape[0], 1))
                                             ^^^^^^^
AttributeError: 'list' object has no attribute 'shape'

I tried to add the following after line 238 of _tree.py:

if isinstance(v, list):
                v = np.stack(v, axis=-1)

This case then returns sample x features x features x classes.

Concerning case 3 and 4, I tried with an XGBoost model. Case 3 is shaped samples x features.

NegatedObjectIdentity · 2023-09-04T12:17:47Z

I tried my example again and I still get the same error as I reported above.

Output of my IPython console:

Python 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:23:48) [MSC v.1936 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.15.0 -- An enhanced Interactive Python.

runfile('/ML-Dev/Bug_report/example.py', wdir='ML-Dev/Bug_report')
Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
Traceback (most recent call last):

File ~\anaconda3\envs\mlenv\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

File \ml-dev\bug_report\example.py:36
    explainations_mult_inter = explainer_mult(data_mult.data, interactions=True)

File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:238 in __call__
    v = self.shap_interaction_values(X)

File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:525 in shap_interaction_values
    X, y, X_missing, flat_output, tree_limit, _ = self._validate_inputs(X, y, tree_limit, False)

File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:265 in _validate_inputs
    tree_limit = self.model.values.shape[0]

AttributeError: 'TreeEnsemble' object has no attribute 'values'

conda-forge shap 0.42.1
conda-forge lightgbm 4.0.0
conda-forge python 3.11.5
Windows 11

NegatedObjectIdentity · 2023-09-04T14:04:38Z

Update with the current master branch (04.09.2023 16:00 CET, pip install git+https://github.com/shap/shap):

Same example as posted in the first post:

Regression+no_interactions | samples x features | OK
Regression+feature_interactions | samples x features x features | OK
Classification binary+no_interactions | samples x features x classes | inconsistent with 4.
Classification binary+feature_interactions | samples x features x features | inconsistent with 3. and 5.
Classification multiclass+no_interactions | samples x features x classes | inconsistent with 4.
Classification multiclass+feature_interactions | crashes with the traceback shown below | error

No change on example 1 to 5.
The error on example 6 has changed to the one shown below.

Python 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:23:48) [MSC v.1936 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.15.0 -- An enhanced Interactive Python.

runfile(/ML-Dev/Bug_report/example.py', wdir='/ML-Dev/Bug_report')
Traceback (most recent call last):

  File ~\anaconda3\envs\mlenv\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File \ml-dev\bug_report\example.py:36
    explainations_mult_inter = explainer_mult(data_mult.data, interactions=True)

  File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:238 in __call__
    v = self.shap_interaction_values(X)

  File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:529 in shap_interaction_values
    X, y, X_missing, flat_output, tree_limit, _ = self._validate_inputs(X, y, tree_limit, False)

  File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:294 in _validate_inputs
    assert self.model.fully_defined_weighting, "The background dataset you provided does " \

AssertionError: The background dataset you provided does not cover all the leaves in the model, so TreeExplainer cannot run with the feature_perturbation="tree_path_dependent" option! Try providing a larger background dataset, no background dataset, or using feature_perturbation="interventional".

This error is weird since option "tree_path_dependent" should not require any background samples because it uses the tree structure, right?

conda-forge shap 0.42.1(Current master in github with pip install git+https://github.com/shap/shap)
conda-forge lightgbm 4.0.0
conda-forge python 3.11.5
Windows 11

znacer · 2023-09-04T18:20:18Z

If I understand well the error and the paper introducing TreeSHAP, There is a need to cover all leaves to compute these values.
In the code, in the case of a lightgbm model, every tree is explored using a BFS algorithm. And during this exploration each node weighted based on the number of leaves it contains.
What is weird is that some trees appears to be empty.
To get thoose tree here is the code I used:

for i in range(len(explainer_mult.model.trees)):
    if np.min(explainer_mult.model.trees[i].node_sample_weight) <= 0:
        print(f"tree {i}: explainer_mult.model.trees[i]")

A way to avoid that is to add restrictions to your model (e.g. limit max_depth).
This ends up in the error I encounter.

Maybe the way nodes are weighted in the case of LGBM has to be updated ?

NegatedObjectIdentity · 2023-09-05T08:10:55Z

Thanks @znacer for your awesome work! I also investigated the background dataset issue of task 6 a little. I set the min_child_weight parameter of the lightgbm to a higher value. This should solve the issue if @znacer your explanation is right, since preventing the weights to approach zero, right? However, for me nothing changed. So I did a dirty hack and set fully_defined_weighting = True at line 1180 in _tree.py:

# ensure that the passed background dataset lands in every leaf
if np.min(self.trees[i].node_sample_weight) <= 0:
  self.fully_defined_weighting = True

Which let me pass the assert in line 297 in _tree.py:

if self.feature_perturbation == "tree_path_dependent":
            assert self.model.fully_defined_weighting, "The background dataset you provided does " \
                                                        "not cover all the leaves in the model, " \
                                                        "so TreeExplainer cannot run with the " \
                                                        "feature_perturbation=\"tree_path_dependent\" option! " \
                                                        "Try providing a larger background " \
                                                        "dataset, no background dataset, or using " \
                                                        "feature_perturbation=\"interventional\"."

Still, as far as I understand, this assert is a bug and the assert should not be carried out at all if no data are passed as background dataset, but only if one passes data as background dataset. Hence, the if should be different and the assert should only be carried out if background data is not None and feature_perturbation is tree_path_dependent.

Passing the assert leads to the following error, which is the same as mention by @znacer:

Python 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:23:48) [MSC v.1936 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.15.0 -- An enhanced Interactive Python.

runfile('/ML-Dev/Bug_report/example.py', wdir='/ML-Dev/Bug_report')
Traceback (most recent call last):

  File ~\anaconda3\envs\mlenv\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File \ml-dev\bug_report\example.py:36
    explainations_mult_inter = explainer_mult(data_mult.data, interactions=True)

  File ~\anaconda3\envs\mlenv\Lib\site-packages\shap\explainers\_tree.py:247 in __call__
    ev_tiled = np.tile(self.expected_value, (v.shape[0], 1))

AttributeError: 'list' object has no attribute 'shape'

Adding the code proposed by @znacer below line 238:

if isinstance(v, list):
                v = np.stack(v, axis=-1)

leads to the expected output of samples x features x features x classes as already shown by @znacer without an error. Hence this seems to fix the issue. However, I did not look into the correctness of the results.

Letting only the inconsistency of task 4 open, which should be samples x features x features x classes as well.

CloseChoice · 2024-01-13T19:48:51Z

In my opinion there are two problems with this approach:

It is quite difficult to implement correctly for all model classes since we need to distinguish between binary classification, regression and multi-class classification. I guess this can be handled, but it requires some additional internal attributes that we need to set for each model type
It might be confusing to the user if one always needs to slice the explanations correctly to sum the shap values correctly:

import shap
import xgboost 
import numpy as np

X, y = shap.datasets.adult(n_points=50)

xgb = xgboost.XGBClassifier(max_depth=1).fit(X, y)
ex_xgb = shap.TreeExplainer(xgb)
e_xgb_bin = ex_xgb(X, interactions=False)
class1_pred = xgb.predict(X, output_margin=True)
# stacking outputs might be a bit tedious for the user
xgb_pred = np.vstack([-class1_pred, class1_pred]).T

assert (np.abs(e_xgb_bin.values.sum(1) + e_xgb_bin.base_values - xgb_pred).max(axis=(0, 1))
            < 1e-4
        )
# or just knowing that we need the last index for the class might also lead to confusion
assert (np.abs(e_xgb_bin.values.sum(1)[:, -1] + e_xgb_bin.base_values[:, -1] - class1_pred).max()
            < 1e-4
        )

Alternatively I would propose that we align the output of shap and interaction values on the model output. The advantage is, that we have this readily available in internal attributes and that it is straight forward for the user to understand what the shap values are referring to. So that the following checks are always fulfilled:

# for outputs of shape (#observations,)
pred = model.predict(X)  # I know that we might need some additional parameters for the predict, but I omit this for simplicity
assert np.allclose(e_xgb_bin.values.sum(1) + e_xgb_bin.base_values, pred)
 
# for outputs of of shape (#observations, #num_classes_or_targets)
assert np.allclose(e_xgb_bin.values.sum(1, 2) + e_xgb_bin.base_values, pred)

NegatedObjectIdentity · 2024-01-13T21:11:24Z

I am happy to have the SHAP output aligned with model output. My report was on LightGBM only, there I found the inconsistency between including interactions or not when using the TreeExplainer (and one case where it crashes). In my opinion case 4 should be samples x features x features x classes to be consistent with case 3, but the classes is missing and for case 6 where it is crashing it should be samples x features x features x classes as well.

CloseChoice · 2024-01-14T08:03:00Z

Just to make this clear, my proposal would change this to:
samples x features x features x num_outputs, so that for a binary classification this would lead to samples x features x features x 1 (for lightgbm, xgboost, catboost and random forest) but for a multi-class classification would result in samples x features x features x classes

NegatedObjectIdentity · 2024-01-14T08:51:33Z

Since in the binary case results of class 0 are the complement of class 1 I think it is OK. However, I would prefer to have it samples x features x features x classes everywhere since that would be consistent (and I believe this is the structure of LightGBM?). Nevertheless I am fine with samples x features x features x 1 for the binary case as long as case 6 will work finally and also case 4. Because as it is now, this is not usable. And to be clear, I am writing about LightGBM only.

CloseChoice · 2024-01-14T13:15:55Z

Case 6 is probably connected to #3457

CloseChoice · 2024-01-14T21:45:28Z

@NegatedObjectIdentity I just tested it and case 6 should work on latest master (no idea what changed though). You can even have a look at this PR where I explicitly test case 6: https://github.com/shap/shap/pull/3459/files#diff-575fa6d845d524882a8ba5609d8a2c7f877844870c10aa551a67d8fa24461b05R1680

NegatedObjectIdentity added the bug Indicates an unexpected problem or unintended behaviour label Aug 11, 2023

thatlittleboy added this to the 0.44.0 milestone Aug 26, 2023

thatlittleboy added the todo Indicates issues that should be excluded from being marked as stale label Aug 26, 2023

This was referenced Sep 5, 2023

ENH: Support for interaction values for lightgbm #3237

Open

[python-package] LGBMClassifier produces empty trees microsoft/LightGBM#6080

Closed

This was referenced Oct 8, 2023

BREAKING: make shap explanations consistent for xgboost and lightgbm #3318

Merged

BUG: AttributeError for Interaction Values with RandomForestClassifier #3323

Closed

connortann modified the milestones: 0.44.0, 0.45.0 Dec 6, 2023

connortann closed this as completed in #3318 Mar 7, 2024

NegatedObjectIdentity mentioned this issue Mar 15, 2024

BUG: LightGBM with multiclass interaction TreeShap produces explainer error #3574

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Inconsistency and error in returned explainer objects with TreeExplainer #3187

BUG: Inconsistency and error in returned explainer objects with TreeExplainer #3187

NegatedObjectIdentity commented Aug 11, 2023 •

edited

connortann commented Aug 12, 2023

NegatedObjectIdentity commented Aug 13, 2023

thatlittleboy commented Aug 19, 2023

znacer commented Aug 26, 2023 •

edited

NegatedObjectIdentity commented Sep 4, 2023 •

edited

NegatedObjectIdentity commented Sep 4, 2023 •

edited

znacer commented Sep 4, 2023

NegatedObjectIdentity commented Sep 5, 2023 •

edited

CloseChoice commented Jan 13, 2024

NegatedObjectIdentity commented Jan 13, 2024

CloseChoice commented Jan 14, 2024

NegatedObjectIdentity commented Jan 14, 2024 •

edited

CloseChoice commented Jan 14, 2024

CloseChoice commented Jan 14, 2024 •

edited

BUG: Inconsistency and error in returned explainer objects with TreeExplainer #3187

BUG: Inconsistency and error in returned explainer objects with TreeExplainer #3187

Comments

NegatedObjectIdentity commented Aug 11, 2023 • edited

Issue Description

Kind of task | Explainer object shape | My assessment

Minimal Reproducible Example

Traceback

Expected Behavior

Bug report checklist

Installed Versions

connortann commented Aug 12, 2023

NegatedObjectIdentity commented Aug 13, 2023

thatlittleboy commented Aug 19, 2023

znacer commented Aug 26, 2023 • edited

NegatedObjectIdentity commented Sep 4, 2023 • edited

NegatedObjectIdentity commented Sep 4, 2023 • edited

znacer commented Sep 4, 2023

NegatedObjectIdentity commented Sep 5, 2023 • edited

CloseChoice commented Jan 13, 2024

NegatedObjectIdentity commented Jan 13, 2024

CloseChoice commented Jan 14, 2024

NegatedObjectIdentity commented Jan 14, 2024 • edited

CloseChoice commented Jan 14, 2024

CloseChoice commented Jan 14, 2024 • edited

NegatedObjectIdentity commented Aug 11, 2023 •

edited

znacer commented Aug 26, 2023 •

edited

NegatedObjectIdentity commented Sep 4, 2023 •

edited

NegatedObjectIdentity commented Sep 4, 2023 •

edited

NegatedObjectIdentity commented Sep 5, 2023 •

edited

NegatedObjectIdentity commented Jan 14, 2024 •

edited

CloseChoice commented Jan 14, 2024 •

edited