Manifold Restricted Interventional Shapley Values

By Muhammad Faaiz Taufiq, Patrick Blöbaum, Lenon Minorics.

This project is the official implementation of "Manifold Restricted Interventional Shapley Values".

Introduction

Shapley values are model-agnostic methods for explaining model predictions. Many commonly used methods of computing Shapley values, known as off-manifold methods, rely on model evaluations on out-of-distribution input samples. Consequently, explanations obtained are sensitive to model behaviour outside the data distribution, which may be irrelevant for all practical purposes. While on-manifold methods have been proposed which do not suffer from this problem, we show that such methods are overly dependent on the input data distribution, and therefore result in unintuitive and misleading explanations. To circumvent these problems, we propose ManifoldShap, which respects the model’s domain of validity by restricting model evaluations to the data manifold. We show, theoretically and empirically, that ManifoldShap is robust to off-manifold perturbations of the model and leads to more accurate and intuitive explanations than existing state-of-the-art Shapley methods.

Installation

To install this package, run the following command:
pip install -r requirements.txt

Running Synthetic Data Experiments

Here, we outline the commands to run the synthetic data experiments. For each experiment, the parameters can be changed using the arguments provided in the arguments.py file.

Off-manifold perturbation (regression) example

To run the experiments with the perturbed model $g_\delta$ defined as:
$g_\delta(x_1, x_2) \coloneqq x_1 + \delta x_2 1((x_1,x_2) \notin \mathcal{P}_\alpha)$
run the command:
python perturbation_experiment_with_different_dag.py

Off-manifold perturbation (classification) example

To instead run the experiments with perturbed classifier model $g_\delta$ defined as:
$g_\delta(x_1, x_2) \coloneqq 1(x_1 > 1/2, (x_1, x_2)\in \mathcal{P}\alpha) + 1(x_1 (1-\delta) > 1/2, (x_1,x_2) \notin \mathcal{P}\alpha)$
run the command:
python perturbation_off_manifold_main.py

Increasing correlations example

To run the experiment with increasing feature correlations, run the command:
python shap_values_for_increasing_correlation.py

Varying manifold size experiments

To run the experiment with varying manifold sizes, run the command:
python shap_values_vs_epsilon_values.py

ManifoldShap vs RJBShap experiments

To run the experiment comparing ManifoldShap with RJBShap, run the command:
python density_exp.py

Running Real-world Dataset Experiments

To run the real-world dataset experiments, please refer to the detailed steps provided in fooling-shap-experiments.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.