Cloud Pak for Data/metrics/explainability/contrastive_anamoly_sample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8db41f68",
   "metadata": {},
   "source": [
    "# Contrastive Anamoly Explanations generation via Python SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edb7ecee",
   "metadata": {},
   "source": [
    "Likelihood compensation (LC) is a framework for explaining observed deviations of a black-box regression model $y=f(\\mathbf{x})$, where $y$ is a real-valued target variable and $\\mathbf{x}$ is a multivariate numerical vector as the input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc9a0122",
   "metadata": {},
   "source": [
    "### Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9db309",
   "metadata": {},
   "source": [
    "In the field of XAI (explainable AI), LC falls into the category of local explanation. However, it addresses a unique problem setting that differs from most of XAI approaches. In words, the task is as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c91e3e",
   "metadata": {},
   "source": [
    "With a $\\mathbf{x}=\\mathbf{x}^t$ observed, a black-box model predicted the target to be $f(\\mathbf{x}^t)$. However, the actual observation $y = y^t$ significantly deviated from the prediction. Which input variables are most responsible for the deviation and how?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f1bcf04",
   "metadata": {},
   "source": [
    "### Difference from existing XAI methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6def2d8b",
   "metadata": {},
   "source": [
    "We need to keep in mind that LC needs both $\\mathbf{x}=\\mathbf{x}^t$ and $y=y^t$. This setting differs from most of the XAI approaches for regression, such as LIME and Shapley values, which typically require only $\\mathbf{x}=\\mathbf{x}^t$ to explain local properties of $f(\\mathbf{x})$. In contrast, LC is to explain the deviation:\n",
    "\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a795db34",
   "metadata": {},
   "source": [
    "\n",
    "Most of the existing XAI approaches to regression:\n",
    "    \n",
    "Given $\\mathbf{x}=\\mathbf{x}^t$, explain the black-box function $f(\\mathbf{x})$ in the vicinity of $\\mathbf{x}=\\mathbf{x}^t$.\n",
    "\n",
    "LC:\n",
    "    \n",
    "Given $\\mathbf{x}=\\mathbf{x}^t$ AND $y=y^t$, locally explain where a large deviation $y^t - f(\\mathbf{x}^t)$ came from.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b892be25",
   "metadata": {},
   "source": [
    "**Note** : **This notebook works only with Default python3.9 and Default Python 3.8 environments in case of WatsonStudio and CPD**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b287222e",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b8f9b47",
   "metadata": {},
   "source": [
    "## Package installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ef0f689",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pyspark==3.3.1 | tail -n 1\n",
    "!pip install matplotlib\n",
    "!pip install seaborn\n",
    "!pip install --upgrade --user ibm-metrics-plugin --no-cache | tail -n 1\n",
    "!pip install --upgrade --user ibm-watson-openscale --no-cache | tail -n 1\n",
    "!pip install numpy==1.23.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87bb370d",
   "metadata": {},
   "source": [
    "### Initialize spark session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94483b39",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark import SparkContext, SparkConf\n",
    "import pandas as pd\n",
    "\n",
    "sparkconf = SparkConf().setMaster(\"local[*]\")\n",
    "spark = SparkSession.builder.appName(\"TestMetricFramework\").config(conf=sparkconf).getOrCreate()\n",
    "spark.sparkContext._conf.getAll()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb0f3db",
   "metadata": {},
   "source": [
    "### Import dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f75fe88d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X, y = fetch_california_housing(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "            X, y, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f72d3fde",
   "metadata": {},
   "source": [
    "### Building a black-box prediction function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fca6073",
   "metadata": {},
   "source": [
    "For end-to-end demonstration purposes, we train a Linear Regression model and think of it as a black-box prediction function once trained. We will not use any information on the internal parameters of the model and the training data later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aee94d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "model = LinearRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e270f7b7",
   "metadata": {},
   "source": [
    "## Configure Openscale"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4074ead",
   "metadata": {},
   "source": [
    "The notebook will now import the necessary libraries and set up a Python OpenScale client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12d5c0f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "WOS_CREDENTIALS = {\n",
    "    \"url\": \"<cluster url>\",\n",
    "    \"username\": \"\",\n",
    "    \"password\": \"\",\n",
    "    \"instance_id\": \"<service instance id>\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a597508f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale import APIClient as OpenScaleAPIClient\n",
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator, CloudPakForDataAuthenticator\n",
    "from ibm_watson_openscale.supporting_classes.enums import TargetTypes,ServiceTypes\n",
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import *\n",
    "\n",
    "authenticator = CloudPakForDataAuthenticator(\n",
    "    url=WOS_CREDENTIALS[\"url\"],\n",
    "    username=WOS_CREDENTIALS[\"username\"],\n",
    "    password=WOS_CREDENTIALS[\"password\"],\n",
    "    disable_ssl_verification=True\n",
    ")\n",
    "\n",
    "client = OpenScaleAPIClient(\n",
    "    service_url=WOS_CREDENTIALS[\"url\"],\n",
    "    service_instance_id=WOS_CREDENTIALS[\"instance_id\"],\n",
    "    authenticator=authenticator\n",
    ")\n",
    "\n",
    "print(client.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "558e6cd5",
   "metadata": {},
   "source": [
    "### Detecting anomalies/outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "916c9411",
   "metadata": {},
   "source": [
    "lc comes with anomaly_score(), a function for computing anomaly score for observed y values. The anomaly detection model requires a model parameter named sigma_yf, which is the standard deviation of the deviation $y - f(\\mathbf{x})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c363db08",
   "metadata": {},
   "outputs": [],
   "source": [
    "sigma_yf = (y_test - model.predict(X_test)).std()\n",
    "print('predictive std dev. sigma_yf={}'.format(sigma_yf))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7962be",
   "metadata": {},
   "source": [
    "We use anomaly_score() to compute the anomaly score. Let us focus on the sample with the highest anomaly score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aec37fff",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import numpy as np\n",
    "from ibm_metrics_plugin.metrics.explainability.explainers.contrastive_anamoly import lc_util as util\n",
    "a = util.anomaly_score(X_test,y_test,model.predict,sigma_yf=sigma_yf)\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb; sb.set()\n",
    "fig,ax=plt.subplots(figsize=(6,3))\n",
    "x = np.arange(0,len(a))\n",
    "ax.scatter(x,a,color='black',alpha=0.5)\n",
    "ax.set_xlabel('sample index',fontsize=18)\n",
    "ax.set_ylabel('anomaly score',fontsize=18)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77fe7977",
   "metadata": {},
   "outputs": [],
   "source": [
    "idx_argmax = np.argmax(a)\n",
    "print('max value a[{}]={}'.format(idx_argmax,a[idx_argmax]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa773d3f",
   "metadata": {},
   "source": [
    "## Computing LC score for detected outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730c1248",
   "metadata": {},
   "source": [
    "We pick the sample with the highest anomaly score ($n=21$) and compute the LC score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42cd64d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "idx_argmax = np.argmax(a)\n",
    "print('maximum anomaly score: a[{}]={}'.format(idx_argmax,a[idx_argmax]))\n",
    "\n",
    "# Pick the sample with the highest anomaly score\n",
    "x_outlier = X_test[idx_argmax,:]\n",
    "y_outlier = y_test[idx_argmax]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1008d7c7",
   "metadata": {},
   "source": [
    "### Define the input data row for which contrastive anamoly explanation needs to be generated"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db51b919",
   "metadata": {},
   "source": [
    "As identified above, the test sample at index idx_argmax is an anamoly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5414cac0",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = np.append(x_outlier, y_outlier)\n",
    "cols = list(fetch_california_housing().feature_names)\n",
    "cols.append(\"PRICE\")\n",
    "test_df_in = pd.DataFrame([test_data], columns=cols)\n",
    "testDF_spark = spark.createDataFrame(test_df_in)\n",
    "testDF_spark.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7de24f0",
   "metadata": {},
   "source": [
    "### Set configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36992194",
   "metadata": {},
   "source": [
    "It is important to generate lc_stats when the input data contains NaN's so that the accuracy of the explanation is not impacted.\n",
    "\n",
    "The below method uses some learnings made on the training data to impute NaNs in the test data. Hence, it is mandatory to provide training data\n",
    "to compute lc_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50c09344",
   "metadata": {},
   "source": [
    "Uncomment the below cell only if the test data contains NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e15f0bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_stats = None\n",
    "'''from ibm_metrics_plugin.metrics.explainability.entity.training_stats import TrainingStats\n",
    "feature_cols = ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'\n",
    "training_data_info = {\n",
    "            \"label_column\": \"PRICE\",\n",
    "            \"feature_columns\":  feature_cols,\n",
    "            \"problem_type\": \"regression\"\n",
    "        }\n",
    "df_boston = pd.DataFrame(X,columns=feature_cols)\n",
    "df_boston['PRICE'] = pd.Series(y)\n",
    "training_stats = TrainingStats(df_boston, training_data_info)\n",
    "lc_stats = training_stats.compute_lc_stats()'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "550e01dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set lc_inputs\n",
    "X_scales=X_test.std(axis=0)\n",
    "lc_inputs = {\n",
    "                    \"sigma_yf\": sigma_yf,\n",
    "                    \"x_scales\": X_scales,\n",
    "                    \"lc_stats\": lc_stats\n",
    "    \n",
    "                }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e2f1291",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set metrics plugin level configuration\n",
    "configuration = {}\n",
    "configuration['configuration'] = {\n",
    "            \"problem_type\": \"regression\",\n",
    "            \"label_column\": \"PRICE\",\n",
    "            \"prediction\": \"prediction\",\n",
    "            \"input_data_type\": \"structured\",\n",
    "            \"feature_columns\": ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'],\n",
    "            \"explainability\": {\n",
    "                \"metrics_configuration\":{\n",
    "                \"contrastive_anamoly\": lc_inputs\n",
    "                }\n",
    "            }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38942d2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = client.ai_metrics.compute_metrics(spark=spark, configuration=configuration, data_frame=testDF_spark, scoring_fn=model.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf6040ec",
   "metadata": {},
   "source": [
    "### Generate Contrastive Anamoly Explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bffe548b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read output and print\n",
    "import json\n",
    "metrics = results.get(\"metrics_result\")\n",
    "lc_metrics = {}\n",
    "\n",
    "if(metrics.get(\"explainability\")):\n",
    "        explain_metrics = metrics.get(\"explainability\")\n",
    "        lc_metrics = explain_metrics.get(\"contrastive_anamoly\")\n",
    "        \n",
    "if not lc_metrics:\n",
    "    print(\"unable to compute lc metrics\")\n",
    "else:\n",
    "    print(lc_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5609c51",
   "metadata": {},
   "source": [
    "### Plotting LC scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eb3c030",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(figsize=(6,4))  \n",
    "delta = list(lc_metrics[0].values())\n",
    "cols = lc_metrics[0].keys()\n",
    "ax.bar(cols,delta)\n",
    "ax.tick_params(axis='x', rotation=90) \n",
    "ax.set_title('LC score $\\delta$')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a6eb31",
   "metadata": {},
   "source": [
    "### Understanding computed LC score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc855c06",
   "metadata": {},
   "source": [
    "The computed LC score suggests:\n",
    "\n",
    "1) If MedInc had been larger by 1.0, the fit would have been better (and the anomaly score would have been lower). That is, MedInc is too small.\n",
    "\n",
    "2) If AveRooms had been smaller by 0.25, the fit would have been better (and the anomaly score would have been lower). That is, AveRooms is a bit too large.\n",
    "\n",
    "3) If AveBedrms had been larger by 1.9, the fit would have been better (and the anomaly score would have been lower). That is, AveBedrms is large.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee93c1d9",
   "metadata": {},
   "source": [
    "### Shutdown spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0298fd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv39test",
   "language": "python",
   "name": "venv39test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}