training statistics/4.6/training_statistics_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook for generating configuration for online subscriptions in IBM Watson OpenScale \n",
    "\n",
    "This notebook shows how to generate the following artefacts:\n",
    "\n",
    "1. Configuration JSON needed to configure an IBM Watson OpenScale online subscription. This JSON also contains information related to fairness configuration.\n",
    "2. Drift Configuration Archive\n",
    "3. Explainability Configuration Archive\n",
    "\n",
    "In order to use this notebook you need to do the following:\n",
    "\n",
    "1. Read the training data into a pandas dataframe called \"data_df\".  There is sample code below to show how this can be done if the training data is in IBM Cloud Object Storage. \n",
    "2. Edit the below cells and provide the training data and fairness configuration information. \n",
    "3. Run the notebook. It will generate a JSON, drift and explainability archives. A download link for the artifacts will be present at the  end of the notebook.\n",
    "4. Download the JSON, archives by clicking on the link and upload it in the IBM AI OpenScale GUI.\n",
    "\n",
    "If you have multiple models (deployments), you will have to repeat the above steps for each model (deployment).\n",
    "\n",
    "**Note:** Please restart the kernel after executing below cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas\n",
    "!pip install ibm-cos-sdk\n",
    "!pip install pyspark\n",
    "!pip install --upgrade ibm-watson-openscale==3.0.28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** For IBM Watson OpenScale Cloud and Cloud Pak for Data version 4.6.x, use the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When this notebook is to be run on a zLinux cluster,\n",
    "# install scikit-learn==1.0.2 using conda before installing ibm-metrics-plugin\n",
    "# !conda install scikit-learn=1.0.2\n",
    "!pip install --upgrade \"ibm-metrics-plugin>=4.6.5.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Openscale Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the environment of execution use one of the below method for OpenScale configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide your IBM Watson OpenScale Cloud credentials in the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = \"***\"\n",
    "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "\n",
    "from ibm_watson_openscale import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "\n",
    "\n",
    "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
    "#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment this line if using IAM token as authenticator\n",
    "client = APIClient(authenticator=authenticator)\n",
    "client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide your IBM Watson OpenScale CPD credentials in the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "WOS_CREDENTIALS = {\n",
    "    \"url\": \"<cluster url>\",\n",
    "    \"username\": \"\",\n",
    "    \"password\": \"\",\n",
    "    \"instance_id\": \"<service instance id>\"\n",
    "}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "from ibm_watson_openscale import APIClient as OpenScaleAPIClient\n",
    "from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator\n",
    "\n",
    "authenticator = CloudPakForDataAuthenticator(\n",
    "    url=WOS_CREDENTIALS[\"url\"],\n",
    "    username=WOS_CREDENTIALS[\"username\"],\n",
    "    password=WOS_CREDENTIALS[\"password\"],\n",
    "    disable_ssl_verification=True\n",
    ")\n",
    "\n",
    "client = OpenScaleAPIClient(\n",
    "    service_url=WOS_CREDENTIALS[\"url\"],\n",
    "    service_instance_id=WOS_CREDENTIALS[\"instance_id\"],\n",
    "    authenticator=authenticator\n",
    ")\n",
    "\n",
    "client.version\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read training data into a pandas data frame\n",
    "\n",
    "The first thing that you need to do is to read the training data into a pandas dataframe called \"data_df\".  Given below is sample code for doing this if the training data is in IBM Cloud Object Storage.  Please edit the below cell and make changes so that you can read your training data from the location where it is stored.  Please ensure that the training data is present in a data frame called \"data_df\".\n",
    "\n",
    "*Note: Pandas' read\\_csv method converts the columns to its data types. If you want the column type to not be interpreted, specify the dtype param to read_csv method in this cell. More on this method [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)*\n",
    "\n",
    "*Note: By default NA values will be dropped while computing training data distribution and training the drift archive. Please ensure to handle the NA values during Pandas' read\\_csv method*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ----------------------------------------------------------------------------------------------------\n",
    "# IBM Confidential\n",
    "# OCO Source Materials\n",
    "# 5900-A3Q, 5737-H76\n",
    "# Copyright IBM Corp. 2018, 2023\n",
    "# The source code for this Notebook is not published or other-wise divested of its trade \n",
    "# secrets, irrespective of what has been deposited with the U.S.Copyright Office.\n",
    "# ----------------------------------------------------------------------------------------------------\n",
    "\n",
    "VERSION = \"5.4.8\"\n",
    "\n",
    "# Version history\n",
    "\n",
    "# 5.4.8 : Official notebook for IBM CP4D 4.6.5\n",
    "#         Upgrade ibm-metrics-plugin to >= 4.6.5.0\n",
    "# 5.4.7 : Official notebook for IBM CP4D 4.6.4\n",
    "#         Upgrade ibm-metrics-plugin to >= 4.6.4.0\n",
    "# 5.4.6 : Take optional class labels input for global explainability\n",
    "# 5.4.5 : Remove numpy and scipy versions to be installed.\n",
    "# 5.4.4 : Add support for lime global explanation\n",
    "# 5.4.3 : Add numpy and scipy versions to be installed.\n",
    "# 5.4.2 : Remove explainability configuration while saving training_distribution\n",
    "# 5.4.1 : Add sample size for generating global explanation\n",
    "# 5.4.0 : Add support for SHAP Global explanation\n",
    "# 5.3.6 : Fix issue with explainability archive generation for regression model\n",
    "# 5.3.5 : Official notebook for IBM CPD 4.5.0. \n",
    "#         Upgrade ibm-wos-utils to 4.5.0. \n",
    "#         Added code to generate explainability perturbations archive.\n",
    "# 5.3.4 : Upgrade ibm-wos-utils to 4.1.1 (scikit-learn has been upgraded to 1.0.2)\n",
    "# 5.3.3 : Upgrade ibm-wos-utils to 4.0.34\n",
    "# 5.3.2 : Upgrade ibm-wos-utils to 4.0.31\n",
    "# 5.3.1 : Official notebook for IBM CPD 4.0.5\n",
    "\n",
    "# code to read file in COS to pandas dataframe object\n",
    "import sys\n",
    "import types\n",
    "import pandas as pd\n",
    "from ibm_botocore.client import Config\n",
    "import ibm_boto3\n",
    "\n",
    "def __iter__(self): return 0\n",
    "\n",
    "api_key = \"<API Key>\"\n",
    "resource_instance_id = \"<COS Resource Instance ID>\"\n",
    "auth_endpoint = \"https://iam.ng.bluemix.net/oidc/token\"\n",
    "service_endpoint =  \"<COS Service Endpoint>\"\n",
    "bucket =  \"<Bucket Name>\"\n",
    "file_name= \"<File Name>\"\n",
    "\n",
    "cos_client = ibm_boto3.client(service_name=\"s3\",\n",
    "    ibm_api_key_id=api_key,\n",
    "    ibm_auth_endpoint=auth_endpoint,\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=service_endpoint)\n",
    "\n",
    "body = cos_client.get_object(Bucket=bucket,Key=file_name)[\"Body\"]\n",
    "\n",
    "# add missing __iter__ method, so pandas accepts body as file-like object\n",
    "if not hasattr(body, \"__iter__\"): body.__iter__ = types.MethodType( __iter__, body )\n",
    "\n",
    "data_df = pd.read_csv(body)\n",
    "data_df.head()\n",
    "\n",
    "#Print columns from data frames\n",
    "#print(\"column names:{}\".format(list(data_df.columns.values)))\n",
    "\n",
    "# Uncomment following 2 lines if you want to read training data from local CSV file when running through local Jupyter notebook\n",
    "#data_df = pd.read_csv(\"<FULLPATH_TO_CSV_FILE>\")\n",
    "#data_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select the services for which configuration information needs to be generated\n",
    "\n",
    "This notebook generates configuration information related to fairness, explainability, and drift services. The below flags can be used by the user to control service specific configuration information.\n",
    "\n",
    "Details of the service specific flags available:\n",
    "\n",
    "- enable_fairness : Flag to allow generation of fairness specific data distribution needed for configuration\n",
    "- enable_explainability : Flag to allow generation of explainability specific information\n",
    "- enable_drift: Flag to allow generation of drift detection model needed by drift service\n",
    "\n",
    "\n",
    "service_configuration_support = { <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"enable_fairness\": True,   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"enable_explainability\": True,    \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"enable_drift\": False  \n",
    "    }  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_configuration_support = {\n",
    "    \"enable_fairness\": True,\n",
    "    \"enable_explainability\": True,\n",
    "    \"enable_drift\": True\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Data and Fairness Configuration Information\n",
    "\n",
    "Please provide information about the training data which is used to train the model.  In order to explain the configuration better, let us first consider an example of a Loan Processing Model which is trying to predict whether a person should get a loan or not. The training data for such a model will potentially contain the following columns: Credit_History, Monthly_salary, Applicant_Age, Loan_amount, Gender, Marital_status, Approval.  The \"Approval\" column contains the target field (label column or class label) and it can have the following values: \"Loan Granted\", \"Loan Denied\" or \"Loan Partially Granted\".  In this model we would like to ensure that the model is not biased against Gender=Female or Gender=Transgender.  We would also like to ensure that the model is not biased against the age group 15 to 30 years or age group 61 to 120 years. \n",
    "\n",
    "For the above model, the configuration information that we need to provide is:\n",
    "\n",
    "- class_label:  This is the name of the column in the training data dataframe (data_df) which contains the target field (also known as label column or the class label).  For the Loan Processing Model it would be \"Approval\".\n",
    "- feature_columns: This is a comma separated list of column names which contain the feature column names (in the training data dataframe data_df).  For the Loan Processing model this would be: [\"Credit_History\", \"Monthly_salary\", \"Applicant_Age\", \"Loan_amount\", \"Gender\", \"Marital_status\"]\n",
    "- categorical_columns: The list of column names (in data_df) which contain categorical values.  This should also include those columns which originally contained categorical values and have now been converted to numeric values. E.g., in the Loan Processing Model, the Marital_status column originally could have values: Single, Married, Divorced, Separated, Widowed.  These could have been converted to numeric values as follows: Single -> 0, Married -> 1, Divorced -> 2, Separated -> 3 and Widowed -> 4.  Thus the training data will have numeric values.  Please identify such columns as categorical.  Thus the list of categorical columns for the Loan Processing Model will be Credit_History, Gender and Marital_status. \n",
    "- prediction : The name of the prediction column. This column is optional and is needed only when unscored training data is provided. \n",
    "- probability : The name of the probability column. This will not be needed for regression models. This column is also optional and is only needed when unscored training data is provided.\n",
    "\n",
    "For the Loan Processing Model, this information will be provided as follows:\n",
    "\n",
    "training_data_info = { <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"class_label\": \"Approval\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"feature_columns\": [\"Credit_History\", \"Monthly_salary\", \"Applicant_Age\", \"Loan_amount\", \"Gender\", \"Marital_status\"],    \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"categorical_columns\": [\"Credit_History\",\"Gender\",\"Marital_status\"],   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"prediction\": \"prediction\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"probability\": \"probability\"\n",
    "<br>\n",
    "    }  \n",
    "    \n",
    "  **Note:** Please note that categorical columns selected should be subset of feature columns. If there are no categorical columns among the feature columns selected , please set \"categorical_columns as [] or None\"\n",
    "\n",
    "Please edit the next cell and provide the above information for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_info = {\n",
    "    \"class_label\": \"<EDIT THIS>\",\n",
    "    \"feature_columns\": [\"<EDIT THIS>\"],\n",
    "    \"categorical_columns\": [\"<EDIT THIS>\"],\n",
    "    \"probability\": \"<EDIT THIS>\",\n",
    "    \"prediction\": \"<EDIT THIS>\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the Model Type\n",
    "\n",
    "In the next cell, specify the type of your model.  If your model is a binary classification model, then set the type to \"binary\". If it is a multi-class classifier then set the type to \"multiclass\". If it is a regression model (e.g., Linear Regression), then set it to \"regression\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set model_type. Acceptable values are:[\"binary\",\"multiclass\",\"regression\"]\n",
    "model_type = \"binary\"\n",
    "#model_type = \"multiclass\"\n",
    "#model_type = \"regression\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the Fairness Configuration\n",
    "\n",
    "You need to provide the following information for the fairness configuration: \n",
    "\n",
    "- fairness_attributes:  These are the attributes on which you wish to monitor fairness. In the Loan Processing Model, we wanted to ensure that the model is not biased against people of specific age group and people belonging to a specific gender.  Hence \"Applicant_Age\" and \"Gender\" will be the fairness attributes for the Loan Processing Model.\n",
    "- With Indirect Bias support, you can also monitor protected attributes for fairness. The protected attributes are those attributes which are present in the training data but are not used to train the model. For example, sensitive attributes like gender, race, age may be present in training data but are not used for training. To check if there exists indirect bias with respect to some protected attribute due to possible correlation with some feature column, it can be specified in fairness configuration.\n",
    "- type: The data type of the fairness attribute (e.g., float or int or double)\n",
    "- minority:  The minority group for which we want to ensure that the model is not biased.  For the Loan Processing Model we wanted to ensure that the model is not biased against people in the age group 15 to 30 years & 61 to 120 years as well as people with Gender = Female or Gender = Transgender.  Hence the minority group for the fairness attribute \"Applicant_Age\" will be [15,30] and [61,120] and the minority group for fairness attribute \"Gender\" will be: \"Female\", \"Transgender\".  \n",
    "- majority: The majority group for which the model might be biased towards.  For the Loan Processing Model, the majority group for the fairness attribute \"Applicant_Age\" will be [31,60], i.e., all the ages except the minority group.  For the fairness attribute \"Gender\" the majority group will be: \"Male\".  \n",
    "- threshold:  The fairness threshold beyond which the Model is considered to be biased.  For the Loan Processing Model, let us say that the Bank is willing to tolerate the fact that Female and Transgender applicants will get up to 20% lesser approved loans than Males.  However, if the percentage is more than 20% then the Loan Processing Model will be considered biased.  E.g., if the percentage of approved loans for Female or Transgender applicants is say 25% lesser than those approved for Male applicants then the Model is to be considered as acting in a biased manner.  Thus for this scenario, the Fairness threshold will be 80 (100-20) (this is represented as a value normalized to 1, i.e., 0.8).  \n",
    "\n",
    "The fairness attributes for Loan Processing Model will be specified as:\n",
    "\n",
    "fairness_attributes = [  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"feature\": \"Applicant_Age\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"type\" : \"int\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"majority\": [ [31,60] ],   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"minority\": [ [15, 30], [61,120] ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"threshold\" : 0.8  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"feature\": \"Gender\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"type\" : \"string\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"majority\": [\"Male\"],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"minority\": [\"Female\", \"Transgender\"],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"threshold\" : 0.8  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;]  \n",
    "\n",
    "Please edit the next cell and provide the fairness configuration for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fairness_attributes = [{\n",
    "                           \"type\" : \"<DATA_TYPE>\", #data type of the column eg: float or int or double\n",
    "                           \"feature\": \"<COLUMN_NAME>\", \n",
    "                           \"majority\": [\n",
    "                               [X, Y] # range of values for column eg: [31, 45] for int or [31.4, 45.1] for float\n",
    "                           ],\n",
    "                           \"minority\": [\n",
    "                               [A, B], # range of values for column eg: [10, 15] for int or [10.5, 15.5] for float\n",
    "                               [C, D]   # range of values for column eg: [80, 100] for int or [80.0, 99.9] for float                    \n",
    "                           ],\n",
    "                           \"threshold\": <VALUE> #such that 0<VALUE<=1. eg: 0.8\n",
    "                       }]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the Favorable and Unfavorable class values\n",
    "\n",
    "The second part of fairness configuration is about the favourable and unfavourable class values.  Recall that in the case of Loan Processing Model, the target field (label column or class label) can have the following values: \"Loan Granted\", \"Loan Denied\" and \"Loan Partially Granted\".  Out of these values \"Loan Granted\" and \"Loan Partially Granted\" can be considered as being favorable and \"Loan Denied\" is unfavorable.  In other words in order to measure fairness, we need to know the target field values which can be considered as being favourable and those values which can be considered as unfavourable.  \n",
    "\n",
    "For the Loan Prediction Model, the values can be specified as follows:\n",
    "\n",
    "parameters = {  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"favourable_class\" :  [ \"Loan Granted\", \"Loan Partially Granted\" ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"unfavourable_class\": [ \"Loan Denied\" ]  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "\n",
    "In case of a regression models, the favourable and unfavourable classes will be ranges.  For example, for a model which predicts medicine dosage, the favorable outcome could be between 80 ml to 120 ml or between 5 ml to 20 ml whereas unfavorable outcome will be values between 21 ml to 79ml.  For such a model, the favorable and unfavorable values will be specified as follows:\n",
    "     \n",
    "parameters = {  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"favourable_class\" :  [ [5, 20], [80, 120] ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"unfavourable_class\": [ [21, 79] ]  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "\n",
    "Please edit the next cell to provide information about your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For classification models use the below.\n",
    "parameters = {\n",
    "        \"favourable_class\" :  [ \"<EDIT THIS>\", \"<EDIT THIS>\" ],\n",
    "        \"unfavourable_class\": [ \"<EDIT THIS>\" ]\n",
    "    }\n",
    "# For regression models use the below.  Delete the entry which is not required.\n",
    "parameters = {\n",
    "        \"favourable_class\" :  [ [<EDIT THIS>, <EDIT THIS>], [<EDIT THIS>,<EDIT THIS>] ],\n",
    "        \"unfavourable_class\": [ [<EDIT THIS>, <EDIT THIS>] ]\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the number of records which should be processed for Fairness\n",
    "\n",
    "The final piece of information that needs to be provided is the number of records (min_records) that should be used for computing the fairness. Fairness checks runs hourly.  If min_records is set to 5000, then every hour fairness checking will pick up the last 5000 records which were sent to the model for scoring and compute the fairness on those 5000 records.  Please note that fairness computation will not start till the time that 5000 records are sent to the model for scoring.\n",
    "\n",
    "If we set the value of \"min_records\" to a small number, then fairness computation will get influenced by the scoring requests sent to the model in the recent past. In other words, the model might be flagged as being biased if it is acting in a biased manner on the last few records, but overall it might not be acting in a biased manner.  On the other hand, if the \"min_records\" is set to a very large number, then we will not be able to catch model bias quickly. Hence the value of min_records should be set such that it is neither too small or too large.\n",
    "\n",
    "Please updated the next cell to specify a value for min_records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# min_records = <Minimum number of records to be considered for preforming scoring>\n",
    "min_records = <EDIT THIS>\n",
    "# max_records = <Maximum number of records to be considered while computing fairness> [OPTIONAL]\n",
    "max_records = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Optional] Specify the Explainability Configuration\n",
    "\n",
    "Provide the following explainability configuration to enable global explanation or to enable SHAP local explanation.\n",
    "For headless subscription ie., a subscription configured without a scoring endpoint, SHAP global explanation is not supported.\n",
    "\n",
    " - global_explanation: The global explanation parameters.\n",
    "     - enabled: Enable the global explanation.\n",
    "     - sample_size: The sample size of the records to be considered for computing global explanation in the payload window.\n",
    "     - explanation_method: The explanation method to be used for generating global explanation. Supported values are \"lime\" and \"shap\". Default explanation method for global explanation generation is \"lime\". SHAP parameters should be provided when shap explanation method is selected.\n",
    " - shap[Optional]: The shap explanation parameters are **mandatory** when SHAP explanation method is selected for local or global explanation\n",
    "     - enabled: Enable the shap explanation\n",
    "     - perturbations_count: The no of perturbations created when generating a local explanation\n",
    "     - background_data_set: The background data set to be used when generating the SHAP explanation of a transaction. The background data is used to determine the average predicted value for regression models and the average confidence value for classification models. When generating a local explanation, SHAP computes the shapley values which signify how much each feature contributed to moving the model output or model confidence from the computed average value.\n",
    "     - background_data_set: The list of background data sets the user would like to configure.\n",
    " - lime[Optional]: The lime explanation parameters\n",
    "    - enabled: Enable the lime explanation\n",
    "    - perturbations_count: The no of perturbations created when generating a local explanation. Default value is 5000.\n",
    " - local_explanation_method: The default local explanation method to be used when generating local explanation. Possible values are \"shap\" and \"lime\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "# Note: LIME global explanation feature is supported from Cloud Pak for Data version 4.6.4 onwards\n",
    "# When LIME global explanation is enabled, the explainability archive upload and explainability monitor enablement should be done using python sdk/api. \n",
    "# LIME global explanation configuration is not supported from IBM Watson OpenScale GUI.\n",
    "explainability_parameters = {\n",
    "    \"global_explanation\": {\n",
    "        \"enabled\": True,\n",
    "        \"sample_size\": 50, # the sample size of the records to be considered for computing payload global explanation\n",
    "        \"explanation_method\": \"shap\" # or lime\n",
    "    },\n",
    "    \"shap\": {\n",
    "        \"enabled\": True,\n",
    "        #\"perturbations_count\": 100\n",
    "        #\"background_data_set\": \"data_set_1\", # If not set the background data is auto generated from training data\n",
    "        #\"background_data_sets\": [{\n",
    "        #    \"name\": \"data_set_1\",\n",
    "        #    \"file_name\": \"data_set_1.csv\"\n",
    "        #}]\n",
    "    },\n",
    "    #\"lime\": {\n",
    "    #    \"enabled\": True,\n",
    "    #    \"perturbations_count\": 5000\n",
    "    # },\n",
    "    \"local_explanation_method\": \"shap\", # or lime\n",
    "}\n",
    "\"\"\"\n",
    "# Uncomment the above lines to enable global explanation\n",
    "explainability_parameters = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Input \n",
    "\n",
    "You need not edit anything beyond this point.  Run the notebook and go to the very last cell.  There will be a link to download the JSON file (called: \"Download training data distribution JSON file\").  Download the file and upload it using the IBM AI OpenScale GUI.\n",
    "\n",
    "*Note: drop_na parameter of TrainingStats object should be set to 'False' if NA values are taken care while reading the training data in the above cells*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.utils.training_stats import TrainingStats\n",
    "\n",
    "enable_explainability = service_configuration_support.get(\"enable_explainability\")\n",
    "enable_fairness = service_configuration_support.get(\"enable_fairness\")\n",
    "\n",
    "if enable_explainability or enable_fairness:\n",
    "    fairness_inputs = None\n",
    "    if enable_fairness:\n",
    "        fairness_inputs = {\n",
    "                \"fairness_attributes\": fairness_attributes,\n",
    "                \"min_records\" : min_records,\n",
    "                \"favourable_class\" :  parameters[\"favourable_class\"],\n",
    "                \"unfavourable_class\": parameters[\"unfavourable_class\"]\n",
    "            }\n",
    "        if max_records is not None:\n",
    "            fairness_inputs[\"max_records\"] = max_records\n",
    "    \n",
    "    input_parameters = {\n",
    "        \"label_column\": training_data_info[\"class_label\"],\n",
    "        \"feature_columns\": training_data_info[\"feature_columns\"],\n",
    "        \"categorical_columns\": training_data_info[\"categorical_columns\"],\n",
    "        \"fairness_inputs\": fairness_inputs,  \n",
    "        \"problem_type\" : model_type,\n",
    "        \"probability\": training_data_info[\"probability\"],  \n",
    "        \"prediction\": training_data_info[\"prediction\"]\n",
    "    }\n",
    "\n",
    "    training_stats = TrainingStats(data_df,input_parameters, explain=enable_explainability, fairness=enable_fairness, drop_na=True)\n",
    "    config_json = training_stats.get_training_statistics()\n",
    "    config_json[\"notebook_version\"] = VERSION\n",
    "#print(config_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Optional] Score Function\n",
    "\n",
    "This is required if you are configuring :\n",
    "- Drift (for classification models), and/or \n",
    "- Explainability (for headless subscriptions, for computing SHAP global explanation)\n",
    "\n",
    "Please update the score function which will be used for generating drift detection model which will used for drift detection. Also, if you have a headless subscription, this will be used to generate explainability archive which be used for explanations. \n",
    "\n",
    "The output of the score function should be a 2 arrays :\n",
    "1. Array of probabilities\n",
    "2. Array of model prediction\n",
    "\n",
    "\n",
    "Please note:\n",
    "- User is expected to make sure that the data type of the \"class label\" column selected and the prediction column are same. For eg : If class label is numeric, the prediction array should also be numeric\n",
    "- Each entry of a probability array should have all the probabilities of the unique class label .\n",
    "  For eg: If the model_type=multiclass and unique class labels are A, B, C, D . Each entry in the probability array should be a array of size 4 . Eg : [ [0.50,0.30,0.10,0.10] ,[0.40,0.20,0.30,0.10]...]\n",
    "- **Please update the score function below with the help of templates documented [here](https://github.com/IBM/watson-openscale-samples/blob/main/training%20statistics/Score%20function%20templates%20for%20drift%20detection.md)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Update score function\n",
    "# def score(training_data_frame){\n",
    "#     <Fill in the template using the score function templates provided>\n",
    "# }\n",
    "# Comment the below line when using score function\n",
    "score=None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Optional] Generate explainability archive\n",
    "\n",
    "This is required for generating artifacts required for configuring explainability in IBM Watson OpenScale \n",
    "\n",
    "Output of this is an explainability configuration archive which must be uploaded to IBM Watson OpenScale during explain monitor configuration.\n",
    "\n",
    "Based on the configuration parameters the explainability archive contains some or all of the below files:\n",
    "  - training_statistics.json : The explainability training statistics\n",
    "  - configuration.json : The explainability configuration parameters used while creating the explainability monitor instance\n",
    "  - lime_scored_perturbations.json : The scored lime perturbations response\n",
    "  - shap_background_data_training.csv : The SHAP background data created using the training statistics\n",
    "  - shap_training_data_global_explanation.json : The SHAP training data global explanation\n",
    "  - lime_training_data_global_explanation.json : The LIME training data global explanation\n",
    "\n",
    "**Note: When LIME global explanation is enabled, the explainability archive upload and explainability monitor enablement should be done using python sdk/api. LIME global explanation configuration is not supported from IBM Watson OpenScale GUI.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if enable_explainability:\n",
    "    # Note: Using the entire training data for generating global explanation will take longer duration.\n",
    "    # Use a random sample of training data for faster execution.\n",
    "    sample_size = 5000\n",
    "    if len(data_df) > sample_size:\n",
    "        training_data = data_df.sample(sample_size)\n",
    "    else:\n",
    "        training_data = data_df\n",
    "\n",
    "    # [Optional] The list of unique class labels in the order of model prediction. This is applicable for classification models only.\n",
    "    # If not provided the order of class labels is determined by scoring the training data.\n",
    "    # config_json[\"common_configuration\"][\"class_labels\"] = [\"<EDIT THIS>\"]\n",
    "    explainability_archive = client.config_util.create_explainability_archive(config=config_json, \n",
    "                                                                       training_data=training_data, \n",
    "                                                                       parameters=explainability_parameters, \n",
    "                                                                       scoring_fn=score)\n",
    "    display(explainability_archive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indirect Bias\n",
    "In case of Indirect bias i.e if protected attributes(the sensitive attributes like race, gender etc which are present in the training data but are not used to train the model) are being monitored for fairness:\n",
    "- Bias service identifies correlations between the protected attribute and model features. Correlated attributes are also known as proxy features.\n",
    "- Existence of correlations with model features can result in indirect bias w.r.t protected attribute even though it is not used to train the model.\n",
    "- Highly correlated attributes based on their correlation strength are considered while computing bias for a given protected attribute.\n",
    "\n",
    "The following cell identifies if user has configured protected attribute for fairness by checking the feature, non-feature columns and the fairness configuration. If protected attribute/s are configured then it identifies correlations and stores it in the fairness configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking if protected attributes are configured for fairness monitoring. If yes, then computing correlation information for each meta-field and updating it in the fairness configuration\n",
    "if enable_fairness:\n",
    "    fairness_configuration = config_json.get(\"fairness_configuration\")\n",
    "    training_columns = data_df.columns.tolist()\n",
    "    label_column = training_data_info.get(\"class_label\")\n",
    "    training_columns.remove(label_column)\n",
    "    feature_columns = training_data_info.get(\"feature_columns\")\n",
    "    non_feature_columns = list(set(training_columns) - set(feature_columns))\n",
    "    if non_feature_columns is not None and len(non_feature_columns) > 0:\n",
    "        protected_attributes = []\n",
    "        fairness_attributes_list = [attribute.get(\"feature\") for attribute in fairness_attributes]\n",
    "        for col in non_feature_columns:\n",
    "            if col in fairness_attributes_list:\n",
    "                protected_attributes.append(col)\n",
    "        if len(protected_attributes) > 0:\n",
    "            from ibm_watson_openscale.utils.indirect_bias_processor import IndirectBiasProcessor\n",
    "            fairness_configuration = IndirectBiasProcessor().get_correlated_attributes(data_df, fairness_configuration, feature_columns, protected_attributes, label_column)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "print(\"Finished generating training distribution data\")\n",
    "\n",
    "# Create a file download link\n",
    "import base64\n",
    "from IPython.display import HTML\n",
    "\n",
    "def create_download_link( title = \"Download training data distribution JSON file\", filename = \"training_distribution.json\"):  \n",
    "    if enable_explainability or enable_fairness:\n",
    "        if \"explainability_configuration\" in config_json:\n",
    "            del config_json[\"explainability_configuration\"]\n",
    "        output_json = json.dumps(config_json, indent=2)\n",
    "        b64 = base64.b64encode(output_json.encode())\n",
    "        payload = b64.decode()\n",
    "        html = '<a download=\"{filename}\" href=\"data:text/json;base64,{payload}\" target=\"_blank\">{title}</a>'\n",
    "        html = html.format(payload=payload,title=title,filename=filename)\n",
    "        return HTML(html)\n",
    "    else:\n",
    "        print(\"No download link generated as fairness/explainability services are disabled.\")\n",
    "\n",
    "create_download_link()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fairness Statistics\n",
    "\n",
    "Following code snippet is used to generate a json file containing fairness configuration.\n",
    "This is used to configure Fairness monitor in IBM Watson OpenScale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fairness_configuration = None\n",
    "\n",
    "if enable_fairness:\n",
    "    fairness_configuration = config_json.get(\"fairness_configuration\")\n",
    "\n",
    "# create download link if required\n",
    "def create_download_link( title = \"Download fairness statistics JSON file\", filename = \"fairness_statistics.json\"):\n",
    "    if fairness_configuration:\n",
    "        # Create a file download link\n",
    "        import base64\n",
    "        import json\n",
    "        from IPython.display import HTML\n",
    "        \n",
    "        print(\"Fairness configuration found, persisting to a json file and creating download link...\")\n",
    "        output_json = json.dumps(fairness_configuration, indent=2)\n",
    "        b64 = base64.b64encode(output_json.encode())\n",
    "        payload = b64.decode()\n",
    "        html = '<a download=\"{filename}\" href=\"data:text/json;base64,{payload}\" target=\"_blank\">{title}</a>'\n",
    "        html = html.format(payload=payload,title=title,filename=filename)\n",
    "        return HTML(html)\n",
    "    else:\n",
    "        print(\"Fairness configuration not found.\")\n",
    "create_download_link()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drift configuration archive generation\n",
    "\n",
    "Following code snippet is used to generate artefacts required for configuring drift identification for a model in IBM Watson OpenScale. Output is a drift archive which contains:\n",
    "\n",
    "- Drift Detection Model (used for Accuracy Drift detection for classification models)\n",
    "- Data Constraints (used for Data Consistency Drift detection for classification/regression models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Generate drift detection model\n",
    "from ibm_wos_utils.drift.drift_trainer import DriftTrainer\n",
    "enable_drift = service_configuration_support.get(\"enable_drift\")\n",
    "if enable_drift:\n",
    "    drift_detection_input = {\n",
    "        \"feature_columns\":training_data_info.get(\"feature_columns\"),\n",
    "        \"categorical_columns\":training_data_info.get(\"categorical_columns\"),\n",
    "        \"label_column\": training_data_info.get(\"class_label\"),\n",
    "        \"problem_type\": model_type\n",
    "    }\n",
    "    \n",
    "    drift_trainer = DriftTrainer(data_df,drift_detection_input)\n",
    "    if model_type != \"regression\":\n",
    "        #Note: batch_size can be customized by user as per the training data size\n",
    "        drift_trainer.generate_drift_detection_model(score, batch_size=data_df.shape[0], check_for_ddm_quality=False)\n",
    "    \n",
    "    #Note:\n",
    "    # - Two column constraints are not computed beyond two_column_learner_limit(default set to 200)\n",
    "    # - Categorical columns with large (determined by categorical_unique_threshold; default > 0.8) number of unique values relative to total rows in the column are discarded. \n",
    "    #User can adjust the value depending on the requirement\n",
    "\n",
    "    # user_overrides - Used to override drift constraint learning to selectively learn \n",
    "    # constraints on feature columns. Its a list of configuration, each specifying \n",
    "    # whether to learn distribution and/or range constraint on given set of columns.\n",
    "    # First configuration of a given column would take preference.\n",
    "    # \n",
    "    # \"constraint_type\" can have two possible values : single|double - signifying \n",
    "    # if this configuration is for single column or two column constraint learning.\n",
    "    #\n",
    "    # \"learn_distribution_constraint\" : True|False - signifying whether to learn \n",
    "    # distribution constraint for given config or not.\n",
    "    #\n",
    "    # \"learn_range_constraint\" : True|False - signifying whether to learn range \n",
    "    # constraint for given config or not. Only applicable to numerical feature columns.\n",
    "    # \n",
    "    # \"features\" : [] - provides either a list of feature columns to be governed by \n",
    "    # given configuration for constraint learning.\n",
    "    # Its a list of strings containing feature column names if \"constraint_type\" is \"single\".\n",
    "    # Its a list of list of strings containing feature column names if \"constraint_type\" if \n",
    "    # \"double\". If only one column name is provided, all of the two column constraints \n",
    "    # involving this column will be dictated by given configuration during constraint learning.\n",
    "    # This list is case-insensitive.\n",
    "    #\n",
    "    # In the example below, first config block says do not learn distribution and range single \n",
    "    # column constraints for features \"MARITAL_STATUS\", \"PROFESSION\", \"IS_TENT\" and \"age\".\n",
    "    # Second config block says do not learn distribution and range two column constraints \n",
    "    # where \"IS_TENT\", \"PROFESSION\", and \"AGE\" are one of the two columns. Whereas, specifically, \n",
    "    # do not learn two column distribution and range constraint on combination of \"MARITAL_STATUS\" \n",
    "    # and \"PURCHASE_AMOUNT\".\n",
    "    # \"user_overrides\"= [\n",
    "    #     {\n",
    "    #         \"constraint_type\": \"single\",\n",
    "    #         \"learn_distribution_constraint\": False,\n",
    "    #         \"learn_range_constraint\": False,\n",
    "    #         \"features\": [\n",
    "    #           \"MARITAL_STATUS\",\n",
    "    #           \"PROFESSION\",\n",
    "    #           \"IS_TENT\",\n",
    "    #           \"age\"\n",
    "    #         ]\n",
    "    #     },\n",
    "    #     {\n",
    "    #         \"constraint_type\": \"double\",\n",
    "    #         \"learn_distribution_constraint\": False,\n",
    "    #         \"learn_range_constraint\": False,\n",
    "    #         \"features\": [\n",
    "    #           [\n",
    "    #             \"IS_TENT\"\n",
    "    #           ],\n",
    "    #           [\n",
    "    #             \"MARITAL_STATUS\"\n",
    "    #             \"PURCHASE_AMOUNT\"\n",
    "    #           ],\n",
    "    #           [\n",
    "    #             \"PROFESSION\"\n",
    "    #           ],\n",
    "    #           [\n",
    "    #             \"AGE\"\n",
    "    #           ]\n",
    "    #         ]\n",
    "    #     }\n",
    "    # ]\n",
    "    \n",
    "    drift_trainer.learn_constraints(\n",
    "        two_column_learner_limit=200, categorical_unique_threshold=0.8, user_overrides=[])\n",
    "    drift_trainer.create_archive()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Generate a download link for drift detection model\n",
    "from IPython.display import HTML\n",
    "import base64\n",
    "import io\n",
    "\n",
    "def create_download_link_for_ddm( title = \"Download Drift detection model\", filename = \"drift_detection_model.tar.gz\"):  \n",
    "    \n",
    "    #Retains stats information    \n",
    "    if enable_drift:\n",
    "        with open(filename,\"rb\") as file:\n",
    "            ddm = file.read()\n",
    "        b64 = base64.b64encode(ddm)\n",
    "        payload = b64.decode()\n",
    "        \n",
    "        html = '<a download=\"{filename}\" href=\"data:text/json;base64,{payload}\" target=\"_blank\">{title}</a>'\n",
    "        html = html.format(payload=payload,title=title,filename=filename)\n",
    "        return HTML(html)\n",
    "    else:\n",
    "        print(\"Drift Detection is not enabled. Please enable and rerun the notebook\")\n",
    "\n",
    "create_download_link_for_ddm()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "37b82c9850f337b3f8e26ef3a35bf87c9f2cfa1e4ad2c96ec00819afd6ebf7e9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}