notebooks/GPT_insights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating insights from the model's predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append('..')\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=3, suppress=True)\n",
    "\n",
    "from joblib import load\n",
    "\n",
    "# Interpretability\n",
    "import shap\n",
    "\n",
    "import os\n",
    "import openai\n",
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Set transformers output to Pandas DataFrame instead of NumPy array\n",
    "from sklearn import set_config\n",
    "set_config(transform_output=\"pandas\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and prepare the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = pd.read_csv('../data/raw/loan-data-test.csv')\n",
    "X_test.drop('Loan_ID', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading preprocessing pipeline and model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../models/model.pkl', 'rb') as f:\n",
    "    model = load(f)\n",
    "\n",
    "with open('../models/preprocessor.pkl', 'rb') as f:\n",
    "    preprocessor = load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preprocessing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mapping number of dependents to numerical values\n",
    "X_test.Dependents.replace('3+', 3, inplace=True)\n",
    "\n",
    "# Transforming the test set\n",
    "X_test = preprocessor.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the SHAP values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Unscaling the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the scaler and encoder object from the pipeline\n",
    "scaler = preprocessor.named_transformers_['numerical']['scaler']\n",
    "encoder = preprocessor.named_transformers_['categorical']['onehot']\n",
    "\n",
    "# Unscale the  data \n",
    "X_test_num_unscaled = scaler.inverse_transform(X_test[scaler.feature_names_in_])\n",
    "X_test_num_unscaled_df = pd.DataFrame(data=X_test_num_unscaled, columns=scaler.feature_names_in_)\n",
    "\n",
    "# Get the one-hot encoded features\n",
    "X_test_cat = X_test[encoder.get_feature_names_out()]\n",
    "\n",
    "# Reset the index before concatenation\n",
    "X_test_num_unscaled_df.reset_index(drop=True, inplace=True)\n",
    "X_test_cat.reset_index(drop=True, inplace=True)\n",
    "\n",
    "# Concat the unscaled numeric data and the categorical data\n",
    "X_test_unscaled = pd.concat([X_test_num_unscaled_df, X_test_cat], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a shap explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainer = shap.LinearExplainer(model, X_test)\n",
    "shap_explanation = explainer(X_test)\n",
    "\n",
    "# Replace the scaled values by the original ones\n",
    "shap_explanation.data = X_test_unscaled.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Format explanations in JSON\n",
    "\n",
    "This is a simplified version of the function [explanation_to_json](../src/reporting/utils.py) that is used in the web app.\n",
    "The original functions performs additional steps to make the explanations more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = X_test.columns\n",
    "explanation_jsons = []\n",
    "i = 10\n",
    "\n",
    "# Predict the status\n",
    "predicted_status = model.predict(X_test.iloc[[i]])\n",
    "\n",
    "# Predict the probability\n",
    "predicted_proba = np.round(\n",
    "    model.predict_proba(X_test.iloc[[i]])[0][1] * 100\n",
    ").astype(int)\n",
    "\n",
    "# Get the explanation for the predicted status\n",
    "for name, value, shap_value in zip(feature_names, shap_explanation.data[i], shap_explanation.values[i]):\n",
    "    explanation_json = {}\n",
    "    explanation_json[\"Name\"] = name\n",
    "    explanation_json[\"Value\"] = value\n",
    "    explanation_json[\"SHAP Value\"] = shap_value\n",
    "    explanation_json[\"Effect on Approval\"] = \"Positive\" if shap_value > 0 else \"Negative\"\n",
    "\n",
    "    explanation_jsons.append(explanation_json)\n",
    "\n",
    "# explanation_jsons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   0. 5720.    0.  110.  360.    1.    0.    0.    0.    1.    1.]\n",
      "[-0.083  0.147 -0.317  0.029 -0.006 -0.012  0.027  0.116 -0.115 -0.047\n",
      "  0.531]\n"
     ]
    }
   ],
   "source": [
    "print(shap_explanation.data[0])\n",
    "print(shap_explanation.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'Name': 'Dependents', 'Value': 0.0, 'SHAP Value': -0.08346984991178112, 'Effect on Approval': 'Negative'}, {'Name': 'Applicant Income', 'Value': 5720, 'SHAP Value': 0.14706541892812847, 'Effect on Approval': 'Positive'}, {'Name': 'Coapplicant Income', 'Value': 0, 'SHAP Value': -0.3171539419119009, 'Effect on Approval': 'Negative'}, {'Name': 'Loan Amount', 'Value': 110000, 'SHAP Value': 0.029009653363208033, 'Effect on Approval': 'Positive'}, {'Name': 'Loan Amount Term', 'Value': 360, 'SHAP Value': -0.005628890210890795, 'Effect on Approval': 'Negative'}, {'Name': 'Gender', 'Value': 'Male', 'SHAP Value': -0.011940938055301362, 'Effect on Approval': 'Negative'}, {'Name': 'Self Employed', 'Value': 'No', 'SHAP Value': 0.026858270835239666, 'Effect on Approval': 'Positive'}, {'Name': 'Property Area', 'Value': 'Urban', 'SHAP Value': -0.04697147224187069, 'Effect on Approval': 'Negative'}, {'Name': 'Has Credit History', 'Value': 'Yes', 'SHAP Value': 0.5307220385544934, 'Effect on Approval': 'Positive'}]\n"
     ]
    }
   ],
   "source": [
    "explanation_jsons = []\n",
    "feature_names = [\n",
    "    'Dependents', 'Applicant Income', 'Coapplicant Income', 'Loan Amount',\n",
    "    'Loan Amount Term', 'Gender', \n",
    "    # 'Married', 'Education', \n",
    "    'Self Employed', 'Property Area: Rural',\n",
    "    'Property Area: Semi-urban', 'Property Area: Urban', 'Has Credit History'\n",
    "]\n",
    "\n",
    "numerical_features = [\n",
    "    'Applicant Income', 'Coapplicant Income', \n",
    "    'Loan Amount', 'Loan Amount Term'\n",
    "]\n",
    "\n",
    "explanation_jsons = []\n",
    "\n",
    "for name, value, shap_value in zip(feature_names, \n",
    "    shap_explanation.data[0], shap_explanation.values[0]):     \n",
    "            \n",
    "    explanation_json = {}\n",
    "\n",
    "    # Deleted features\n",
    "    # elif name == \"Married\":\n",
    "    #     value = \"Yes\" if value == 1 else \"No\"\n",
    "    # elif name == \"Education\":\n",
    "    #     value = \"Not Graduate\" if value == 1 else \"Graduate\"\n",
    "    \n",
    "    # > Map the values to strings for interpretability\n",
    "    if name == \"Gender\":\n",
    "        value = \"Male\" if value == 1 else \"Female\"\n",
    "    elif name == \"Self Employed\":\n",
    "        value = \"Yes\" if value == 1 else \"No\"\n",
    "    elif name == \"Has Credit History\":\n",
    "        value = \"Yes\" if value == 1 else \"No\"\n",
    "\n",
    "    # > Map \"Property Area\" to it's original category\n",
    "    # keep only the value that is equal to 1 \n",
    "    # since the property area is one-hot encoded\n",
    "    elif name == \"Property Area: Rural\":\n",
    "        if value == 1:\n",
    "            name = \"Property Area\"\n",
    "            value = \"Rural\"\n",
    "        else:\n",
    "            continue\n",
    "    elif name == \"Property Area: Semi-urban\":\n",
    "        if value == 1:\n",
    "            name = \"Property Area\"\n",
    "            value = \"Semi-urban\"\n",
    "        else:\n",
    "            continue\n",
    "    elif name == \"Property Area: Urban\":\n",
    "        if value == 1:\n",
    "            name = \"Property Area\"\n",
    "            value = \"Urban\"\n",
    "        else:\n",
    "            continue\n",
    "\n",
    "\n",
    "    # Map the \"Loan Amount\" values to thousands\n",
    "    # since the original data is in thousands\n",
    "    elif name == \"Loan Amount\":\n",
    "        value = value * 1000\n",
    "\n",
    "    explanation_json[\"Name\"] = name\n",
    "    \n",
    "    # Round numerical features value\n",
    "    if name in numerical_features:\n",
    "        explanation_json[\"Value\"] = round(value)\n",
    "    else:\n",
    "        explanation_json[\"Value\"] = value\n",
    "\n",
    "    explanation_json[\"SHAP Value\"] = shap_value\n",
    "    explanation_json[\"Effect on Approval\"] = \"Positive\" if shap_value > 0 else \"Negative\"\n",
    "    \n",
    "    explanation_jsons.append(explanation_json)\n",
    "\n",
    "print(explanation_jsons)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(\"\\n\\n---------->\\nshap_explanation\")\n",
    "# print(shap_explanation.data[0])\n",
    "# # print(\">\")\n",
    "# print(shap_explanation.values[0])\n",
    "# print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Value</th>\n",
       "      <th>SHAP Value</th>\n",
       "      <th>Effect on Approval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dependents</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.083470</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Applicant Income</td>\n",
       "      <td>5720</td>\n",
       "      <td>0.147065</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Coapplicant Income</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.317154</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Loan Amount</td>\n",
       "      <td>110000</td>\n",
       "      <td>0.029010</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loan Amount Term</td>\n",
       "      <td>360</td>\n",
       "      <td>-0.005629</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Gender</td>\n",
       "      <td>Male</td>\n",
       "      <td>-0.011941</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Self Employed</td>\n",
       "      <td>No</td>\n",
       "      <td>0.026858</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Property Area</td>\n",
       "      <td>Urban</td>\n",
       "      <td>-0.046971</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Has Credit History</td>\n",
       "      <td>Yes</td>\n",
       "      <td>0.530722</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Name   Value  SHAP Value Effect on Approval\n",
       "0          Dependents     0.0   -0.083470           Negative\n",
       "1    Applicant Income    5720    0.147065           Positive\n",
       "2  Coapplicant Income       0   -0.317154           Negative\n",
       "3         Loan Amount  110000    0.029010           Positive\n",
       "4    Loan Amount Term     360   -0.005629           Negative\n",
       "5              Gender    Male   -0.011941           Negative\n",
       "6       Self Employed      No    0.026858           Positive\n",
       "7       Property Area   Urban   -0.046971           Negative\n",
       "8  Has Credit History     Yes    0.530722           Positive"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(explanation_jsons)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Visualizing the JSON arrays as a DataFrame**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Value</th>\n",
       "      <th>SHAP Value</th>\n",
       "      <th>Effect on Approval</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Has Credit History</td>\n",
       "      <td>Yes</td>\n",
       "      <td>0.530722</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Coapplicant Income</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.317154</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Applicant Income</td>\n",
       "      <td>5720</td>\n",
       "      <td>0.147065</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dependents</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.083470</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Property Area</td>\n",
       "      <td>Urban</td>\n",
       "      <td>-0.046971</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Loan Amount</td>\n",
       "      <td>110000</td>\n",
       "      <td>0.029010</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Self Employed</td>\n",
       "      <td>No</td>\n",
       "      <td>0.026858</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Gender</td>\n",
       "      <td>Male</td>\n",
       "      <td>-0.011941</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Loan Amount Term</td>\n",
       "      <td>360</td>\n",
       "      <td>-0.005629</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Name   Value  SHAP Value Effect on Approval\n",
       "0  Has Credit History     Yes    0.530722           Positive\n",
       "1  Coapplicant Income       0   -0.317154           Negative\n",
       "2    Applicant Income    5720    0.147065           Positive\n",
       "3          Dependents     0.0   -0.083470           Negative\n",
       "4       Property Area   Urban   -0.046971           Negative\n",
       "5         Loan Amount  110000    0.029010           Positive\n",
       "6       Self Employed      No    0.026858           Positive\n",
       "7              Gender    Male   -0.011941           Negative\n",
       "8    Loan Amount Term     360   -0.005629           Negative"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explanation_df = pd.DataFrame(explanation_jsons)\n",
    "\n",
    "# Sort feature by impact score\n",
    "explanation_df[\"SHAP Value (Abs)\"] = explanation_df[\"SHAP Value\"].abs()\n",
    "explanation_df.sort_values(by=\"SHAP Value (Abs)\", ascending=False, inplace=True)\n",
    "explanation_df.drop(\"SHAP Value (Abs)\", axis=1, inplace=True)\n",
    "explanation_df.reset_index(drop=True, inplace=True)\n",
    "explanation_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generation GPT predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = load_dotenv(\"../credentials.env\")\n",
    "openai.api_key = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompting GPT-3.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Below are the definitions of the features:\n",
      "- Dependents: Number of dependents of the applicant \n",
      "- ApplicantIncome: Income of the applicant\n",
      "- CoapplicantIncome: Income of the co-applicant\n",
      "- LoanAmount: Loan amount \n",
      "- Loan_Amount_Term: Term of the loan in months\n",
      "- Gender: then gender of the applicant\n",
      "- Self Employed: wheather the applicant is self-employed or not\n",
      "- Property Area:Rural: \"Yes\" if the property is in a rural area, \"No\" otherwise\n",
      "- PropertyArea: Semiurban: \"Yes\" if the property is in a semiurban area, \"No\" otherwise\n",
      "- Property_Area: Urban: \"Yes\" if the property is in an urban area, \"No\" otherwise\n",
      "- Has Credit History: \"Yes\" if the applicant has a credit history, \"No\" otherwise\n",
      "\n",
      "Below are the names, values, SHAP values, and effects for each prediction in a JSON format:\n",
      "[{'Name': 'Dependents', 'Value': 0.0, 'SHAP Value': -0.08346984991178112, 'Effect on Approval': 'Negative'}, {'Name': 'Applicant Income', 'Value': 5720, 'SHAP Value': 0.14706541892812847, 'Effect on Approval': 'Positive'}, {'Name': 'Coapplicant Income', 'Value': 0, 'SHAP Value': -0.3171539419119009, 'Effect on Approval': 'Negative'}, {'Name': 'Loan Amount', 'Value': 110000, 'SHAP Value': 0.029009653363208033, 'Effect on Approval': 'Positive'}, {'Name': 'Loan Amount Term', 'Value': 360, 'SHAP Value': -0.005628890210890795, 'Effect on Approval': 'Negative'}, {'Name': 'Gender', 'Value': 'Male', 'SHAP Value': -0.011940938055301362, 'Effect on Approval': 'Negative'}, {'Name': 'Self Employed', 'Value': 'No', 'SHAP Value': 0.026858270835239666, 'Effect on Approval': 'Positive'}, {'Name': 'Property Area', 'Value': 'Urban', 'SHAP Value': -0.04697147224187069, 'Effect on Approval': 'Negative'}, {'Name': 'Has Credit History', 'Value': 'Yes', 'SHAP Value': 0.5307220385544934, 'Effect on Approval': 'Positive'}]\n",
      "\n",
      "Below is the prediction of the model:\n",
      "Predicted status: [1]\n",
      "Probability of approval: 51%\n",
      "\n",
      "-----\n",
      "Based on the information on feature names, values, SHAP values, and effects, \n",
      "generate a report to explain the model's decision in simple terms.\n",
      "Below is an example of response so that you can get the pattern,\n",
      "rewrite it to fit the current context based on the information above\n",
      "but Keep the same markdown structure (e.g. for the level 3 titles ###).\n",
      "The bulleted list should be ordered by magnitude of impact.\n",
      "\n",
      "Your loan application has been approved. Several factors contributed to this decision.\n",
      "\n",
      "### What you did well\n",
      "- **Income**: You have an income of \\$4,235. This factor significantly boosts your chances of approval as a higher income increases the likelihood of getting the loan approved.\n",
      "- **Co-applicant's Income**: You have a co-applicant with an income of \\$3000. This factor significantly boosts your chances of approval, as a higher co-applicant income increases the likelihood of getting the loan approved.\n",
      "- **Requested Loan Amount:** Your loan request of \\$77,000 falls within the lower range of our allowable amount, which spans from \\$9,000 to \\$700,000. This contributed positively to the approval decision.\n",
      "- **Credit History:** You have a credit history, which is required for loan approval.\n",
      "\n",
      "### What you need to work on\n",
      "- **Loan Term Duration:** The chosen loan term of 360 months (30 years) exceeds the midpoint in our range of 12 to 480 months. Opting for a longer loan term slightly diminishes your chances of approval.\n",
      "\n",
      "IMPORTANT:\n",
      "- Do not mention the gender as it is not a factor to improve or work on.\n",
      "- Do not recommend improve the applican't or co-applicant's income as it is not something that can be improved.\n",
      "\n",
      "\n",
      "Conclude with a summary of the most important factors and their effects on the decision.\n",
      "\n",
      "Recommend actions to improve the chances of approval.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# You are a loan eligibility officer. \n",
    "# Behind our system, we use a machine learning model to predict the loan status of the applicant.\n",
    "system_prompt = \"\"\"\n",
    "The system evaluates loan applications using applicant data. \n",
    "You need to explain the system's decision, considering features and their impacts, and this explanation is tailored for the non-technical applicant. \n",
    "No greetings or closings are necessary. \n",
    "Emphasize the features that had the most influence on the system's decision and how they affected that decision.\n",
    "When you mention a feature, include the feature's name and value.\n",
    "Use the term \"system\" to reference the model and avoid technical jargon related to the SHAP values.\n",
    "\n",
    "IMPORTANT\n",
    "---------\n",
    "Higher ApplicantIncome, CoapplicantIncome and LoanAmount are associated with a higher probability of approval. \n",
    "Higher LoanAmount and Loan_Amount_Term are associated with a lower probability of approval.\n",
    "Loan Amount ranges from $9 to $700 (in thousands).\n",
    "Loan Amount Term ranges from 12 to 480 months.\n",
    "\"\"\"\n",
    "\n",
    "system_prompt = \"\"\"\n",
    "The system evaluates loan applications using applicant data. \n",
    "You need to explain the system's decision, considering features and their impacts, and this explanation is tailored for the non-technical applicant. \n",
    "No greetings or closings are necessary. \n",
    "Emphasize the features that had the most influence on the system's decision and how they affected that decision.\n",
    "When you mention a feature, include the feature's name and value.\n",
    "Use the term \"system\" to reference the model and avoid technical jargon related to the SHAP values.\n",
    "\n",
    "IMPORTANT\n",
    "---------\n",
    "Higher ApplicantIncome, CoapplicantIncome and LoanAmount are associated with a higher probability of approval. \n",
    "Higher LoanAmount and Loan_Amount_Term are associated with a lower probability of approval.\n",
    "Loan Amount ranges from $9 to $700 (in thousands).\n",
    "Loan Amount Term ranges from 12 to 480 months.\n",
    "\"\"\"\n",
    "\n",
    "response_template = \"\"\"\n",
    "Your loan application has been approved. Several factors contributed to this decision.\n",
    "\n",
    "### What you did well\n",
    "- **Income**: You have an income of \\$4,235. This factor significantly boosts your chances of approval as a higher income increases the likelihood of getting the loan approved.\n",
    "- **Co-applicant's Income**: You have a co-applicant with an income of \\$3000. This factor significantly boosts your chances of approval, as a higher co-applicant income increases the likelihood of getting the loan approved.\n",
    "- **Requested Loan Amount:** Your loan request of \\$77,000 falls within the lower range of our allowable amount, which spans from \\$9,000 to \\$700,000. This contributed positively to the approval decision.\n",
    "- **Credit History:** You have a credit history, which is required for loan approval.\n",
    "\n",
    "### What you need to work on\n",
    "- **Loan Term Duration:** The chosen loan term of 360 months (30 years) exceeds the midpoint in our range of 12 to 480 months. Opting for a longer loan term slightly diminishes your chances of approval.\n",
    "\n",
    "IMPORTANT:\n",
    "- Do not mention the gender as it is not a factor to improve or work on.\n",
    "- Do not recommend improve the applican't or co-applicant's income as it is not something that can be improved.\n",
    "\"\"\"\n",
    "# In conclusion, the factors that most impacted the decision were your income and the co-applicant's income, the requested loan amount and its duration, along with the number of dependents.\n",
    "\n",
    "\n",
    "query = f\"\"\"\n",
    "Below are the definitions of the features:\n",
    "- Dependents: Number of dependents of the applicant \n",
    "- ApplicantIncome: Income of the applicant\n",
    "- CoapplicantIncome: Income of the co-applicant\n",
    "- LoanAmount: Loan amount \n",
    "- Loan_Amount_Term: Term of the loan in months\n",
    "- Gender: then gender of the applicant\n",
    "- Self Employed: wheather the applicant is self-employed or not\n",
    "- Property Area:Rural: \"Yes\" if the property is in a rural area, \"No\" otherwise\n",
    "- PropertyArea: Semiurban: \"Yes\" if the property is in a semiurban area, \"No\" otherwise\n",
    "- Property_Area: Urban: \"Yes\" if the property is in an urban area, \"No\" otherwise\n",
    "- Has Credit History: \"Yes\" if the applicant has a credit history, \"No\" otherwise\n",
    "\n",
    "Below are the names, values, SHAP values, and effects for each prediction in a JSON format:\n",
    "{explanation_jsons}\n",
    "\n",
    "Below is the prediction of the model:\n",
    "Predicted status: {predicted_status}\n",
    "Probability of approval: {predicted_proba}%\n",
    "\n",
    "-----\n",
    "Based on the information on feature names, values, SHAP values, and effects, \n",
    "generate a report to explain the model's decision in simple terms.\n",
    "Below is an example of response so that you can get the pattern,\n",
    "rewrite it to fit the current context based on the information above\n",
    "but Keep the same markdown structure (e.g. for the level 3 titles ###).\n",
    "The bulleted list should be ordered by magnitude of impact.\n",
    "{response_template}\n",
    "\n",
    "Conclude with a summary of the most important factors and their effects on the decision.\n",
    "\n",
    "Recommend actions to improve the chances of approval.\n",
    "\"\"\"\n",
    "\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating the report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: This is $ a string with $ signs, but \\$ some are already escaped: \\$ \\$.\n",
      "Escaped: This is \\$ a string with \\$ signs, but \\$ some are already escaped: \\$ \\$.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "def selectively_escape_dollar_sign(string):\n",
    "    def replace_unescaped(match):\n",
    "        # Replace unescaped dollar signs with their escaped form\n",
    "        return match.group().replace('$', r'\\$')\n",
    "\n",
    "    # Use a regular expression to find unescaped dollar signs\n",
    "    pattern = re.compile(r'(?<!\\\\)\\$')\n",
    "    return pattern.sub(replace_unescaped, string)\n",
    "\n",
    "# Example usage:\n",
    "query_template = \"This is $ a string with $ signs, but \\\\$ some are already escaped: \\\\$ \\\\$.\"\n",
    "\n",
    "escaped_query_template = selectively_escape_dollar_sign(query_template)\n",
    "\n",
    "print(\"Original:\", query_template)\n",
    "print(\"Escaped:\", escaped_query_template)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_response(system_prompt, query):\n",
    "\t\"\"\"Generate a response to a query based on a system prompt\"\"\"\n",
    "\tcompletion = openai.chat.completions.create(\n",
    "\t\tmodel=\"gpt-3.5-turbo\",\n",
    "\t\tmessages=[\n",
    "\t\t\t{\"role\": \"system\", \"content\": system_prompt},\n",
    "\t\t\t{\"role\": \"user\", \"content\": query}\n",
    "\t\t]\n",
    "\t)\n",
    "\n",
    "\treturn completion.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Congratulations! Your loan application has been approved. The system evaluated several factors to make this decision. Let's take a closer look at these factors and their effects on the approval decision.\n",
      "\n",
      "### Factors that contributed to the approval decision:\n",
      "\n",
      "- **Income**: Your income of $5,720 positively influenced the approval decision. A higher income increases the likelihood of getting the loan approved.\n",
      "\n",
      "- **Credit History**: Having a credit history played a significant role in the approval decision. The system considers this as a crucial factor, and your credit history being positive positively impacted the decision.\n",
      "\n",
      "- **Requested Loan Amount**: The system took into account the loan amount you requested, which was $110,000. This amount falls within the acceptable range of loan amounts, which spans from $9,000 to $700,000. Your loan request contributed to the approval decision.\n",
      "\n",
      "### Factors to be aware of:\n",
      "\n",
      "- **Loan Term Duration**: The chosen loan term of 360 months (30 years) slightly diminished your chances of approval. The system prefers loan terms that are closer to the midpoint of the acceptable range, which is from 12 to 480 months.\n",
      "\n",
      "To summarize, the most significant factors that influenced the system's decision were your income, credit history, and the loan amount you requested. These factors significantly contributed to the approval decision.\n",
      "\n",
      "To improve your chances of approval in the future, it would be beneficial to consider the following actions:\n",
      "\n",
      "- Opt for a loan term duration closer to the midpoint of the acceptable range. This could increase your chances of approval.\n",
      "\n",
      "It's important to remember that the approval decision depends on various factors, and the system considers them collectively. By considering these areas for improvement, you can increase the likelihood of approval for future loan applications.\n"
     ]
    }
   ],
   "source": [
    "response = generate_response(system_prompt, query)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|    | Name               | Value   |   Impact Score | Effect on Approval   |\n",
      "|---:|:-------------------|:--------|---------------:|:---------------------|\n",
      "|  0 | Has Credit History | Yes     |     0.530722   | Positive             |\n",
      "|  1 | Coapplicant Income | 0       |    -0.317154   | Negative             |\n",
      "|  2 | Applicant Income   | 5720    |     0.147065   | Positive             |\n",
      "|  3 | Dependents         | 0.0     |    -0.0834698  | Negative             |\n",
      "|  4 | Property Area      | Urban   |    -0.0469715  | Negative             |\n",
      "|  5 | Loan Amount        | 110000  |     0.0290097  | Positive             |\n",
      "|  6 | Self Employed      | No      |     0.0268583  | Positive             |\n",
      "|  7 | Gender             | Male    |    -0.0119409  | Negative             |\n",
      "|  8 | Loan Amount Term   | 360     |    -0.00562889 | Negative             |\n"
     ]
    }
   ],
   "source": [
    "# Rename SHAP value to Impact Score for better understanding\n",
    "explanation_df.rename(columns={\"SHAP Value\": \"Impact Score\"}, inplace=True)\n",
    "\n",
    "# Convert my dataframe as markdown table\n",
    "print(explanation_df.to_markdown())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will save the report as Markdown file\n",
    "# The report will contains the dataframe as a table and the response from GPT-3\n",
    "report = f\"\"\"\n",
    "# Loan Approval Decision Report\n",
    "\n",
    "## Applicant Information\n",
    "\n",
    "{explanation_df.to_markdown()}\n",
    "\n",
    "## Model Decision\n",
    "\n",
    "{response}\n",
    "\"\"\"\n",
    "\n",
    "with open('../reports/loan_approval_decision_report.md', 'w') as f:\n",
    "    f.write(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# Loan Approval Decision Report\n",
      "\n",
      "## Applicant Information\n",
      "\n",
      "|    | Name               | Value   |   Impact Score | Effect on Approval   |\n",
      "|---:|:-------------------|:--------|---------------:|:---------------------|\n",
      "|  0 | Has Credit History | Yes     |     0.530722   | Positive             |\n",
      "|  1 | Coapplicant Income | 0       |    -0.317154   | Negative             |\n",
      "|  2 | Applicant Income   | 5720    |     0.147065   | Positive             |\n",
      "|  3 | Dependents         | 0.0     |    -0.0834698  | Negative             |\n",
      "|  4 | Property Area      | Urban   |    -0.0469715  | Negative             |\n",
      "|  5 | Loan Amount        | 110000  |     0.0290097  | Positive             |\n",
      "|  6 | Self Employed      | No      |     0.0268583  | Positive             |\n",
      "|  7 | Gender             | Male    |    -0.0119409  | Negative             |\n",
      "|  8 | Loan Amount Term   | 360     |    -0.00562889 | Negative             |\n",
      "\n",
      "## Model Decision\n",
      "\n",
      "Your loan application has been approved. Several factors contributed to this decision.\n",
      "\n",
      "### What you did well\n",
      "- **Income**: You have an income of $5,720. This factor significantly boosts your chances of approval as a higher income increases the likelihood of getting the loan approved.\n",
      "- **Credit History:** You have a credit history, which is required for loan approval.\n",
      "- **Requested Loan Amount:** Your loan request of $110,000 falls within the lower range of our allowable amount, which spans from $9,000 to $700,000. This contributed positively to the approval decision.\n",
      "\n",
      "### What you need to work on\n",
      "- **Loan Term Duration:** The chosen loan term of 360 months (30 years) exceeds the midpoint in our range of 12 to 480 months. Opting for a longer loan term slightly diminishes your chances of approval.\n",
      "\n",
      "Conclude with a summary of the most important factors and their effects on the decision.\n",
      "\n",
      "The most important factors that influenced the decision were:\n",
      "1. **Credit History**: Having a credit history had the most positive impact on the approval decision, increasing your chances of approval significantly.\n",
      "2. **Income**: Your higher income played a significant role in the approval decision, increasing the likelihood of getting the loan approved.\n",
      "3. **Loan Amount**: Requesting a loan within the allowable limit positively influenced the decision.\n",
      "\n",
      "Recommend actions to improve the chances of approval.\n",
      "\n",
      "To improve your chances of approval in the future:\n",
      "- Opt for a shorter loan term, as longer loan terms slightly reduce the likelihood of approval. Consider choosing a term closer to the midpoint of our allowable range.\n",
      "- Continue to maintain a good credit history, as it positively impacts the approval decision.\n",
      "- Increase your income if possible, as higher incomes increase the likelihood of loan approval.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(report)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}