notebooks/Classification_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification models using python and scikit-learn\n",
    "\n",
    "There are many users of online trading platforms and these companies would like to run analytics on and predict churn based on user activity on the platform. Keeping customers happy so they do not move their investments elsewhere is key to maintaining profitability.\n",
    "\n",
    "In this notebook, we'll use scikit-learn to predict classes. scikit-learn provides implementations of many classification algorithms. In here, we have chosen the random forest classification algorithm to walk through all the different steps.\n",
    "\n",
    "\n",
    "<a id=\"top\"></a>\n",
    "## Table of Contents\n",
    "\n",
    "1. [Load libraries](#load_libraries)\n",
    "2. [Data exploration](#explore_data)\n",
    "3. [Prepare data for building classification model](#prepare_data)\n",
    "4. [Split data into train and test sets](#split_data)\n",
    "5. [Helper methods for graph generation](#helper_methods)\n",
    "6. [Prepare Random Forest classification model](#prepare_model)\n",
    "7. [Train Random Forest classification model](#train_model)\n",
    "8. [Test Random Forest classification model](#test_model)\n",
    "9. [Evaluate Random Forest classification model](#evaluate_model)\n",
    "10.[Build K-Nearest classification model](#model_knn)\n",
    "11. [Comparative study of both classification algorithms](#compare_classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick set of instructions to work through the notebook\n",
    "\n",
    "If you are new to Notebooks, here's a quick overview of how to work in this environment.\n",
    "\n",
    "1. The notebook has 2 types of cells - markdown (text) such as this and code such as the one below. \n",
    "2. Each cell with code can be executed independently or together (see options under the Cell menu). When working in this notebook, we will be running one cell at a time.\n",
    "3. To run the cell, position cursor in the code cell and click the Run (arrow) icon. The cell is running when you see the * next to it. Some cells have printable output.\n",
    "4. Work through this notebook by reading the instructions and executing code cell by cell. Some cells will require modifications before you run them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load_libraries\"></a>\n",
    "## 1. Load libraries\n",
    "[Top](#top)\n",
    "\n",
    "\n",
    "Install python modules\n",
    "NOTE! Some pip installs require a kernel restart.\n",
    "The shell command pip install is used to install Python modules. Some installs require a kernel restart to complete. To avoid confusing errors, run the following cell once and then use the Kernel menu to restart the kernel before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas==0.24.2\n",
    "!pip install --user pandas_ml==0.6.1\n",
    "#downgrade matplotlib to bypass issue with confusion matrix being chopped out\n",
    "!pip install matplotlib==3.1.0\n",
    "!pip install --user scikit-learn==0.21.3\n",
    "!pip install -q scikit-plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "\n",
    "import pandas as pd, numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.colors as mcolors\n",
    "import matplotlib.patches as mpatches\n",
    "import scikitplot as skplt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"explore_data\"></a>\n",
    "## 2. Data exploration\n",
    "[Top](#top)\n",
    "\n",
    "In this tutorial, we use a data set that contains information about customers of an online trading platform to classify whether a given customer’s probability of churn will be high, medium, or low. This provides a good example to learn how a classification model is built from start to end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_churn_pd = pd.read_csv(\"https://raw.githubusercontent.com/IBM/ml-learning-path-assets/master/data/mergedcustomers_missing_values_GENDER.csv\")\n",
    "df_churn_pd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use numpy and matplotlib to get some statistics and visualize data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print(\"The dataset contains columns of the following data types : \\n\" +str(df_churn_pd.dtypes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice below that Gender has three missing values. This will be handled in one of the preprocessing steps that is to follow. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The dataset contains following number of records for each of the columns : \\n\" +str(df_churn_pd.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are not satisfied with the representational data, now is the time to get more data to be used for training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( \"Each category within the churnrisk column has the following count : \")\n",
    "print(df_churn_pd.groupby(['CHURNRISK']).size())\n",
    "#bar chart to show split of data\n",
    "index = ['High','Medium','Low']\n",
    "churn_plot = df_churn_pd['CHURNRISK'].value_counts(sort=True, ascending=False).plot(kind='bar',\n",
    "            figsize=(4,4),title=\"Total number for occurences of churn risk \" \n",
    "            + str(df_churn_pd['CHURNRISK'].count()), color=['#BB6B5A','#8CCB9B','#E5E88B'])\n",
    "churn_plot.set_xlabel(\"Churn Risk\")\n",
    "churn_plot.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"prepare_data\"></a>\n",
    "## 3. Data preparation\n",
    "[Top](#top)\n",
    "\n",
    "Data preparation is a very important step in machine learning model building. This is because the model can perform well only when the data it is trained on is good and well prepared. Hence, this step consumes the bulk of a data scientist's time spent building models.\n",
    "\n",
    "During this process, we identify categorical columns in the dataset. Categories need to be indexed, which means the string labels are converted to label indices. These label indices are encoded using One-hot encoding to a binary vector with at most a single value indicating the presence of a specific feature value from among the set of all feature values. This encoding allows algorithms which expect continuous features to use categorical features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "#remove columns that are not required\n",
    "df_churn_pd = df_churn_pd.drop(['ID'], axis=1)\n",
    "\n",
    "df_churn_pd.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### [Preprocessing Data](https://scikit-learn.org/stable/modules/preprocessing.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn provides a method to fill empty values with something that would be applicable in its context. We used the <i><b> SimpleImputer <b></i> class that is provided by Sklearn and filled the missing values with the most frequent value in the column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [One Hot Encoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the categorical columns \n",
    "categoricalColumns = ['GENDER', 'STATUS', 'HOMEOWNER']\n",
    "\n",
    "print(\"Categorical columns : \" )\n",
    "print(categoricalColumns)\n",
    "\n",
    "impute_categorical = SimpleImputer(strategy=\"most_frequent\")\n",
    "\n",
    "onehot_categorical =  OneHotEncoder(handle_unknown='ignore')\n",
    "\n",
    "categorical_transformer = Pipeline(steps=[('impute',impute_categorical),('onehot',onehot_categorical)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numerical columns from the data set are identified, and StandardScaler is applied to each of the columns. This way, each value is subtracted with the mean of its column and divided by its standard deviation.<br>\n",
    "### [Standard Scaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the numerical columns \n",
    "numericalColumns = df_churn_pd.select_dtypes(include=[np.float,np.int]).columns\n",
    "\n",
    "print(\"Numerical columns : \" )\n",
    "print(numericalColumns)\n",
    "\n",
    "scaler_numerical = StandardScaler()\n",
    "\n",
    "numerical_transformer = Pipeline(steps=[('scale',scaler_numerical)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preprocessing techniques that are applied must be customized for each of the columns. Sklearn provides a library called the [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html?highlight=columntransformer#sklearn.compose.ColumnTransformer), which allows a sequence of these techniques to be applied to selective columns using a pipeline.\n",
    "\n",
    "Only the specified columns in transformers are transformed and combined in the output, and the non-specified columns are dropped. By specifying remainder='passthrough', all remaining columns that were not specified in transformers will be automatically passed through"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessorForCategoricalColumns = ColumnTransformer(transformers=[('cat', categorical_transformer, \n",
    "                                                                     categoricalColumns)],\n",
    "                                            remainder=\"passthrough\")\n",
    "preprocessorForAllColumns = ColumnTransformer(transformers=[('cat', categorical_transformer, categoricalColumns),\n",
    "                                                            ('num',numerical_transformer,numericalColumns)],\n",
    "                                            remainder=\"passthrough\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning algorithms cannot use simple text. We must convert the data from text to a number. Therefore, for each string that is a class we assign a label that is a number. For example, in the customer churn data set, the CHURNRISK output label is classified as high, medium, or low and is assigned labels 0, 1, or 2. We use the [LabelEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html?highlight=labelencoder#sklearn.preprocessing.LabelEncoder) class provided by Sklearn for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# prepare data frame for splitting data into train and test datasets\n",
    "\n",
    "features = []\n",
    "features = df_churn_pd.drop(['CHURNRISK'], axis=1)\n",
    "\n",
    "label_churn = pd.DataFrame(df_churn_pd, columns = ['CHURNRISK']) \n",
    "label_encoder = LabelEncoder()\n",
    "label = df_churn_pd['CHURNRISK']\n",
    "\n",
    "label = label_encoder.fit_transform(label)\n",
    "print(\"Encoded value of Churnrisk after applying label encoder : \" + str(label))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  These are some of the popular preprocessing steps that are applied on the data sets. Look at [Data Processing in detail](https://developer.ibm.com/articles/data-preprocessing-in-detail/) for more information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "area = 75\n",
    "x = df_churn_pd['ESTINCOME']\n",
    "y = df_churn_pd['DAYSSINCELASTTRADE']\n",
    "z = df_churn_pd['TOTALDOLLARVALUETRADED']\n",
    "\n",
    "pop_a = mpatches.Patch(color='#BB6B5A', label='High')\n",
    "pop_b = mpatches.Patch(color='#E5E88B', label='Medium')\n",
    "pop_c = mpatches.Patch(color='#8CCB9B', label='Low')\n",
    "def colormap(risk_list):\n",
    "    cols=[]\n",
    "    for l in risk_list:\n",
    "        if l==0:\n",
    "            cols.append('#BB6B5A')\n",
    "        elif l==2:\n",
    "            cols.append('#E5E88B')\n",
    "        elif l==1:\n",
    "            cols.append('#8CCB9B')\n",
    "    return cols\n",
    "\n",
    "fig = plt.figure(figsize=(12,6))\n",
    "fig.suptitle('2D and 3D view of churnrisk data')\n",
    "\n",
    "# First subplot\n",
    "ax = fig.add_subplot(1, 2,1)\n",
    "\n",
    "ax.scatter(x, y, alpha=0.8, c=colormap(label), s= area)\n",
    "ax.set_ylabel('DAYS SINCE LAST TRADE')\n",
    "ax.set_xlabel('ESTIMATED INCOME')\n",
    "\n",
    "plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "\n",
    "# Second subplot\n",
    "ax = fig.add_subplot(1,2,2, projection='3d')\n",
    "\n",
    "ax.scatter(z, x, y, c=colormap(label), marker='o')\n",
    "\n",
    "ax.set_xlabel('TOTAL DOLLAR VALUE TRADED')\n",
    "ax.set_ylabel('ESTIMATED INCOME')\n",
    "ax.set_zlabel('DAYS SINCE LAST TRADE')\n",
    "\n",
    "plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id=\"split_data\"></a>\n",
    "## 4. Split data into test and train\n",
    "[Top](#top)\n",
    "\n",
    "Scikit-learn provides in built API to split the original dataset into train and test datasets. random_state is set to a number to be able to reproduce the same data split combination through multiple runs. \n",
    "\n",
    "[Split arrays or matrices into random train and test subsets](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(features,label , random_state=0)\n",
    "print(\"Dimensions of datasets that will be used for training : Input features\"+str(X_train.shape)+ \n",
    "      \" Output label\" + str(y_train.shape))\n",
    "print(\"Dimensions of datasets that will be used for testing : Input features\"+str(X_test.shape)+ \n",
    "      \" Output label\" + str(y_test.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"helper_methods\"></a>\n",
    "## 5. Helper methods for graph generation\n",
    "[Top](#top)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def colormap(risk_list):\n",
    "    cols=[]\n",
    "    for l in risk_list:\n",
    "        if l==0:\n",
    "            cols.append('#BB6B5A')\n",
    "        elif l==2:\n",
    "            cols.append('#E5E88B')\n",
    "        elif l==1:\n",
    "            cols.append('#8CCB9B')\n",
    "    return cols\n",
    "\n",
    "def two_d_compare(y_test,y_pred,model_name):\n",
    "    #y_pred = label_encoder.fit_transform(y_pred)\n",
    "    #y_test = label_encoder.fit_transform(y_test)\n",
    "    area = (12 * np.random.rand(40))**2 \n",
    "    plt.subplots(ncols=2, figsize=(10,4))\n",
    "    plt.suptitle('Actual vs Predicted data : ' +model_name + '. Accuracy : %.2f' % accuracy_score(y_test, y_pred))\n",
    "\n",
    "    plt.subplot(121)\n",
    "    plt.scatter(X_test['ESTINCOME'], X_test['DAYSSINCELASTTRADE'], alpha=0.8, c=colormap(y_test))\n",
    "    plt.title('Actual')\n",
    "    plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "\n",
    "    plt.subplot(122)\n",
    "    plt.scatter(X_test['ESTINCOME'], X_test['DAYSSINCELASTTRADE'],alpha=0.8, c=colormap(y_pred))\n",
    "    plt.title('Predicted')\n",
    "    plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "\n",
    "    plt.show()\n",
    "    \n",
    "x = X_test['TOTALDOLLARVALUETRADED']\n",
    "y = X_test['ESTINCOME']\n",
    "z = X_test['DAYSSINCELASTTRADE']\n",
    "\n",
    "pop_a = mpatches.Patch(color='#BB6B5A', label='High')\n",
    "pop_b = mpatches.Patch(color='#E5E88B', label='Medium')\n",
    "pop_c = mpatches.Patch(color='#8CCB9B', label='Low')\n",
    "\n",
    "def three_d_compare(y_test,y_pred,model_name):\n",
    "    fig = plt.figure(figsize=(12,10))\n",
    "    fig.suptitle('Actual vs Predicted (3D) data : ' +model_name + '. Accuracy : %.2f' % accuracy_score(y_test, y_pred))\n",
    "    \n",
    "    ax = fig.add_subplot(121, projection='3d')\n",
    "    ax.scatter(x, y, z, c=colormap(y_test), marker='o')\n",
    "    ax.set_xlabel('TOTAL DOLLAR VALUE TRADED')\n",
    "    ax.set_ylabel('ESTIMATED INCOME')\n",
    "    ax.set_zlabel('DAYS SINCE LAST TRADE')\n",
    "    plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "    plt.title('Actual')\n",
    "\n",
    "    ax = fig.add_subplot(122, projection='3d')\n",
    "    ax.scatter(x, y, z, c=colormap(y_pred), marker='o')\n",
    "    ax.set_xlabel('TOTAL DOLLAR VALUE TRADED')\n",
    "    ax.set_ylabel('ESTIMATED INCOME')\n",
    "    ax.set_zlabel('DAYS SINCE LAST TRADE')\n",
    "    plt.legend(handles=[pop_a,pop_b,pop_c])\n",
    "    plt.title('Predicted')\n",
    "\n",
    "    plt.show()\n",
    "    \n",
    "\n",
    "def model_metrics(y_test,y_pred):\n",
    "    print(\"Decoded values of Churnrisk after applying inverse of label encoder : \" + str(np.unique(y_pred)))\n",
    "\n",
    "    skplt.metrics.plot_confusion_matrix(y_test,y_pred,text_fontsize=\"small\",cmap='Greens',figsize=(6,4))\n",
    "    plt.show()\n",
    "    \n",
    "    print(\"The classification report for the model : \\n\\n\"+ classification_report(y_test, y_pred))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"prepare_model\"></a>\n",
    "## 6. Prepare Random Forest classification model\n",
    "[Top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We instantiate a decision-tree based classification algorithm, namely, RandomForestClassifier. Next we define a pipeline to chain together the various transformers and estimators defined during the data preparation step before. \n",
    "Scikit-learn provides APIs that make it easier to combine multiple algorithms into a single pipeline.\n",
    "\n",
    "We fit the pipeline to training data and apply the trained model to transform test data and generate churn risk class prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Understanding Random Forest Classifier](https://towardsdatascience.com/understanding-random-forest-58381e0602d2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model_name = \"Random Forest Classifier\"\n",
    "\n",
    "randomForestClassifier = RandomForestClassifier(n_estimators=100, max_depth=2,random_state=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipelines are a convenient way of designing your data processing in a machine learning flow. The following code example shows how pipelines are set up using sklearn.\n",
    "\n",
    "Read more [Here](https://scikit-learn.org/stable/modules/classes.html?highlight=pipeline#module-sklearn.pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_model = Pipeline(steps=[('preprocessorAll',preprocessorForAllColumns),('classifier', randomForestClassifier)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"train_model\"></a>\n",
    "## 7. Train Random Forest classification model\n",
    "[Top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Build models\n",
    "\n",
    "rfc_model.fit(X_train,y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"test_model\"></a>\n",
    "## 8. Test Random Forest classification model\n",
    "[Top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "y_pred_rfc = rfc_model.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"evaluate_model\"></a>\n",
    "## 9. Evaluate Random Forest classification model\n",
    "[Top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model results\n",
    "\n",
    "In a supervised classification problem such as churn risk classification, we have a true output and a model-generated predicted output for each data point. For this reason, the results for each data point can be assigned to one of four categories:\n",
    "\n",
    "1. True Positive (TP) - label is positive and prediction is also positive\n",
    "2. True Negative (TN) - label is negative and prediction is also negative\n",
    "3. False Positive (FP) - label is negative but prediction is positive\n",
    "4. False Negative (FN) - label is positive but prediction is negative\n",
    "\n",
    "These four numbers are the building blocks for most classifier evaluation metrics. A fundamental point when considering classifier evaluation is that pure accuracy (i.e. was the prediction correct or incorrect) is not generally a good metric. The reason for this is because a dataset may be highly unbalanced. For example, if a model is designed to predict fraud from a dataset where 95% of the data points are not fraud and 5% of the data points are fraud, then a naive classifier that predicts not fraud, regardless of input, will be 95% accurate. For this reason, metrics like precision and recall are typically used because they take into account the type of error. In most applications there is some desired balance between precision and recall, which can be captured by combining the two into a single metric, called the F-measure.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_d_compare(y_test,y_pred_rfc,model_name)\n",
    "\n",
    "#three_d_compare(y_test,y_pred_rfc,model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion matrix \n",
    "\n",
    "In the graph below we have printed a confusion matrix and a self-explanotary classification report.\n",
    "\n",
    "The confusion matrix shows that, 42 mediums were wrongly predicted as high, 2 mediums were wrongly predicted as low and 52 mediums were accurately predicted as mediums."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = label_encoder.inverse_transform(y_test)\n",
    "y_pred_rfc = label_encoder.inverse_transform(y_pred_rfc)\n",
    "model_metrics(y_test,y_pred_rfc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Precision Recall Fscore support](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Understanding the Confusion Matrix](https://towardsdatascience.com/confusion-matrix-for-your-multi-class-machine-learning-model-ff9aa3bf7826)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparative study\n",
    "In the bar chart below, we have compared the random forest classification algorithm output classes against the actual values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uniqueValues, occurCount = np.unique(y_test, return_counts=True)\n",
    "frequency_actual = (occurCount[0],occurCount[2],occurCount[1])\n",
    "\n",
    "uniqueValues, occurCount = np.unique(y_pred_rfc, return_counts=True)\n",
    "frequency_predicted_rfc = (occurCount[0],occurCount[2],occurCount[1])\n",
    "\n",
    "n_groups = 3\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "index = np.arange(n_groups)\n",
    "bar_width = 0.1\n",
    "opacity = 0.8\n",
    "\n",
    "rects1 = plt.bar(index, frequency_actual, bar_width,\n",
    "alpha=opacity,\n",
    "color='g',\n",
    "label='Actual')\n",
    "\n",
    "rects6 = plt.bar(index + bar_width, frequency_predicted_rfc, bar_width,\n",
    "alpha=opacity,\n",
    "color='purple',\n",
    "label='Random Forest - Predicted')\n",
    "\n",
    "plt.xlabel('Churn Risk')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Actual vs Predicted frequency.')\n",
    "plt.xticks(index + bar_width, ('High', 'Medium', 'Low'))\n",
    "plt.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"model_knn\"></a>\n",
    "## 10. Build K-Nearest classification model\n",
    "[Top](#top)\n",
    "\n",
    "K number of nearest points around the data point to be predicted are taken into consideration. These K points at this time, already belong to a class. The data point under consideration, is said to belong to the class with which most number of points from these k points belong to. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "model_name = \"K-Nearest Neighbor Classifier\"\n",
    "\n",
    "knnClassifier = KNeighborsClassifier(n_neighbors = 5, metric='minkowski', p=2)\n",
    "\n",
    "knn_model = Pipeline(steps=[('preprocessorAll',preprocessorForAllColumns),('classifier', knnClassifier)]) \n",
    "\n",
    "knn_model.fit(X_train,y_train)\n",
    "\n",
    "y_pred_knn = knn_model.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = label_encoder.transform(y_test)\n",
    "two_d_compare(y_test,y_pred_knn,model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = label_encoder.inverse_transform(y_test)\n",
    "y_pred_knn = label_encoder.inverse_transform(y_pred_knn)\n",
    "model_metrics(y_test,y_pred_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"compare_classification\"></a>\n",
    "## 11. Comparative study of both classification algorithms. \n",
    "[Top](#top)\n",
    "\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "uniqueValues, occurCount = np.unique(y_test, return_counts=True)\n",
    "frequency_actual = (occurCount[0],occurCount[2],occurCount[1])\n",
    "\n",
    "uniqueValues, occurCount = np.unique(y_pred_rfc, return_counts=True)\n",
    "frequency_predicted_rfc = (occurCount[0],occurCount[2],occurCount[1])\n",
    "\n",
    "uniqueValues, occurCount = np.unique(y_pred_knn, return_counts=True)\n",
    "frequency_predicted_knn = (occurCount[0],occurCount[2],occurCount[1])\n",
    "\n",
    "n_groups = 3\n",
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "index = np.arange(n_groups)\n",
    "bar_width = 0.1\n",
    "opacity = 0.8\n",
    "\n",
    "rects1 = plt.bar(index, frequency_actual, bar_width,\n",
    "alpha=opacity,\n",
    "color='g',\n",
    "label='Actual')\n",
    "\n",
    "\n",
    "rects6 = plt.bar(index + bar_width*2, frequency_predicted_rfc, bar_width,\n",
    "alpha=opacity,\n",
    "color='purple',\n",
    "label='Random Forest - Predicted')\n",
    "\n",
    "rects4 = plt.bar(index + bar_width*4, frequency_predicted_knn, bar_width,\n",
    "alpha=opacity,\n",
    "color='b',\n",
    "label='K-Nearest Neighbor - Predicted')\n",
    "\n",
    "plt.xlabel('Churn Risk')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Actual vs Predicted frequency.')\n",
    "plt.xticks(index + bar_width, ('High', 'Medium', 'Low'))\n",
    "plt.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until evaluation provides satisfactory scores, you would repeat the data preprocessing through evaluating steps by tuning what are called the hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Choosing the right estimator](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For a comparative study of some of the current most popular algorithms Please refer to this [tutorial](https://developer.ibm.com/tutorials/learn-classification-algorithms-using-python-and-scikit-learn/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><font size=-1 color=gray>\n",
    "&copy; Copyright 2019 IBM Corp. All Rights Reserved.\n",
    "<p>\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file\n",
    "except in compliance with the License. You may obtain a copy of the License at\n",
    "https://www.apache.org/licenses/LICENSE-2.0\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the\n",
    "License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n",
    "express or implied. See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "</font></p>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3.7",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}