1-data-science-and-ml-tools/scikit-learn-workflow-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Simple Scikit-Learn Classification Workflow\n",
    "\n",
    "This notebook shows a breif workflow you might use with `scikit-learn` to build a machine learning model to classify whether or not a patient has heart disease.\n",
    "\n",
    "It follows the diagram below:\n",
    "\n",
    "<img src=\"../images/sklearn-workflow.png\"/>\n",
    "\n",
    "**Note:** This workflow assumes your data is ready to be used with machine learning models (is numerical, has no missing values)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard imports\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get the data ready"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   63    1   3       145   233    1        0      150      0      2.3      0   \n",
       "1   37    1   2       130   250    0        1      187      0      3.5      0   \n",
       "2   41    0   1       130   204    0        0      172      0      1.4      2   \n",
       "3   56    1   1       120   236    0        1      178      0      0.8      2   \n",
       "4   57    0   0       120   354    0        1      163      1      0.6      2   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   0     1       1  \n",
       "1   0     2       1  \n",
       "2   0     2       1  \n",
       "3   0     2       1  \n",
       "4   0     2       1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import dataset\n",
    "heart_disease = pd.read_csv(\"../data/heart-disease.csv\")\n",
    "\n",
    "# View the data\n",
    "heart_disease.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this example, we're going to use all of the columns except the target column to predict the targert column.\n",
    "\n",
    "In other words, using a patient's medical and demographic data to predict whether or not they have heart disease."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create X (all the feature columns)\n",
    "X = heart_disease.drop(\"target\", axis=1)\n",
    "\n",
    "# Create y (the target column)\n",
    "y = heart_disease[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((227, 13), (76, 13), (227,), (76,))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split the data into training and test sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "# View the data shapes\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Choose the model/estimator\n",
    "\n",
    "You can do this using the [Scikit-Learn machine learning map](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html).\n",
    "\n",
    "<img src=\"../images/sklearn-ml-map.png\" width=500/>\n",
    "\n",
    "In Scikit-Learn, machine learning models are referred to as estimators.\n",
    "\n",
    "In this case, since we're working on a classification problem, we've chosen the RandomForestClassifier estimator which is part of the ensembles module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "model = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Fit the model to the data and use it to make a prediction\n",
    "\n",
    "A model will (attempt to) learn the patterns in a dataset by calling the `fit()` function on it and passing it the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/daniel/Desktop/ml-course/work-in-progress/env/lib/python3.7/site-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "                       max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=10,\n",
       "                       n_jobs=None, oob_score=False, random_state=None,\n",
       "                       verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a model has learned patterns in data, you can use them to make a prediction with the `predict()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions\n",
    "y_preds = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0,\n",
       "       0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0,\n",
       "       1, 1, 0, 0, 0, 0, 1, 1, 0, 0])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will be in the same format as y_test\n",
    "y_preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age          59.0\n",
       "sex           1.0\n",
       "cp            0.0\n",
       "trestbps    110.0\n",
       "chol        239.0\n",
       "fbs           0.0\n",
       "restecg       0.0\n",
       "thalach     142.0\n",
       "exang         1.0\n",
       "oldpeak       1.2\n",
       "slope         1.0\n",
       "ca            1.0\n",
       "thal          3.0\n",
       "Name: 206, dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.loc[206]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age          59.0\n",
       "sex           1.0\n",
       "cp            0.0\n",
       "trestbps    110.0\n",
       "chol        239.0\n",
       "fbs           0.0\n",
       "restecg       0.0\n",
       "thalach     142.0\n",
       "exang         1.0\n",
       "oldpeak       1.2\n",
       "slope         1.0\n",
       "ca            1.0\n",
       "thal          3.0\n",
       "target        0.0\n",
       "Name: 206, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heart_disease.loc[206]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a prediction on a single sample (has to be array)\n",
    "model.predict(np.array(X_test.loc[206]).reshape(1, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate the model\n",
    "A trained model/estimator can be evaluated by calling the `score()` function and passing it a collection of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9911894273127754"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# On the training set\n",
    "model.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# On the test set (unseen)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Experiment to improve (hyperparameter tuning)\n",
    "\n",
    "A model's first evaluation metrics aren't always its last. One way to improve a models predictions is with hyperparameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying model with 10 estimators...\n",
      "Model accruacy on test set: 0.7236842105263158\n",
      "\n",
      "Trying model with 20 estimators...\n",
      "Model accruacy on test set: 0.7368421052631579\n",
      "\n",
      "Trying model with 30 estimators...\n",
      "Model accruacy on test set: 0.7368421052631579\n",
      "\n",
      "Trying model with 40 estimators...\n",
      "Model accruacy on test set: 0.7368421052631579\n",
      "\n",
      "Trying model with 50 estimators...\n",
      "Model accruacy on test set: 0.6973684210526315\n",
      "\n",
      "Trying model with 60 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "\n",
      "Trying model with 70 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "\n",
      "Trying model with 80 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "\n",
      "Trying model with 90 estimators...\n",
      "Model accruacy on test set: 0.75\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Try different numbers of estimators (n_estimators is a hyperparameter you can change)\n",
    "np.random.seed(42)\n",
    "for i in range(10, 100, 10):\n",
    "    print(f\"Trying model with {i} estimators...\")\n",
    "    model = RandomForestClassifier(n_estimators=i).fit(X_train, y_train)\n",
    "    print(f\"Model accruacy on test set: {model.score(X_test, y_test)}\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** It's best practice to test different hyperparameters with a validation set or cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying model with 10 estimators...\n",
      "Model accruacy on test set: 0.7236842105263158\n",
      "Cross-validation score: 78.53551912568305%\n",
      "\n",
      "Trying model with 20 estimators...\n",
      "Model accruacy on test set: 0.75\n",
      "Cross-validation score: 79.84699453551912%\n",
      "\n",
      "Trying model with 30 estimators...\n",
      "Model accruacy on test set: 0.7763157894736842\n",
      "Cross-validation score: 80.50819672131148%\n",
      "\n",
      "Trying model with 40 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "Cross-validation score: 82.15300546448088%\n",
      "\n",
      "Trying model with 50 estimators...\n",
      "Model accruacy on test set: 0.7368421052631579\n",
      "Cross-validation score: 81.1639344262295%\n",
      "\n",
      "Trying model with 60 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "Cross-validation score: 83.47540983606557%\n",
      "\n",
      "Trying model with 70 estimators...\n",
      "Model accruacy on test set: 0.7236842105263158\n",
      "Cross-validation score: 81.83060109289617%\n",
      "\n",
      "Trying model with 80 estimators...\n",
      "Model accruacy on test set: 0.7631578947368421\n",
      "Cross-validation score: 82.81420765027322%\n",
      "\n",
      "Trying model with 90 estimators...\n",
      "Model accruacy on test set: 0.75\n",
      "Cross-validation score: 82.81967213114754%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Try different numbers of estimators with cross-validation and no cross-validation\n",
    "np.random.seed(42)\n",
    "for i in range(10, 100, 10):\n",
    "    print(f\"Trying model with {i} estimators...\")\n",
    "    model = RandomForestClassifier(n_estimators=i).fit(X_train, y_train)\n",
    "    print(f\"Model accruacy on test set: {model.score(X_test, y_test)}\")\n",
    "    print(f\"Cross-validation score: {np.mean(cross_val_score(model, X, y, cv=5)) * 100}%\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Save a model for later use\n",
    "\n",
    "A trained model can be exported and saved so it can be imported and used later. One way to save a model is using Python's `pickle` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Save trained model to file\n",
    "pickle.dump(model, open(\"random_forest_model_1.pkl\", \"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load a saved model and make a prediction on a single example\n",
    "loaded_model = pickle.load(open(\"random_forest_model_1.pkl\", \"rb\"))\n",
    "loaded_model.predict(np.array(X_test.loc[206]).reshape(1, -1))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}