Module-5/week-2/random_forests/Random_Forests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To live! like a tree alone and free,\n",
    "> and like a forest in brotherhood/sisterhood...\n",
    "> Nazim Hikmet\n",
    "\n",
    "<center> <h1> Random Forests (Yes! No Forest Image) </h1> </center>\n",
    "\n",
    "\n",
    "\n",
    "## Objectives\n",
    "\n",
    "- Review of decision trees\n",
    "\n",
    "- Defining Bootstrapping process\n",
    "\n",
    "- Explaining the need for bootstrapping\n",
    "\n",
    "- Identifying the need for bootstrapping for decision trees.\n",
    "\n",
    "- Comparing Random forests and bagging methods\n",
    "\n",
    "- Evaulating a model by random forest model\n",
    "\n",
    "<center> <h1> Bootstrapping </h1> </center>\n",
    "\n",
    "<img src= \"img/bootstrapping2.jpg\" style=\"height:400px\">\n",
    "\n",
    "\n",
    "Consider a case that we want to invest to either X or Y a fixed amount of money.\n",
    "\n",
    "- $\\alpha$ amount to X and $1- \\alpha$ amount to Y\n",
    "\n",
    "- We want to minimize variance in our investment! This is minimize $V(\\alpha X + (1-\\alpha )Y)$\n",
    "\n",
    "- The $\\alpha$ that minimizes the variance: \n",
    "\n",
    "$$ \\alpha = \\frac{\\sigma^{2}_{Y} - \\sigma_{XY}}{\\sigma_{X}^{2} + \\sigma^{2}_{Y} - 2\\sigma_{XY}}$$\n",
    "\n",
    "where $\\sigma_{XY}$ is the covariance of X and Y. \n",
    "\n",
    "Q: Do you see the problem here?\n",
    "\n",
    "Hint: Let's rewrite the equation with hats! What are the hats refering to ?\n",
    "\n",
    "\n",
    "$$ \\widehat{\\alpha}= \\frac{\\widehat{\\sigma}^{2}_{Y} - \\widehat{\\sigma}_{XY}}{\\widehat{\\sigma}_{X}^{2} + \\widehat{\\sigma}^{2}_{Y} - 2\\widehat{\\sigma}_{XY}}$$\n",
    "\n",
    "Q: How much do we certain about $\\widehat{\\alpha}$?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = np.random.multivariate_normal(mean = [10,10], cov = [[1, 0.5], [0.5, 1.25]] , size = 100)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.04003137 0.44588873]\n",
      " [0.44588873 1.07625358]]\n",
      "0.5149001319482571\n"
     ]
    }
   ],
   "source": [
    "sigma_x = sample[:,0].var()\n",
    "\n",
    "sigma_y = sample[:, 1].var()\n",
    "\n",
    "sigma_xy = np.cov(sample.T)[0,1]\n",
    "\n",
    "print(np.cov(sample.T))\n",
    "\n",
    "alpha_hat = (sigma_y - sigma_xy)/(sigma_x + sigma_y - 2* sigma_xy)\n",
    "\n",
    "print(alpha_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "alphas = []\n",
    "\n",
    "for i in range(1000):\n",
    "    s = np.random.multivariate_normal(mean = [10,10], cov = [[1, 0.5], [0.5, 1.25]] , size = 100)\n",
    "    s_x = s[:,0].var()\n",
    "    s_y = s[:,1].var()\n",
    "    s_xy = np.cov(s.T)[0,1]\n",
    "    a_hat = (s_y - s_xy)/(s_x + s_y - 2* s_xy)\n",
    "    alphas.append(a_hat)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the mean of the alphas\n",
    "# find the standard error of alphas - note that the degree of freedom is 1 in this case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q: what is wrong with this approach?\n",
    "\n",
    "A: What can we do about this -- Bootstapping \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 1, 2, 1, 1, 1, 2, 2, 2, 2, 1, 2, 2, 2, 2, 1, 2, 2, 1, 1, 2,\n",
       "       1, 1, 1, 1, 1, 2, 1, 1, 2, 2, 6, 2, 3, 1, 3, 1, 1, 1, 2, 1, 1, 1,\n",
       "       1, 1, 1, 2, 1, 1, 2, 1, 4, 1, 1, 1, 1, 2, 2, 2, 3, 2])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Bootstrapping\n",
    "\n",
    "boot_sample = sample[np.random.choice(np.arange(100), replace = True, size = 100)]\n",
    "\n",
    "np.unique(boot_sample, return_counts= True, axis = 0)[1]\n",
    "\n",
    "## Homework: find alpha_hats with bootstrapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src= \"img/bootstrap1.png\" style=\"height:400px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bagging\n",
    "\n",
    "\n",
    "Let's us one more time recall that if $Z_{1}, \\cdots, Z_{n}$ are independent observations with variance $\\sigma^{2}$ then the variance of the mean $\\bar{Z}$ is given by $\\frac{\\sigma^{2}}{n}$. \n",
    "\n",
    "So what?\n",
    "\n",
    "We will use this idea calculate $$ \\hat{f}^{1}(x), \\cdots, \\hat{f}^{B}(x)$$ where each $\\hat{f}^{i}$ represents a decision tree fitted to the bootstrapped data.\n",
    "\n",
    "\n",
    "Then we will make a prediction by: \n",
    "\n",
    "$$ \\hat{f}_{\\text{avg}}(x) = \\frac{1}{B}\\sum_{b=1}^{B} \\hat{f}^{b}(x)$$\n",
    "\n",
    "Note that this is for regression and for the classification we can get majority vote.\n",
    "\n",
    "__Problem__ We still have some problem with this approach and random forests will address this problem. Can you see the issue?\n",
    "\n",
    "- If we have a strong predictor then this will dominate in each tree.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "## you can download the data from -- https://www.kaggle.com/ishaanv/ISLR-Auto#Heart.csv\n",
    "\n",
    "## or http://faculty.marshall.usc.edu/gareth-james/ISL/data.html\n",
    "heart = pd.read_csv('data/Heart.csv', index_col = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 303 entries, 1 to 303\n",
      "Data columns (total 14 columns):\n",
      "Age          303 non-null int64\n",
      "Sex          303 non-null int64\n",
      "ChestPain    303 non-null object\n",
      "RestBP       303 non-null int64\n",
      "Chol         303 non-null int64\n",
      "Fbs          303 non-null int64\n",
      "RestECG      303 non-null int64\n",
      "MaxHR        303 non-null int64\n",
      "ExAng        303 non-null int64\n",
      "Oldpeak      303 non-null float64\n",
      "Slope        303 non-null int64\n",
      "Ca           299 non-null float64\n",
      "Thal         301 non-null object\n",
      "AHD          303 non-null object\n",
      "dtypes: float64(2), int64(9), object(3)\n",
      "memory usage: 35.5+ KB\n"
     ]
    }
   ],
   "source": [
    "heart.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "heart.dropna(axis= 0, how= 'any', inplace = True)\n",
    "\n",
    "y = heart.AHD\n",
    "\n",
    "heart.drop(columns= 'AHD', inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = pd.get_dummies(heart)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc = RandomForestClassifier(n_estimators=300, max_features= 'sqrt', max_depth= 7,  oob_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = cross_validate(rfc, X, y,return_estimator= True, cv = 5 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.83333333, 0.91666667, 0.76271186, 0.79661017, 0.81355932])"
      ]
     },
     "execution_count": 238,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv['test_score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [],
   "source": [
    "est = cv['estimator'][3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8235294117647058"
      ]
     },
     "execution_count": 253,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est.oob_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "est.feature_importances_\n",
    "\n",
    "index = X.columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_imp = pd.DataFrame(est.feature_importances_, index = index, columns = ['importances'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>importances</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Thal_normal</th>\n",
       "      <td>0.119966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ca</th>\n",
       "      <td>0.113553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Oldpeak</th>\n",
       "      <td>0.112316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MaxHR</th>\n",
       "      <td>0.100357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ChestPain_asymptomatic</th>\n",
       "      <td>0.089637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thal_reversable</th>\n",
       "      <td>0.085006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>0.078399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RestBP</th>\n",
       "      <td>0.071167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chol</th>\n",
       "      <td>0.062618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExAng</th>\n",
       "      <td>0.038043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Slope</th>\n",
       "      <td>0.033557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <td>0.023677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RestECG</th>\n",
       "      <td>0.019079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ChestPain_nonanginal</th>\n",
       "      <td>0.017189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ChestPain_nontypical</th>\n",
       "      <td>0.012284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ChestPain_typical</th>\n",
       "      <td>0.011822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fbs</th>\n",
       "      <td>0.005730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thal_fixed</th>\n",
       "      <td>0.005602</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        importances\n",
       "Thal_normal                0.119966\n",
       "Ca                         0.113553\n",
       "Oldpeak                    0.112316\n",
       "MaxHR                      0.100357\n",
       "ChestPain_asymptomatic     0.089637\n",
       "Thal_reversable            0.085006\n",
       "Age                        0.078399\n",
       "RestBP                     0.071167\n",
       "Chol                       0.062618\n",
       "ExAng                      0.038043\n",
       "Slope                      0.033557\n",
       "Sex                        0.023677\n",
       "RestECG                    0.019079\n",
       "ChestPain_nonanginal       0.017189\n",
       "ChestPain_nontypical       0.012284\n",
       "ChestPain_typical          0.011822\n",
       "Fbs                        0.005730\n",
       "Thal_fixed                 0.005602"
      ]
     },
     "execution_count": 267,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_imp.sort_values(by = 'importances', ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extra Material \n",
    "\n",
    "- [On the variance](https://newonlinecourses.science.psu.edu/stat414/node/167/)\n",
    "\n",
    "- [Discussion on Random Forests and overfitting](https://en.wikipedia.org/wiki/Talk%3ARandom_forest)\n",
    "\n",
    "- [Tricky stuff with respect to feature importance](http://rnowling.github.io/machine/learning/2015/08/10/random-forest-bias.html)\n",
    "\n",
    "- [ISLR - section 8.2](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}