02_BasicMLAlgorithms/decision-trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trees, Forests, and Ensembles\n",
    "\n",
    "![japanese maple](assets/trees/japanese-maple-2947680_640.jpg)\n",
    "\n",
    "(image: [pixabay](https://pixabay.com/en/japanese-maple-foliage-green-leaves-2947680/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Topics\n",
    "\n",
    "- Decision Trees\n",
    "- Ensemble Forests\n",
    "- Classification\n",
    "- Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Where are we?\n",
    "\n",
    "(lost in the forest?)\n",
    "\n",
    "![spot the trees](assets/linear-regression/machine-learning-cheet-sheet.png)\n",
    "\n",
    "(image: [sas.com](https://www.sas.com/en_us/insights/analytics/machine-learning.html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decision Trees\n",
    "- Classification: predicts a class\n",
    "- Regression: predicts a number\n",
    "\n",
    "http://scikit-learn.org/stable/modules/tree.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decision Trees\n",
    "\n",
    "Advantanges:\n",
    "- Easy to visualize and understand\n",
    "- $O(log(N))$ prediction cost\n",
    "- Flexible for simple tasks: binary / multi-class classification, regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decision Trees\n",
    "\n",
    "Disadvantages:\n",
    "- Overfitting\n",
    "- Unbalanced dataset can cause biased trees\n",
    "- Instability: small changes in data can result in completely different tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ensemble Methods\n",
    "\n",
    "![ensemble](assets/trees/teamwork-2499638_960_720.jpg)\n",
    "\n",
    "(image: [pixabay](https://pixabay.com/en/teamwork-team-gear-board-chalk-2499638/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ensemble Methods\n",
    "\"Teaming up weaker models to create a stronger model\"\n",
    "\n",
    "- Random Forest\n",
    "- AdaBoost\n",
    "- Gradient Boosted Trees\n",
    "- etc\n",
    "\n",
    "http://scikit-learn.org/stable/modules/ensemble.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Random Forests\n",
    "\n",
    "- Fits a few decision trees on subsets of the dataset, then averages the results\n",
    "  - Improves accuracy and reduces overfitting\n",
    "  - https://amueller.github.io/COMS4995-s18/slides/aml-10-021918-trees-forests/#36\n",
    "- Classifier: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n",
    "- Regressor: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### AdaBoost\n",
    "\n",
    "- http://scikit-learn.org/stable/modules/ensemble.html#adaboost\n",
    "- Fit a sequence of small decision trees on repeatedly modified versions of data\n",
    "  \n",
    "  - adjust weights on the training samples\n",
    "    - incorrectly predicted training sample => more weight\n",
    "    - correctly predicted sample => less weight\n",
    "    \n",
    "    \n",
    "- Combine predictions using sum or majority vote\n",
    "\n",
    "- \"Samples that were gotten wrong get more attention\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### AdaBoost\n",
    "\n",
    "- Classifier: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html\n",
    "- Regressor: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosted Trees\n",
    "\n",
    "- Optimized using Gradient Descent\n",
    "- Better generalization\n",
    "- Classifier: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html\n",
    "- Regressor: http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosting\n",
    "\n",
    "![really?](assets/trees/blog_Gradient-Boosting-Image.png)\n",
    "\n",
    "(image: http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosting: Concept\n",
    "\n",
    "1. Nth tree has some prediction error.\n",
    "\n",
    "2. Train the next (N+1)th tree to \"boost\" the model so that it can compensate for the gap (error) from the Nth tree. \n",
    "\n",
    "3. Add up the predictions from the all trees to fill up the gaps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosting: Concept\n",
    "\n",
    "1. Train first tree, measure training error (\"residuals\")\n",
    "$$F_1(x) = y$$\n",
    "2. Train second tree on the residuals, using dataset $\\big(x, y-F_1(x)\\big)$\n",
    "$$h_1(x) = y - F_1(x)$$\n",
    "3. Combine to get a better model\n",
    "$$F_2(x) = F_1(x) + h_1(x)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Won't run\n",
    "f1 = DecisionTreeClassifier()\n",
    "f1.fit(X, y)\n",
    "f1_pred = f1.predict(X)\n",
    "\n",
    "# Train second tree on residuals\n",
    "h1 = DecisionTreeClassifier()\n",
    "h1.fit(X, (y - f1_pred)) # with cross validation or forest\n",
    "h1_pred = h1.predict(X)\n",
    "\n",
    "# combine to get a better model\n",
    "f2_pred = f1_pred + h1_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosting: Concept\n",
    "\n",
    "General formulation:\n",
    "$$F_{m+1}(x) = F_m(x) + h_m(x) = y$$\n",
    "\n",
    "Where $m$ is tuned by cross validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosting: Gradient Descent\n",
    "\n",
    "- Objective: minimize cost function: $L\\big(y, F(x)\\big)$\n",
    "- Apply Tree Boosting, but compute residuals from the **gradients of the cost function**\n",
    "- $n$ training examples $(x_1, ... x_n)$\n",
    "- Residual for the $i^{th}$ example, $m^{th}$ tree:\n",
    "\n",
    "$$r_{im} = -\\biggl[\\frac{\\partial{L\\big(y_i, F_{m-1}(x_i)\\big)}}{\\partial{F_{m-1}(x_i)}}\\biggr]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Gradient Boosting: Algorithm\n",
    "\n",
    "1. Compute residual using $F_{m-1}(x_i)$ and $y_i$\n",
    "2. Train decision tree $h_m(x)$, using dataset ${(x_i, r_{im})}$\n",
    "3. Compute update multiplier:\n",
    "$$\\gamma_m = \\underset{\\gamma}{\\arg \\min} \\sum^n_{i=1} L\\big(y, F_{m-1} + \\gamma h_{m}(x)\\big)$$\n",
    "4. Get next model using multiplier:\n",
    "$$F_m(x) = F_{m-1}(x) + \\gamma_mh_m(x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### XGBoost\n",
    "\n",
    "eXtreme Gradient Boosting\n",
    "- A parallel, distributed gradient boosting library\n",
    "- https://github.com/dmlc/xgboost\n",
    "- https://xgboost.readthedocs.io/en/latest/model.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluation Metrics\n",
    "\n",
    "- General metrics for Classification and Regression\n",
    "- Decision Tree-specific metrics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Decision Tree-specific Metrics\n",
    "- Gini: gini impurity, which measures the quality of a split\n",
    "  - greater than 0: split needed\n",
    "  - 0: all cases fall in 1 category\n",
    "- Information gain / entropy\n",
    "  - pick the split with the highest information gain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Workshop: Classification with Decision Trees\n",
    "\n",
    "Credits: http://scikit-learn.org/stable/modules/tree.html#classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Setup\n",
    "\n",
    "We'll be using Graphviz to visualize the decision tree after training it.\n",
    "\n",
    "Add this module to your `mldds02` conda environment:\n",
    "\n",
    "```\n",
    "conda install python-graphviz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Goal\n",
    "\n",
    "Train and compare decision tree-based classifiers to predict the `sector` from research and development expenditure type, cost type, and expenditure amount."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dataset\n",
    "\n",
    "#### Research and Development Expenditure by Type of Cost\n",
    "\n",
    "https://data.gov.sg/dataset/research-and-development-expenditure-by-type-of-cost\n",
    "\n",
    "1. Download dataset from the above URL\n",
    "2. Extract the folder and note the path for use in `read_csv` below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Prepare Dataset\n",
    "\n",
    "1. Encode string labels to numbers\n",
    "2. Ensure dataset is balanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sector</th>\n",
       "      <th>type_of_expenditure</th>\n",
       "      <th>type_of_cost</th>\n",
       "      <th>rnd_expenditure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Land, Buildings &amp; Other Structures</td>\n",
       "      <td>231.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Vehicles, Plant, Machinery &amp; Equipment</td>\n",
       "      <td>670.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Researchers</td>\n",
       "      <td>1914.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Technicians</td>\n",
       "      <td>75.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Other Supporting Staff</td>\n",
       "      <td>134.54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           sector   type_of_expenditure  \\\n",
       "0  Private Sector   Capital Expenditure   \n",
       "1  Private Sector   Capital Expenditure   \n",
       "2  Private Sector  Manpower Expenditure   \n",
       "3  Private Sector  Manpower Expenditure   \n",
       "4  Private Sector  Manpower Expenditure   \n",
       "\n",
       "                             type_of_cost  rnd_expenditure  \n",
       "0      Land, Buildings & Other Structures           231.79  \n",
       "1  Vehicles, Plant, Machinery & Equipment           670.82  \n",
       "2                             Researchers          1914.63  \n",
       "3                             Technicians            75.70  \n",
       "4                  Other Supporting Staff           134.54  "
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('../data/research-and-development-expenditure-by-type-of-cost/research-and-development-expenditure-by-type-of-cost.csv',\n",
    "\n",
    "                 usecols=['sector', 'type_of_expenditure', 'type_of_cost', 'rnd_expenditure'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Label Encoding for Classification\n",
    "\n",
    "Training data needs to be numeric.\n",
    "\n",
    "To convert string labels to numbers, we will do something called \"label encoding\".\n",
    "\n",
    "There are multiple ways to do this: http://pbpython.com/categorical-encoding.html\n",
    "- Dummy columns\n",
    "- Integer labels\n",
    "\n",
    "For this dataset, we'll try assigning integer labels to each unique string value in the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sector</th>\n",
       "      <th>type_of_expenditure</th>\n",
       "      <th>type_of_cost</th>\n",
       "      <th>rnd_expenditure</th>\n",
       "      <th>sector_c</th>\n",
       "      <th>type_of_expenditure_c</th>\n",
       "      <th>type_of_cost_c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Land, Buildings &amp; Other Structures</td>\n",
       "      <td>231.79</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Vehicles, Plant, Machinery &amp; Equipment</td>\n",
       "      <td>670.82</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Researchers</td>\n",
       "      <td>1914.63</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Technicians</td>\n",
       "      <td>75.70</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Other Supporting Staff</td>\n",
       "      <td>134.54</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Private Sector</td>\n",
       "      <td>Other Operating Expenditure</td>\n",
       "      <td>Other Operating Costs</td>\n",
       "      <td>2188.18</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Government Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Land, Buildings &amp; Other Structures</td>\n",
       "      <td>27.17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Government Sector</td>\n",
       "      <td>Capital Expenditure</td>\n",
       "      <td>Vehicles, Plant, Machinery &amp; Equipment</td>\n",
       "      <td>30.05</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Government Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Researchers</td>\n",
       "      <td>304.45</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Government Sector</td>\n",
       "      <td>Manpower Expenditure</td>\n",
       "      <td>Technicians</td>\n",
       "      <td>33.21</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              sector          type_of_expenditure  \\\n",
       "0     Private Sector          Capital Expenditure   \n",
       "1     Private Sector          Capital Expenditure   \n",
       "2     Private Sector         Manpower Expenditure   \n",
       "3     Private Sector         Manpower Expenditure   \n",
       "4     Private Sector         Manpower Expenditure   \n",
       "5     Private Sector  Other Operating Expenditure   \n",
       "6  Government Sector          Capital Expenditure   \n",
       "7  Government Sector          Capital Expenditure   \n",
       "8  Government Sector         Manpower Expenditure   \n",
       "9  Government Sector         Manpower Expenditure   \n",
       "\n",
       "                             type_of_cost  rnd_expenditure  sector_c  \\\n",
       "0      Land, Buildings & Other Structures           231.79         2   \n",
       "1  Vehicles, Plant, Machinery & Equipment           670.82         2   \n",
       "2                             Researchers          1914.63         2   \n",
       "3                             Technicians            75.70         2   \n",
       "4                  Other Supporting Staff           134.54         2   \n",
       "5                   Other Operating Costs          2188.18         2   \n",
       "6      Land, Buildings & Other Structures            27.17         0   \n",
       "7  Vehicles, Plant, Machinery & Equipment            30.05         0   \n",
       "8                             Researchers           304.45         0   \n",
       "9                             Technicians            33.21         0   \n",
       "\n",
       "   type_of_expenditure_c  type_of_cost_c  \n",
       "0                      0               0  \n",
       "1                      0               5  \n",
       "2                      1               3  \n",
       "3                      1               4  \n",
       "4                      1               2  \n",
       "5                      2               1  \n",
       "6                      0               0  \n",
       "7                      0               5  \n",
       "8                      1               3  \n",
       "9                      1               4  "
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "le1 = LabelEncoder()\n",
    "\n",
    "# encode sector from strings to integer labels\n",
    "df['sector_c'] = le1.fit_transform(df['sector'])\n",
    "\n",
    "le2 = LabelEncoder()\n",
    "df['type_of_expenditure_c'] = le2.fit_transform(df['type_of_expenditure'])\n",
    "\n",
    "le3 = LabelEncoder()\n",
    "df['type_of_cost_c'] = le3.fit_transform(df['type_of_cost'])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Ensuring Balanced Dataset for Decision Tree Algorithms\n",
    "\n",
    "For decision trees, it is important to balance the dataset to reduce class bias.\n",
    "\n",
    "If a dataset contains too many samples of one sector (e.g. 'Private Sector', the tree will learn to pick that sector more frequently, which means it's no better than random selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Private Sector', 'Government Sector', 'Higher Education Sector',\n",
       "       'Public Research Institutes'], dtype=object)"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sector.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Government Sector', 'Higher Education Sector', 'Private Sector',\n",
       "       'Public Research Institutes'], dtype=object)"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "le1.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    30\n",
       "2    30\n",
       "1    30\n",
       "0    30\n",
       "Name: sector_c, dtype: int64"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Detect if a dataset is unbalanced\n",
    "df['sector_c'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We got lucky here with the dataset, as there are equal numbers of value_counts for each sector.\n",
    "\n",
    "Suppose we need to balance the dataset, we can use this technique:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    60\n",
       "3    30\n",
       "2    30\n",
       "1    30\n",
       "Name: sector_c, dtype: int64"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# simulate an unbalanced dataset by replicating columns for sector_c=0\n",
    "\n",
    "df.loc[df.sector_c == 0]\n",
    "\n",
    "# make a copy so as not to affect our original df\n",
    "unbalanced_df = pd.concat([df, df.loc[df.sector_c == 0]], ignore_index=True)\n",
    "\n",
    "# show the unbalanced dataset, sector_c = 0 will have double the number of entries\n",
    "unbalanced_df['sector_c'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    30\n",
       "2    30\n",
       "1    30\n",
       "0    30\n",
       "Name: sector_c, dtype: int64"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sector_groups = unbalanced_df.groupby('sector_c')\n",
    "\n",
    "# use pandas.DataFrame.sample to create a DataFrame\n",
    "# where all sector groups are re-sampled to the smallest sized group\n",
    "balanced_df = sector_groups.apply(lambda x: x.sample(sector_groups.size().min())).\\\n",
    "    reset_index(drop=True)\n",
    "\n",
    "# show the balanced_df, all sectors are balanced\n",
    "balanced_df['sector_c'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Select Features\n",
    "\n",
    "We'll now break our DataFrame into data and target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "data = df[['rnd_expenditure', 'type_of_expenditure_c', 'type_of_cost_c']]\n",
    "target = df['sector_c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rnd_expenditure</th>\n",
       "      <th>type_of_expenditure_c</th>\n",
       "      <th>type_of_cost_c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>231.79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>670.82</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1914.63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>75.70</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>134.54</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   rnd_expenditure  type_of_expenditure_c  type_of_cost_c\n",
       "0           231.79                      0               0\n",
       "1           670.82                      0               5\n",
       "2          1914.63                      1               3\n",
       "3            75.70                      1               4\n",
       "4           134.54                      1               2"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    2\n",
       "2    2\n",
       "3    2\n",
       "4    2\n",
       "Name: sector_c, dtype: int64"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Train the Decision Tree Classifier\n",
    "\n",
    "1. Shuffle and split the data set into train and test\n",
    "2. Train a `DecisionTreeClassifier`\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_X, test_X, train_y, test_y = train_test_split(data, target,\n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=5,\n",
       "            max_features=None, max_leaf_nodes=10,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=42,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# scale\n",
    "scaler = StandardScaler()\n",
    "train_X_scaled = scaler.fit_transform(train_X)\n",
    "test_X_scaled = scaler.transform(test_X)\n",
    "\n",
    "dtc = DecisionTreeClassifier(random_state=42, max_leaf_nodes=10, max_depth=5)\n",
    "dtc.fit(train_X_scaled, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 3, 1, 1, 2, 0, 3, 0, 0, 3, 1, 1, 1, 1, 3, 3, 1, 3, 0, 1, 3, 0,\n",
       "       1, 1, 1, 1, 1, 1, 1, 2], dtype=int64)"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtc.predict(test_X_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Public Research Institutes' 'Public Research Institutes'\n",
      " 'Higher Education Sector' 'Higher Education Sector' 'Private Sector'\n",
      " 'Government Sector' 'Public Research Institutes' 'Government Sector'\n",
      " 'Government Sector' 'Public Research Institutes'\n",
      " 'Higher Education Sector' 'Higher Education Sector'\n",
      " 'Higher Education Sector' 'Higher Education Sector'\n",
      " 'Public Research Institutes' 'Public Research Institutes'\n",
      " 'Higher Education Sector' 'Public Research Institutes'\n",
      " 'Government Sector' 'Higher Education Sector'\n",
      " 'Public Research Institutes' 'Government Sector'\n",
      " 'Higher Education Sector' 'Higher Education Sector'\n",
      " 'Higher Education Sector' 'Higher Education Sector'\n",
      " 'Higher Education Sector' 'Higher Education Sector'\n",
      " 'Higher Education Sector' 'Private Sector']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    }
   ],
   "source": [
    "pred = dtc.predict(test_X_scaled)\n",
    "    \n",
    "print(le1.inverse_transform(pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Evaluate Metrics: Classification Accuracy\n",
    "\n",
    "Since this is a classification task, the metrics we used for Logistic Regression also apply here, such as\n",
    "- Precision, recall\n",
    "- Confusion matrix\n",
    "- Accuracy\n",
    "\n",
    "References:\n",
    "- http://scikit-learn.org/stable/modules/classes.html#classification-metrics\n",
    "- http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.20      0.17      0.18         6\n",
      "          1       0.33      0.50      0.40        10\n",
      "          2       1.00      0.33      0.50         6\n",
      "          3       0.38      0.38      0.38         8\n",
      "\n",
      "avg / total       0.45      0.37      0.37        30\n",
      "\n",
      "[[1 4 0 1]\n",
      " [3 5 0 2]\n",
      " [0 2 2 2]\n",
      " [1 4 0 3]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "pred_y = dtc.predict(test_X_scaled)\n",
    "\n",
    "print(classification_report(test_y, pred_y))\n",
    "\n",
    "cm = confusion_matrix(test_y, pred_y)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Visualizing the Decision Tree\n",
    "\n",
    "We'll visualize the learned decision tree using graphviz, to see what type of rules it has learnt from the training data.\n",
    "\n",
    "https://graphviz.readthedocs.io/en/stable/api.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "import graphviz\n",
    "\n",
    "def visualize_tree(fitted_tree, feature_names, target_names, filename):\n",
    "    \"\"\"\n",
    "    Args:\n",
    "        fitted_tree: the fitted decision tree. If using ensemble methods\n",
    "            pick the first estimator using model.estimators[0]\n",
    "        feature_names: array containing the feature names\n",
    "        target_names: array containing the target labels\n",
    "        filename: the filename to save the .dot file\n",
    "    \"\"\"\n",
    "    export_graphviz(fitted_tree, out_file=filename,\n",
    "                    feature_names=feature_names,\n",
    "                    class_names=target_names,\n",
    "                    filled=True, rounded=True)\n",
    "\n",
    "    source = graphviz.Source.from_file(filename)\n",
    "    source.render(view=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "ename": "CalledProcessError",
     "evalue": "Command '['dot.bat', '-Tpdf', '-O', '../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_tree_maxleaf10.dot']' returned non-zero exit status 1.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mCalledProcessError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-188-6d8c156d2957>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[0mfilename\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m'../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_tree_maxleaf10.dot'\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m \u001b[0mvisualize_tree\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdtc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfeature_names\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtarget_names\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfilename\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-187-9428cf8a0105>\u001b[0m in \u001b[0;36mvisualize_tree\u001b[1;34m(fitted_tree, feature_names, target_names, filename)\u001b[0m\n\u001b[0;32m     17\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m     \u001b[0msource\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgraphviz\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSource\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfrom_file\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 19\u001b[1;33m     \u001b[0msource\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mview\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\graphviz\\files.py\u001b[0m in \u001b[0;36mrender\u001b[1;34m(self, filename, directory, view, cleanup)\u001b[0m\n\u001b[0;32m    174\u001b[0m         \u001b[0mfilepath\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msave\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdirectory\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    175\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 176\u001b[1;33m         \u001b[0mrendered\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mbackend\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_format\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfilepath\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    177\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    178\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mcleanup\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\graphviz\\backend.py\u001b[0m in \u001b[0;36mrender\u001b[1;34m(engine, format, filepath, quiet)\u001b[0m\n\u001b[0;32m    124\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mos\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdevnull\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'w'\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mstderr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    125\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 126\u001b[1;33m             \u001b[0msubprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcheck_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstartupinfo\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mSTARTUPINFO\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mstderr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    127\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    128\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0merrno\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0merrno\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mENOENT\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\subprocess.py\u001b[0m in \u001b[0;36mcheck_call\u001b[1;34m(*popenargs, **kwargs)\u001b[0m\n\u001b[0;32m    289\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mcmd\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    290\u001b[0m             \u001b[0mcmd\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpopenargs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 291\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mretcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcmd\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    292\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    293\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mCalledProcessError\u001b[0m: Command '['dot.bat', '-Tpdf', '-O', '../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_tree_maxleaf10.dot']' returned non-zero exit status 1."
     ]
    }
   ],
   "source": [
    "feature_names=['rnd_expenditure', 'type_of_expenditure', 'type_of_cost']\n",
    "\n",
    "# target_names=df['sector'].unique() # wrong\n",
    "target_names = le1.classes_ # label encoder's order\n",
    "\n",
    "filename = '../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_tree_maxleaf10.dot'\n",
    "\n",
    "visualize_tree(dtc, feature_names, target_names, filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importance\n",
    "\n",
    "https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "# Gini importance of each feature\n",
    "#\n",
    "# number of times a feature is used to split a node,\n",
    "# weighted by the number of samples it splits.\n",
    "\n",
    "print(dtc.feature_importances_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_features = len(data.columns)\n",
    "\n",
    "plt.barh(range(n_features), dtc.feature_importances_)\n",
    "plt.yticks(range(n_features), data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Exercise - Decision Tree Classification using Entropy\n",
    "\n",
    "Repeat the steps above to:\n",
    "1. Train a decision tree using the 'entropy' criteria using the training set\n",
    "2. Evaluate the classification metrics\n",
    "3. Visualize the decision tree\n",
    "\n",
    "Which criteria performs better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Visualizing Decision Tree Surfaces\n",
    "\n",
    "Here's a neat trick to try in lieu of what we saw with clustering.\n",
    "\n",
    "Credits: http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adapted from: http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_decision_surface(classifier, n_classes, X, y, title):\n",
    "    \"\"\"Plots a decision surface using pair-wise combination\n",
    "    of features\n",
    "    Args:\n",
    "        classifier - the decision tree classifier\n",
    "        n_classes - the number of classes\n",
    "        X - the data\n",
    "        y - the labels\n",
    "        title - the plot title\n",
    "    \"\"\"\n",
    "    plot_colors = 'ryb'\n",
    "    plot_step = 0.02\n",
    "\n",
    "    plt.figure(figsize=(15, 10))\n",
    "\n",
    "    for pairidx, pair in enumerate([[0, 1], [0, 2], [1, 2]]):\n",
    "        # We only take the two corresponding features\n",
    "        x = X.values[:, pair]\n",
    "\n",
    "        clf = classifier.fit(x, y)\n",
    "\n",
    "        # Plot the decision boundary\n",
    "        plt.subplot(2, 3, pairidx + 1)\n",
    "\n",
    "        x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1\n",
    "        y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1\n",
    "        xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),\n",
    "                             np.arange(y_min, y_max, plot_step))\n",
    "        plt.tight_layout(h_pad=0.5, w_pad=0.5, pad=2.5)\n",
    "\n",
    "        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "        Z = Z.reshape(xx.shape)\n",
    "        cs = plt.contourf(xx, yy, Z, cmap=plt.cm.RdYlBu)\n",
    "\n",
    "        plt.xlabel(feature_names[pair[0]])\n",
    "        plt.ylabel(feature_names[pair[1]])\n",
    "\n",
    "        # Plot the training points\n",
    "        for i, color in zip(range(n_classes), plot_colors):\n",
    "            idx = np.where(y == i)\n",
    "            plt.scatter(x[idx, 0], x[idx, 1], c=color, label=target_names[i],\n",
    "                        cmap=plt.cm.RdYlBu, edgecolor='black', s=15)\n",
    "\n",
    "    plt.suptitle(title)\n",
    "    plt.legend(loc='lower right', borderpad=0, handletextpad=0)\n",
    "    plt.axis(\"tight\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = len(target.unique())\n",
    "plot_decision_surface(DecisionTreeClassifier(), n_classes, train_X, train_y,\n",
    "                      \"Decision surface of a Decision Tree classifier using paired features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# To get the category label mapping\n",
    "df[['type_of_expenditure', 'type_of_expenditure_c']].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df[['type_of_cost', 'type_of_cost_c']].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df[['sector', 'sector_c']].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Random Forest Classifier with GridSearchCV\n",
    "\n",
    "Now that we have our baseline tree, the next step is to try an ensemble method, such as Random Forest.\n",
    "\n",
    "Let's also maximize our chances of getting the best model by doing Grid Search cross-validation.\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "RandomForestClassifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 36 candidates, totalling 108 fits\n",
      "Best score: 0.7333333333333333\n",
      "Best parameters: {'max_depth': 7, 'min_samples_split': 2, 'n_estimators': 10}\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.57      0.67      0.62         6\n",
      "          1       0.50      0.40      0.44        10\n",
      "          2       1.00      1.00      1.00         6\n",
      "          3       0.44      0.50      0.47         8\n",
      "\n",
      "avg / total       0.60      0.60      0.60        30\n",
      "\n",
      "[[4 1 0 1]\n",
      " [2 4 0 4]\n",
      " [0 0 6 0]\n",
      " [1 3 0 4]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 108 out of 108 | elapsed:    1.1s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Use GridSearchCV to select the optimum hyperparameters\n",
    "gs_rfc = GridSearchCV(RandomForestClassifier(random_state=42),\n",
    "                      {'max_depth': [5, 6, 7, 8],\n",
    "                       'n_estimators': [7, 9, 10],\n",
    "                       'min_samples_split': [2, 4, 8]},\n",
    "                      verbose=1)\n",
    "\n",
    "gs_rfc.fit(train_X, train_y)\n",
    "\n",
    "# select the best estimator\n",
    "print('Best score:', gs_rfc.best_score_)\n",
    "print('Best parameters:', gs_rfc.best_params_)\n",
    "\n",
    "# predict\n",
    "pred_y = gs_rfc.predict(test_X)\n",
    "\n",
    "# evaluation metrics\n",
    "print(classification_report(test_y, pred_y))\n",
    "cm = confusion_matrix(test_y, pred_y)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=7, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=42, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_rfc.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=1608637542, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=1273642419, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=1935803228, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=787846414, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=996406378, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=1201263687, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=423734972, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=415968276, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=670094950, splitter='best'),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "             max_features='auto', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, presort=False,\n",
       "             random_state=1914837113, splitter='best')]"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_rfc.best_estimator_.estimators_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Visualize the first tree in the forest\n",
    "rfc = gs_rfc.best_estimator_\n",
    "\n",
    "visualize_tree(rfc.estimators_[0], feature_names, target_names,\n",
    "               '../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_rf_first.dot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Visualize the last tree in the forest\n",
    "visualize_tree(rfc.estimators_[-1], feature_names, target_names,\n",
    "               '../data/research-and-development-expenditure-by-type-of-cost/govt_sector_by_expenditure_rf_last.dot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABA8AAAF3CAYAAADKCWLQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XmYXFWZ+PHvG5JAgIQtIRq2iCyOLBIMRGRgEEQBF0QngARZBQd1hCEwiPNTwI1xJAiMI4rDqkEWEQEFBoSJRkajCYkhyBYwCoIhC5AAIaHJ+/vj3oZKp7u6Ol3V1Z3+fp6nnq67nfPWrVun67517rmRmUiSJEmSJHVkQLMDkCRJkiRJvZvJA0mSJEmSVJXJA0mSJEmSVJXJA0mSJEmSVJXJA0mSJEmSVJXJA0mSJEmSVJXJA0mqk4i4IyKOrWG9FyNi256IaU1FxI4RMTMilkbE55odD0BEHBYRT5b7b0yz4+mKiMiI2K7ZcfR2ETEvIt7boLL3iYhHKqZXOcYj4rsR8cVG1N0VvbF9iIgJEXFXHcs7LiJ+XWV5n/2sS9LabGCzA5CknhQR84CRQAvwGvBH4Brgssxc2Z2yM/PgGtfbsDv19JB/BaZkZm/64n4B8NnMvKW7BUXEVcBRwIryMQP458x8uLtlN1NETAHeRXF8tzowM3/TgzEksH1mzu2pOmuRmVOBHStm9cZjvFe2D5k5GZjcg1XW87PeK49HSeqL7HkgqT/6UGYOBbYB/h04C7i8uSH1DhHRmlTeBniwmbG0o94x/Ud5orYF8FfWnmPgs5m5YcWjS4mDKPSH7wd1OZ4qPjP9UoNef69pfyJinWbHIEm9RX/4ciBJ7crMFzLzVuAI4NiI2BkgItaNiAsi4i8RMb/szjykdbuIODQiZkXEkoh4PCIOKudPiYhPls+3i4hfRsQLEbEwIq6v2P71LuwRsVFEXBMRCyLizxHx/1pP3Fq79paxPBcRf4qIDns3RMRZEfHXshv2IxFxQDn/qoj4asV6+0XEUxXT88ptZwMvRcS9wHuAb5fdhneIiA+UXbyXlN2Jz21T999HxP9FxPPl8uNq2ZdtyhhQvv4/R8Sz5X7ZqCzjRWAd4A8R8XgH219c1r0kImZExD4d7atKmbkMuAHYraKst0bEvRGxqHz/JkfExm322RkRMbt8j6+PiPUqlp8ZEc9ExNMRcUKbODt7z++LiG+V+/KJiHh3Of/Jcr90emlMB/vn3RHx+zLe30fEuyuWTYmIr0XEfcDLwLZlnJeXr+OvEfHV1hOpjo7viPhVWeQfymPniA5iOSkiHiqP1T9GxO7trLNnRPym3A/PRMS3I2JwuSzKffRsGcPseOPze0hZ5tIy7jPK+a8f9x0c420/Jx+M4nP+fHls71qxrO1nZpUT6IgYHcXnfGDFvDVpH66KiP+KiJ+Xr2daRLy1Yt33RfFZfyEivlOW+ckO9vm5EfHj8lhdGhH3R8Q7KpZ/Por2rPU9Oaxi2SqXGZQxfiYiHgMeK+e9LSLujojFZUyHV6y/WUTcGsVn83fA66+hTYztftYjYlRE3BTFZ+ZPUXEpVSfHyWrHY9vX0sE+vzQibo+Il4D3RJV2LCKGR8TPyvoXR8TU6B/JN0n9kI2bpH4vM38HPAW0nmx+A9iB4mRyO4pfpr8ExRdVissczgQ2BvYF5rVT7FeAu4BNgC2B/+yg+v8ENgK2Bf4BOAY4vmL5OOARYDjwH8DlERFtC4mIHYHPAnuUvSre30FcHfk48AFg48zcH5jKG79gPwq8VMa2cbneKRHxkbLurYE7ytcygmK/zSrL7XBftuO48vGecn9sCHw7M5dXdOV+R2a2e+IB/L6sZ1PgWuDGqDih70hEbFC+/spuzQGcD4wC/g7YCji3zaaHAwcBbwF2LWMnimTSGcCBwPZA22v4a3nPZwObla/jOmAPiv13NMUJb5e6tkfEpsDPgUvKci8Efh4Rm1Ws9gngZGAo8GfgaorLH7YDxgDvA1pPTNs9vjNz33L5O8pj5/WT4opYxlPsy2OAYcCHgUXthP0a8C8Ux/5ewAHAp8tl76P47O1AcUweUVHG5cCnys/BzsC9bQvu4BivjHF34ArgU+X++h5wa0SsW7Fa5Wem8jKRWtTaPrTWc1657lzga2WMw4EfA2eXMT4CvLuDMlodCtzIG5+Rn0bEoHLZ4xRt4EZlfT+MiDdXKesjFMfq28vP0N1lmZuXMX8nInYq1/0v4BXgzcAJ5WM17X3WyxPx24A/ULQfBwCnRcT7y/U6PE5qOR47cBTFfh4K/Jrq7dhEiv8fIyguifsCkDXWI0l9iskDSSo8DWxanpifBPxLZi7OzKXA14Ejy/VOBK7IzLszc2Vm/rWD6+Rfpeh6OyozX8nM1QYHi+JX3COAszNzaWbOAyZRnMS1+nNmfj8zX6M4mXszxRfUtl4D1qX4Ij8oM+dlZru/0Hfgksx8svwVfjWZOSUzHyhf82zgRxQnvgATgF9k5o8y89XMXJSZs2rYl21NAC7MzCcy80WKk6Ij2/6q25HM/GFZd0tmTqLYHztW2eSMiHgeWAr8PRX7PTPnlu/x8sxcQHGy/Q9ttr8kM5/OzMUUJzetPRcOB67MzDmZ+RIVSYca3/M/ZeaV5Xt+PUXi4stlLHdRjNFQbfDFS8pfQZ+PiPvLeR8AHsvMH5T750fAw8CHKra7KjMfLE+ENwUOBk7LzJcy81ngW7zx3nV6fFfxSYpLRn6fhbmZ+ee2K2XmjMz8bRnvPIoT+Nb34FWKE7u3AZGZD2XmMxXL3h4RwzLzucy8v23ZNTgJ+F5mTsvM1zLzamA5xXgSrap+ZjrRlf33k8z8Xfm+TOaN4+wQ4MHM/Em57BLgb53UOyMzf5yZr1Ic0+u1vqbMvLE8nleWJ9mPAXtWKev88nO9DPggMK88blvKfX4T8I/lMf8x4EvlsTSHoi2r1R7AiMz8cmauyMwngO9THoudHCdr6pbMvC+LcXCWU70de5WiXd6mbP+mZqbJA0lrJZMHklTYAlhM8evR+sCM1hMw4M5yPhQncrWclP8rxa/Xv4uIB6NN1/XScGAwxa+8rf5cxtLq9ZOBzHy5fLrar85ZDAZ2GsWJ6rMRcV1EjKohzlZPVlsYEeMi4n/LbsMvAP9Uxg8d75PO9mVbo1h9Xwyk/WRJezFOjKIr/AtlXRtVxNieCzJzY2A0sIyKRENEbF7uw79GxBLgh+2UVXmi9jJvvC+jWHV/Vr6mWt7z+RXPlwFkZtt51XoefC4zNy4frZcDtN237dVbGfM2wCDgmYr37nsUvypDbcd3R2r6DEVxKcHPIuJv5Xvwdcr3IDPvBb5N8Yv2/Ii4LCKGlZt+jOLE+s9RdOPfqwuxtdoGmFiRhHm+jLvyM1X1M9OJruy/mo6z8oT1KaqrXH9luf4ogIg4Jt64TON5il4b1T4/bY+XcW321wTgTRSf94F0/JnozDbAqDZlf4GyXah2nHRDZaydtWPfpOgRclcUlxl9vpt1S1KvZfJAUr8XEXtQnET9GlhIcXK2U8UJ2Eb5RlfaJ+nget1Kmfm3zDwpM0dRdH3+Tqx+q76FvPELZKutKQbv67LMvDYz/74sLym62kJxycH6Fau+qb3NOyn+WuBWYKvM3Aj4LsXJD3S8Tzrbl209zer7ooVVT6bbFcX4BmdR/Oq/SZkUeKEixg5l5l+AU4GL443xGM6n2Ce7ZuYwissFOi2r9AzFiWbl62hV1/e8C9ru2/bqrTwGnqT4xXV4xXs3LDN3gpqP747U9BkCLqXoHbF9+R58gYr3IDMvycx3AjtRdCk/s5z/+8w8lCLR8VOK8Sy66kngaxWvfePMXL/ssfF6CFW2f6n82+7nrpv7r9UzFJc8AMU4EJXTHXj9uCwvB9gSeDoitqH4Nf+zwGbl52cO1Y/5tsfLL9vsrw0z8xRgAcXnuKPPRGeepOiNU1n20Mw8pFxe9ThpxyrtYUR01h5Wbcey6EE0MTO3pejJc3qU481I0trG5IGkfisihkXEBymuKf9hlt3yKb5EfysiNi/X26Li+trLgeMj4oAoBvjbIiLe1k7Z4yOi9Yv8cxRfRl+rXCeLbuk3AF+LiKHlF/jTKX7l7upr2TEi9i+vyX6F4stua32zgEMiYtPyi/JpXS2foov44sx8pRz34aiKZZOB90bE4RExMIrB0XarYV+29SPgXyLiLeU1/V8Hrs/aricfSnGCsgAYGBFforieviaZeTfFCfbJFeW9CDwfEVtQnpjW6AbguIh4e0SsD5xTUU/d3vMuuh3YISKOKt+jI4C3Az9rb+XyEoC7gEnl52RAFINI/gN0enzPpxjPoSP/TXHJyDujsF25H9oaCiwBXiw/Y6e0LoiIPcreMIMoTgZfAV6LiMERMSEiNiq75i+hzeeuRt8H/qmsIyJigygGDR1ay8ZZXOryV+DoiFin7FlQOdBhp+1DDX4O7BIRH4ni0p7P0H5isNI7I+Kj5fqnUSSIfgtsUMawoIzveIqeB7X6GcXx9YmIGFQ+9oiIvyuP+Z8A50bE+hHxdqArg37+DlgSxQCVQ8r9uXOZ9IUqx0mp7fH4B2CniNgtijFRzq1WeWftWBQDa25XJm9aj7c1OeYkqdczeSCpP7otIpZS/KL1bxTX/lYOWHcWRTfU35bdYH9B2aU9i8EVj6e4/vsF4Jes/osuFNfpToti5PBbgVMz80/trPfPFCc/T1D0fLiWYqC2rlqX4raTCym6OW9O8QscwA8ovjDPozghrHXQsEqfBr5c7rcvUfFrbvnL/SEUA4ctpkhWtI7i3uG+bMcVZay/Av5EcUL4zzXG9z8UgzY+StEl+hW63q38m8C/lgmY84DdKd7jn1Oc/NQkM+8ALqIYqG8uqw/YV6/3vGaZuYjiuvSJFAML/ivwwcxcWGWzYygusfgjxQnujymu7Ybqx/e5wNVlF+/DaSMzb6QYjO5aivEmfkoxxkJbZ1AkqZZSnLxVHrfDynnPUbzfi4ALymWfAOaVx9s/UfQa6ZLMnE5xnfu3yzrmUg6I2QUnUSSdFlH0jvi/imW1tg/VYlwIjKcYSHURRTJoOkVCoCO3UIy58RzFfvpoeZ3+HynG3vgNxcn2LsB9XYhlKcUglkdSJOH+RtHzqXWAyc9SXG7xN+Aq4MoulP0axS/6u1G0CwspElAblatUO06gzfGYxeCYX6Zoix6j+Ax2plo7tn05/SLF/vtOZk6p9fVJUl8SjukiSZLUt5WXITwFTMjM/21n+bnAdpnZ5WSKJElgzwNJkqQ+KSLeHxEbl71lWq/1/22Tw5IkraVMHkiSJPVNe1HcuWIhRdf+j+Sa3TpSkqROedmCJEmSJEmqamCzA2iE4cOH5+jRo9ds4xeKWw8/snjVAZV33HRpu6u3Xa9WHZW3JuW2LevFv1QvW+1bb+eOxnCTVjUwBq3xtjNmzFiYmSM6X7P3WNM29cUH5lRd/tcNR3W4bMdNl67SDnbWZjZSy7CO45TUPf2uPd1ovRy9+Zp9d1STbbAhLevUfBMfqU/5w/2za2pP18rkwejRo5k+ffoabbvy9n8CYL9r91tl/pSjprS7ftv1atVReWtSbtuyfnNK9bLVvh3vu73ZIaiPGL7eFmu8bUT8uY6h9Ig1bVPv22a1O1iu4ux9zu1w2ZSjpqzSDnbWZjbS4v2/2LS6pbVdv2tPNx/K7y7+WLPD0BqIcXuzaIP9mx2G1BAjhmxZU3vqmAeSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKmqpiYPImKriPjfiHgoIh6MiFPbWWe/iHghImaVjy81I1ZJkiRJkvqrgU2uvwWYmJn3R8RQYEZE3J2Zf2yz3tTM/GAT4pMkSZIkqd9ras+DzHwmM+8vny8FHgK2aGZMkiRJkiRpVb1mzIOIGA2MAaa1s3iviPhDRNwRETt1sP3JETE9IqYvWLCggZFKkiRJktS/9IrkQURsCNwEnJaZS9osvh/YJjPfAfwn8NP2ysjMyzJzbGaOHTFiRGMDliRJkiSpH2l68iAiBlEkDiZn5k/aLs/MJZn5Yvn8dmBQRAzv4TAlSZIkSeq3mjpgYkQEcDnwUGZe2ME6bwLmZ2ZGxJ4UCY9FPRimJK1VImIesBR4DWjJzLHNjUiS+i7bVEn9RbPvtrA38AnggYiYVc77ArA1QGZ+F/hH4JSIaAGWAUdmZjYjWElai7wnMxc2OwhJWkvYpkpa6zU1eZCZvwaik3W+DXy7ZyKSJNXb/cte4twFT7EMGAKcO2JLdh+yQbvrrlj2PI9Pm8TGtzzOqy3fZ9jwbdly15M44N9u5nePLGCddWDMW4dz3ZkHMXKT9Xv0dVR6dv4CTpwwkTmzH2DnXXfh8smT2Hxk98fbqSx3h7ftSBA88vDDda1DknqrEybdw9X3LgBWAIM5dv8RXDHxgGaHJanU9DEPJEk9LoG7ImJGRJzc6MrOXfAUxwJzgWPL6Y7Mm34xQwbM5eBDkh/8YCX7jJvL3P87l+HbPcvV1yQHH5L8afECjr3o7kaHXdWJEyYyfdoYXlw6i+nTxnDihIl1L/f+3y9gxu/fWfc6JNVdj7apa7MicXACxX+ME8ppSb2FyQNJ6n/2zszdgYOBz0TEvm1XqOftb5cBZwCjyr/Lqqy7dNE8li6F8eNh+HA48khYvmwZRxxRTI8fD0uXwsy5zR36Zs7sB2hpOR0YRUvL6cyZ/UDdy4XFwJl1r0NS3VVtU1dpT194pTkR9hkrWPU/xormhiNpFSYPJKmfycyny7/PAjcDe7azTt1ufzsEuAB4uvw7pMq6QzcbzdChcOONsHAhXHcdrDtkCNdfX0zfeCMMHQpjttusWzF118677sLAgRcCTzNw4IXsvOsudS8XNgW+Wfc6JNVXZ23qKu3pRus1I8Q+ZDCr/scY3NxwJK3C5IEk9SMRsUFEDG19DrwPmNPIOs8dsSVXA9sBV5fTHRk99lSWrdyOO24PPvGJAUydth3bvftcFs7dnGOPCe64PXjLpiO4+rQDGxlypy6fPImx42ay4dDdGDtuJpdPnlT3cnffYwTv3GNG3euQVD/NaFPXZsfuPwK4guI/xhXltKTeotl3W5Ak9ayRwM3FnXIZCFybmXc2ssLdh2zArVvvWNO6g4dszN/t9xWmHDWF/a7d7/X593zmsAZFt2Y2HzmC235xTZ8pV1LD9Hibuja7YuIBXOHwLlKvZfJAkvqRzHwCeEez45CktYFtqqT+xMsWJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVSYPJEmSJElSVQObHYAkae2016X7AbDftcXf86eey9n7nFvTtov3/yJcO3XVaUmSJDWNyQNJUlNMPO+AKktXtFm+otHhSJIkqQqTB5Kkpth7i+oJgc6WS5Ikqec45oEkSZIkSarK5IEkSZIkSarK5IEkSZIkSarK5IEkSZIkSarK5IEkSZIkSaqqqcmDiNgqIv43Ih6KiAcj4tR21omIuCQi5kbE7IjYvRmxSpIkSZLUXzX7Vo0twMTMvD8ihgIzIuLuzPxjxToHA9uXj3HApeVfSZIkSZLUA5ra8yAzn8nM+8vnS4GHgC3arHYocE0WfgtsHBFv7uFQe8z8517moHNuYcZPj+fRqeexYtnzqyxfsex5Hr/rPGZefzyP3H42s245jvU/+j2Gjf8e+/zrTcx/7mUWv9bCWfP/wqFPPsZZ8//C4tdamvRqtDZ7dv4CPvTeY3jL5mP40HuP4dn5C2pa1ui6G7Vt5Xb77nsI8+fPX9PwJUmSpD6n14x5EBGjgTHAtDaLtgCerJh+itUTDGuNYy+6m822nc/VV73CPuMeZd70i1dZ/uTUizl84aM80vIKw5bN433vX84Pfwgf+CA8uWQhx150N99Y+DTjli9jdq5k3PJlfGPh0016NVqbnThhItOnjeHFpbOYPm0MJ06YWNOyRtfdqG0rt/vNb3Zi/Pjj1zR8SZIkqc/pFcmDiNgQuAk4LTOXtF3czibZThknR8T0iJi+YEF9f+XsSTPnLuJj45Phw+GIw1eydNG8VZYveW4eZ+ZKRgHPrYQjj4Thw2H8eFi6tNj+4RXLmQiMAiYCD69Y3vMvRGu9ObMfoKXldGAULS2nM2f2AzUta3Tdjdp21e3+hVmzZqxR7JIkSVJf1PTkQUQMokgcTM7Mn7SzylPAVhXTWwKr/ZSemZdl5tjMHDtixIjGBNsDxmy3GTfdGCxcCNffMIChm41eZfmwTUbzzRjA08AmA+C662DhQrjxRhg6tNj+bYPXZRLFTpoEvG3wuj3/QrTW23nXXRg48ELgaQYOvJCdd92lpmWNrrtR26663bfYbbd3rlHskiRJUl/U7LstBHA58FBmXtjBarcCx5R3XXgX8EJmPtNjQfawq087kEVPjOTY49Zj6rQdGD121RtQbLXPqdwwfAd2HLgeS4aM5q7/WZejj4af/wy2Gjacq087kLOGj2LaukPYNQYwbd0hnDV8VJNejdZml0+exNhxM9lw6G6MHTeTyydPqmlZo+tu1LaV2+2114PceOOVaxq+JEmS1Oc0+24LewOfAB6IiFnlvC8AWwNk5neB24FDgLnAy8BafaHxyE3W587zDmW/a/drd/ngIRvz1veds8q8KUdNWWV603UG8o2RWzcoQqmw+cgR3PaLa7q8rNF1N2rbyu2Gr7fWDrsiSZIktaupyYPM/DXtj2lQuU4Cn+mZiCRJkiRJUltNH/NAkiRJkiT1biYPJEmSJElSVSYPJEmSJElSVSYPJKkfioh1ImJmRPys2bFIUl9meyqpvzB5IEn906nAQ80OQpLWArankvqFZt+qUZLUwyJiS+ADwNeA0xtVz833PcFxl9zNq69+j4ED4T0vwZDJH2clMGCDkYyYvBxYDqzLRZd+ket++HPmzH6AnXfdhcsnT2LzkSMaFZok1UVPtadSpY8cOJ77fv1HYAUwmL3//u389O4bmx2W+gF7HkhS/3MR8K/AykZWctwld3PQQfDDH8LBB8O6G8BJwChgyUuvAMcDc4HjOe2ULzN92hheXDqL6dPGcOKEiY0MTZLqpUfaU6lSkTg4geJ/6AnltNR49jyQpH4kIj4IPJuZMyJivyrrnQycDLD11luvUV2vvgpHHAHDhxd/b7ut+FnuGqD4teQMilTCGcB3aWk5HRhFS8vpzJm92xrV2dM2vfcrzQ6hz1i8/xebHYJUV2vUno7YsIei09pt9f+hUk+w54Ek9S97Ax+OiHnAdcD+EfHDtitl5mWZOTYzx44YsWaXDwwaBNdfDwsXFn8HDYQLgU0AGAxcADxd/h3MwIEXAk8zcOCF7LzrLmtUpyT1oK63pxut19Mxaq20+v9QqSeYPJCkfiQzz87MLTNzNHAkcG9mHt2Iuq763IHceSccfTTccQcsfwm+T/FVZ9gG6wFXAtsBV3LRpV9i7LiZbDh0N8aOm8nlkyc1IiRJqpuebE+lSnv//duBKyj+h15RTkuN52ULkqSGOGzvbTls70+x37X7AXD+1HM5e59zX1/+k//eZ5X1Jxx3RA9GJ0lS3+TgiGoWkweS1E9l5hRgSpPDkKQ+z/ZUUn/gZQuSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKkqkweSJEmSJKmqmpIHEbF+RHwxIr5fTm8fER9sbGiSJEmSJKk3qLXnwZXAcmCvcvop4KsNiUiSJEmSJPUqA2tc762ZeUREfBwgM5dFRDQwLkmS6uKjn5xa1/J+8t/71LU8SZLWJvX+v6veo9bkwYqIGAIkQES8laIngiRJ7frNKVMAOJ8pr887f+q5rz9/ZMceCmSfcztdpSs2vfcrdS1PktT75bT72JT7mh1Gr7d4/y82OwQ1UK3Jg3OAO4GtImIysDdwXKOCkiRJkiRJvUenYx6Ulyc8DHyUImHwI2BsZk7pbuURcUVEPBsRczpYvl9EvBARs8rHl7pbpyRJkiRJ6ppOex5kZkbETzPzncDP61z/VcC3gWuqrDM1M/vVnR1i3N5wbRfXr7DLQY+sts4Ddz7T3bAkSZIkSf1UrZct/DYi9sjM39ez8sz8VUSMrmeZjfLiLY+w4uIfrr7g2toHBNnt4O0rpqZ0uN6pd74deKzmcov1K+x7wSqTB7xraz50qNcfdWZxswOQJEmSpF6q1uTBe4BPRcSfgZeAoOiUsGvDInvDXhHxB+Bp4IzMfLC9lSLiZOBkgK233rruQTxw5zO0N7bXxPMOqLmM9/zH4a8//02VngCz9tmvC5HBrDuqJxpm3fEYHzqqS0X2S20HQXPAF0mSJEkq1Jo8OLihUXTsfmCbzHwxIg4Bfgps396KmXkZcBnA2LFjs6cC3HuLFTWv66UDkiRJkqS+qNMBE0vZwaOhMnNJZr5YPr8dGBQRwxtdryRJkiRJekOtPQ9+TpEsCGA94C3AI8BODYoLgIh4EzC/HLRxT4pkx6JG1ilJkiRJklZVU/IgM3epnI6I3YFPdbeAg5WLAAAgAElEQVTyiPgRsB8wPCKeAs4BBpV1fhf4R+CUiGgBlgFHZmaPXZIgSZIkSZJq73mwisy8PyL26G7lmfnxTpZ/m+JWjpIkSZIkqUlqSh5ExOkVkwOA3YEFDYlIkiRJkiT1KrX2PBha8byFYgyEm+ofjiRJkiRJ6m1qTR78MTNvrJwREeOBGztYX5IkSZIkrSVqvVXj2TXOkyRJkiRJa5mqPQ8i4mDgEGCLiLikYtEwissXJEmSJEnSWq6zngdPA9OBV4AZFY9bgfc3NjRJ6r8i4i21zJMkdc42VZK6r2rPg8z8A/CHiJicmfY0kKSecxPFnW0q/Rh4ZxNikaS+zjZVkrqps8sWbsjMw4GZEZFtl2fmrg2LTJL6oYh4G7ATsFFEfLRi0TBgveZEJUl9k22qJNVPZ3dbOLX8+8FGByJJAmBHijZ3Y+BDFfOXAid1t/CIWA/4FbAuxf+AH2fmOd0tV5J6KdtUSaqTzi5beKb8++eeCUeS+rfMvAW4JSL2yszfNKCK5cD+mfliRAwCfh0Rd2TmbxtQlyQ1lW2qJNVP1QETI2JpRCzp6NFTQUpSP3RYRAyLiEERcU9ELIyIo7tbaBZeLCcHlY/VLkvrSx5f8QqHP/kY7/vLIxz+5GPcv+wlzpr/Fw598jHOmv8XVix7nhXLnufxu85j5vXH8/hd57Fi2fN1j2P+cy9z0Dm3MHLCFRx0zi3Mf+7lmpZ1tdw58xatcVnq3LPzF/Ch9x7DWzYfw4feewzPzl/Qq8ruLWX0QbapWmscdt7trPOB61jnA9ewzgeu47Dzbm92SOonqiYPMnNoZg4DLgI+D2wBbAmcBXy18eFJUr/1vsxcQtHd9ilgB+DMehQcEetExCzgWeDuzJxWj3Kb5ez5T3JEruQx4IhcybkLnmLc8mXMzpWMW76MJ6dezJNTL+bwhY/ySMsrHL7wUZ6cenHd4zj2orvZbNv5XPr9V9ls2/kce9HdNS3rarkHnXvLGpelzp04YSLTp43hxaWzmD5tDCdOmNiryu4tZfRBtqlaa9z6uyXACcBc4IRyWmq8zm7V2Or9mfmdzFyamUsy81LgY40MTJL6uUHl30OAH2Xm4noVnJmvZeZuFMngPSNi57brRMTJETE9IqYvWNC7f5V8PldyBjAKOANYBkwspycCS56bx5Ln5nFmrmQUcGauZMlz8+oex8y5i/jY+GT4cPjY+GTm3EU1LetquYuXvLrGZalzc2Y/QEvL6cAoWlpOZ87sB3pV2b2ljD6oaW3qKu3pC6/Uq1r1aytglf98K5objvqNWpMHr0XEhDKzOiAiJgCvNTIwSernbouIh4GxwD0RMQKo67fOzHwemAIc1M6yyzJzbGaOHTFiRD2rrbuNYwAXAE8DFwBDgEnl9CRg2CajGbbJaL4ZA3ga+GYMYNgmo+sex5jtNuOmG4OFC+GmG4Mx221W07KulrvpsEFrXJY6t/OuuzBw4IXA0wwceCE777pLryq7t5TRBzWtTV2lPd3IGzyoHgbDKv/5Bjc3HPUbtSYPjgIOB+aXj/HlPElSA2Tm54G9gLGZ+SrwEnBod8uNiBERsXH5fAjwXuDh7pbbTOeP3IrrYwDbA9fHAM4dsSXT1h3CrjGAaesOYat9TmWrfU7lhuE7sOPA9bhh+A5stc+pnZbbVVefdiCLnhjJKScNYtETI7n6tANrWtbVcu8899A1Lkudu3zyJMaOm8mGQ3dj7LiZXD55Uq8qu7eU0dfYpmpt8uE9hwFXANsBV5TTUuN1dqtGADJzHnVoYCVJtSlH7f4EsG9EAPwS+G4din4zcHVErEORQL4hM39Wh3Kb5q2D1+OGrbZfZd7uQzZ4/fnZQzYu1ntfY++eNnKT9bnzvPb/VVZbtiblrmlZ6tzmI0dw2y+u6bVl95Yy+hrbVK1Nbj7nkGaHoH6qpuRB2bXrJGB05TaZeUJjwpKkfu9Simt0v1NOf6Kc98nuFJqZs4Ex3QtNkvoc21RJ6qaakgfALcBU4Bc41oEk9YQ9MvMdFdP3RsQfmhaNJPVttqmS1E21Jg/Wz8yzGhqJJKnSaxHx1sx8HCAitsXkrSStKdtUSeqmWpMHP4uIQzLz9oZGI0lqdSbwvxHxBBDANsDxzQ1Jkvos21RJ6qZakwenAl+IiBUUNxINIDPToT0lqQEy856I2B7YkaLNfTgzlzc5LEnqk2xTJan7arpVY2YOzcwBmbleZg4rp00cSFKDRMRngCGZOTsz/wCsHxGfbnZcktQX2aZKUvfVlDyIwtER8cVyequI2LOxoUlSv3ZSZj7fOpGZz1Hc9UaS1HW2qZLUTTUlDyhua7MXcFQ5/SLwXw2JSJIEMCDKm5EDlPcQH9zEeCSpL7NNlaRuqnXMg3GZuXtEzIQiWxsRNriS1Dj/A9wQEd8FEvgn4M7mhiRJfZZtqiR1U63Jg1fLDG0CRMQIYGXDopIknQWcDJxCMbjXXcB/NzUiSeq7bFMlqZtqTR5cAtwMjIyIrwH/CPy/hkUlSf1cZq4Evls+VhMRN2Xmx3o2Kknqm2xTJan7akoeZObkiJgBHFDO+khmPtS4sCRJndi22QFI0lrENlWSOlFrzwOA9YHWSxeGNCYcSVKNstkBSNJaxDZVkjpR660avwRcDWwKDAeujAgvW5AkSZIkqR+o9VaNHwf2yMxzM/Mc4F3AhHoEEBFXRMSzETGng+UREZdExNyImB0Ru9ejXknq46LzVSRJNbJNlaRO1Jo8mAesVzG9LvB4nWK4CjioyvKDge3Lx8nApXWqV5J6nYi4p/z7jU5WPasHwpGkPs02VZLqp9YxD5YDD0bE3RTXhB0I/DoiLgHIzM+taQCZ+auIGF1llUOBazIzgd9GxMYR8ebMfGZN66zm5vue4LhL7ubVV7/HwIGwYjkMHgj7t8CKITsBK4DBvOWto3h2/gu89OLzDFhnPdZbF3YdsxvfuOgLnHXa15kz+wF23nUXLp88ic1HjliljsWvtfDVBX/ljyteYSAwetC6DI7g8VdX8LbB69Ky7HkGD9mYFcue58mpF7PkuXkM22Q0W+1zKoOHbFzX1zv/uZc59qK7mTl3EWO224yrTzuQkZus3+myrpT79m02ZkAEc+Y91+VyVJtn5y/gxAkTqx53zSq7HrE18vX1Qm+OiH8APhwR19Hm17DMvL/8e1czgpOkPsY2VZLqpNbkwc3lo9WU+ofSoS2AJyumnyrnNSR5cNwld3PQQXDEEXD99fDLX8K++8JNN20EnACcAVzAnx6/CjgWOJOVr13Ayy/PZvq03fjgAcey7OUjaWn5AdOnXciJEyZy2y+uWaWObyx8mneveIWbgQuBa15dzgSKHTxp+TKum3oxb33fOTw59WIOX/goZ+ZKvrnwUW4o59fTsRfdzWbbzufSs5KbbpzPsRfdzZ3nHdrpsq6Ue/rpCxg3Di79N7pcjmpz4oSJTJ82pupx16yy6xFbI19fL/Ql4PPAlhRNRKUE9u/xiNbQwnuncs9v/7La/CO+sOpVbzs+ck9jA/nk1LoWt3j/L7Y7f9N7v9Kt7auptWxJq1lr2lRJarZakwd3ZOazlTMiYsfMfKQBMbXV3jVoq42IGxEnU1zWwNZbb73Glb36apE4GD68+HvbbXDkkXDTTSsoEgejyr/fBc6smB5DS8s1LF1yKXA6MIqWltOZM3u31ep4eMVybii3PJ3iOozWkiYClz03D4Alz83jzFzJKODMXMl/l/PraebcRVx6VjJ8OHxsfHLKSYtqWtaVcpcufWOfdrUc1WbO7AdoafkB1Y67ZpVdj9ga+fp6m8z8MfDjiPhiZvbpM8ZJ57SfFDiih+OYeN4Bna/UJSvqXJ6kRultbeqLf1nKb06Z0uwwpIbZ5aBHmHjeDc0OQ100dXJt69WaPJhaNro3AETEROBE4O1rFF3XPAVsVTG9JfB025Uy8zLgMoCxY8eu8e12Bg0qehy09jwYNgyuuw5gMHABrT0PiiEgvklx2n8BsAsDB17IkPU3YdnLF9LScjoDB17Izrvuslodbxu8Lt9cvowzKVLgG1SUNAkYtsloKP9+s7XnQQx4fX49jdluM266cT4fG5/cdGMwZrvNalrWlXKHDn1jn3a1HNVm5113Yfq06sdds8quR2yNfH29VWZ+JSI+DOxbzpqSmT9rZkx91d5beLIv9Xe2qVLP8f/u2qvWARP3Az4RETdGxK+AHYA9GxbVqm4FjinvuvAu4IVGjXcAcNXnDuTOO+Hoo+GOO+D55+DOn8PggS8AVwDbAVfwlre+iQ02vAHYngHrXM36689k7LiZ/Oyeqxk7biYbDt2NseNmcvnkSavVcdbwUfzf4PXYEbgc2GLQuvzf4PXYNQYwbd0hbLXPqQBstc+p3DB8B3YcuB43DN/h9fn1dPVpB7LoiZGcctIgFj0xkqtPO7CmZV0p9y2bjmDh3M3XqBzV5vLJkzo97ppVdj1ia+Tr660i4nzgVOCP5ePUcp4kqYtsUyWp+2rqeZCZz0TEncDZwErg7Mx8sR4BRMSPKJITwyPiKeAcYFBZ73eB24FDgLnAy8Dx9ai3I4ftvS2H7f0p9rt2v1Xmnz/13Jqvy+3sWuxN1xnIhW/apsPlZ5eDIg4esnHdxzhoa+Qm63c4/kC1Zd0pV/W3+cgRDRsDoLtl1yO2Rr6+XuwDwG6ZuRIgIq4GZlK0w5KkrrFNlaRuqil5UN5l4RlgZ4rLBq6IiF9l5hndDSAzP97J8gQ+0916JKkP2hhYXD7fqJmBSNJawDZVkrqh1jEP/iszf1o+fz4i3o2ZWklqpPOBmRHxvxQDx+6L7a4krSnbVEnqplovW/hpRGwDbJ+Zv6C4rOCihkYmSf1YZv4oIqYAe1B80T0rM//WujwidsrMB5sVnyT1JbapktR9tV62cBLFbRA3Bd5KcenCd4F63/9KklQqB4e9tYPFPwB278FwJKlPs02VpO6p9W4LnwH2BpYAZOZjwOaNCkqS1KlodgCStBaxTZWkTtSaPFiema/fsDMiBgLZmJAkSTWwDZak+rFNlaRO1Jo8+GVEfAEYEhEHAjcCtzUuLEmSJEmS1FvUmjz4PLAAeAD4FHA78P8aFZQkqVMrOl9FklQj21RJ6kRNyYPMXJmZ38/M8Zn5j+Xz17t3RcRNjQtRkvqfKBwdEV8qp7eOiD1bl2fmu5oXnST1LbapktR9tfY86My2dSpHklT4DrAX8PFyeinwX80LR5L6NNtUSeqmmm7VWAMHmZGk+hqXmbtHxEyAzHwuIgY3OyhJ6qNsUyWpm+rV80CSVF+vRsQ6lMnZiBgBrGxuSJLUZ9mmSlI31St54L1xJam+LgFuBkZGxNeAXwNfb25IktRn2aZKUjdVvWwhIu7JzAMi4huZeVaVVastkyR1UWZOjogZwAHlrI9k5kPNjEmS+irbVEnqvs7GPHhzRPwD8OGIuI42PQwy8/7y710Nik+S+rP1gdZutkOaHIsk9XW2qZLUDZ0lD74EfB7YEriwzbIE9m9EUJLU35W3ExsP3ESRuL0yIm7MzK92s9ytgGuAN1Fc73tZZl7c3XglqTdrRJtqeyqpv6maPMjMHwM/jogvZuZXeigmSVJxO7ExmfkKQET8O3A/0K3kAdACTMzM+yNiKDAjIu7OzD92s1xJ6s0a0abankrqV2q6VWNmfiUiPgzsW86akpk/a1xYktTvzQPWA14pp9cFHu9uoZn5DPBM+XxpRDwEbAE07MvuimXPM2/6xSxZNI9Bkbzv1eW8ljBwAAx5BW6a8zDDR2zGJ8b/M7NmPAjxKmN2H8OF3zmHs077OnNmP8DOu+7C5ZMnsfnIEY0Ks1eLcXs3OwSpr5tHndvUZrSnEsCkhc9w88sBrAAGc9j6ycThb252WOoHakoeRMT5wJ7A5HLWqRGxd2ae3bDIJKl/Ww48GBF3U1wmdiDw64i4BCAzP9fdCiJiNDAGmNbdsqqZN/1i9hn3KEccvpLrroPfTYMLvwXXXwf3/hwmHPARttxlT+7//QLgBOB0Zvz+m3zwgGNZ9vKRtLT8gOnTLuTECRO57RfXNDLUXmvRBl4lKHVTQ9vUnmpPJaBMHJwAnAFcwM0vX8HEJsek/qGm5AHwAWC3zFwJEBFXAzMBkweS1Bg3l49WU+pZeERsSHHt72mZuaSd5ScDJwNsvfXW3apr6aJ5HHH4SoYPhyOPhHvugeHD4Ygj4bbb4JUlL/L87AfKtU8HRgFnsnTJZa9Pt7SczpzZu3UrDkn9WsPa1K60pyPXqfWrt1TNCorEwajy73ebG476ja60YBsDi8vnGzUgFknSGxYBt7cmbespIgZRfNGdnJk/aW+dzLwMuAxg7Nix2Z36hm42mutveKPnwdANYeHCoufBBgNg/WEbsuUuu/Db+/5KMTbv6cA3GTpsE5a9fCEtLaczcOCF7LzrLt0JQ1L/1pA2tavt6dvWXa9b7alUGAxcQGvPg2JaarwBNa53PjAzIq4qex3MAL7euLAkqd87EngsIv4jIv6uXoVGRACXAw9lZtu76DTE6LGnMnXaDhxz3Hrc/T/rsmgBHD0B7vg5DHgFJt/zUy6fPInd9xjBgAFXMGCdHXnnHjP42T1XM3bcTDYcuhtjx83k8smTeiJcSWunurepzWhPJYDD1k/gCmA74IpyWmq8WgdM/FFETAH2oLi9zVmZ+bfW5RGxU2Y+2JgQJan/ycyjI2IYxQjhV0ZEAlcCP8rMpd0oem/gE8ADETGrnPeFzLy9exF3bPCQjdlhn3Nenz5/6rmrLN9x57cB8D+/um61bfvrGAeS6qtBbWqPt6cSwMThb3aMAzVFzZctlCPK3trB4h8Au9clIkkSAJm5JCJuAoYApwGHAWdGxCWZ+Z9rWOavKZLAktSv1LtNtT2V1N/UetlCZ2w4JamOIuLDEXEzcC8wCNgzMw8G3kFxkaMkqUa2qZLUffUa8tULbSSpviYA38rMX7XOiIhvZOZZEXFCE+OSpL7INlWSuqlePQ8kSfW1feWX3NLBAJl5TxPikaS+zDZVkrqpXj0PVtSpHEnq1yLiFODTwLYRMbti0VDgvuZEJUl9k22qJNVPTcmD8lY0E4BtM/PLEbE18KbM/B1AZr6rgTFKUn9yLXAHxS1yP18xf2lmLm5OSJLUZ9mmSlKd1Nrz4DvASmB/4MvAUuAmils3SpLqJDNfAF6guJ2YJKkbbFMlqX5qTR6My8zdI2ImQGY+FxGDGxiXJEmSJEnqJWodMPHViFiH8q4KETGCoieCJEmSJElay9WaPLgEuBkYGRFfA34NfL0eAUTEQRHxSETMjYjPt7P8uIhYEBGzyscn61GvJEmSJEmqTU2XLWTm5IiYARxQzvpIZj7U3crL3gz/BRwIPAX8PiJuzcw/tln1+sz8bHfrqymmcXsXT67tfN37/trxlRt7b+ENKCRJkiRJa4eu3KpxfaD10oUhdap/T2BuZj4BEBHXAYcCbZMHPebUO98OwG4Ht1kwdfV17/ntXzosZ++PvWm1ee++/99qDKK21bpi8f5frH+hkiRJkqR+odZbNX4JGE9xh4UAroyIGzPzq92sfwvgyYrpp4Bx7az3sYjYF3gU+JfMfLLtChFxMnAywNZbb73GAc264zEAphw1hRdveeT1+Q9UWbddZfJgs5fuZdEG+wO8/rczE89rxFiU9oSQJEmSJK2ZWnsefBwYk5mvAETEvwP3A91NHkQ787LN9G3AjzJzeUT8E3A1xS0jV90o8zLgMoCxY8e2LWONPHDnM90uI6fdB/vXljRo5SUPkiRJkqTepNYBE+cB61VMrws8Xof6nwK2qpjeEni6coXMXJSZy8vJ7wPvrEO9kiRJkiSpRrUmD5YDD0bEVRFxJTAHeDEiLomIS7pR/++B7SPiLRExGDgSuLVyhYh4c8Xkh4FuD9QoSZIkSZJqV+tlCzeXj1ZT6lF5ZrZExGeB/6EYjPGKzHwwIr4MTM/MW4HPRcSHgRZgMXBcPeqWJEmSJEm1qTV5sAi4PTNX1juAzLwduL3NvC9VPD8bOLve9UqSmmPhvVN5z38cztCvnwDAqTf97fVlX2rnTjXVfLli23roav2SJEn9Ra3JgyOBiyPiJuDKzPTSAUlSVb+8+FX+4dRBq82fdM49fOioHV+/A82sOyruhfuxN7HZS/fWXMesO1Yvvzs2O2jVOwUv/cIVHa7bmvxoVetIvV15fX1NrXcVkiRJfU9NyYPMPDoihlHcdeHKiEjgSoq7ICxtZICSpL4pp90H7LeG29Wq6+V3pe5qd93Z69CuxNlxHWuVLt5dSJIk9R21DphIZi4BbgKuA94MHAbcHxH/3KDYJEmSJElSL1BT8iAiPhwRNwP3AoOAPTPzYOAdwBkNjE+SJEmSJDVZrWMeTAC+lZm/ap0REd/IzLMi4oQq20mSJEmSpD6u1ssWtq9MHJQOBsjMe+obkiRJkiRJ6k2q9jyIiFOATwPbRsTsikVDgbV4xCdJkiRJktSqs8sWrgXuAM4HPl8xf2lmLm5YVJIkSZIkqdeomjzIzBeAFyhu0ShJkiRJkvqhmm/VKEmSJEmS+ieTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5IkSZIkqSqTB5LUj0TEFRHxbETMaXYsktTX2aZK6k9MHkhS/3IVcFCzg5CktcRV2KZK6idMHkhSP5KZvwIWNzuOnrBi2fM8OvU8Zvz0eB6deh4rlj3f7jqP33UeM647jpk/OYr1P/o9ho3/Hvv8603Mf+5lFr/Wwlnz/8KhTz7GWfP/wuLXWprwSrS2e3b+Aj703mN4y+Zj+NB7j+HZ+QtqWtYT9Tdq29bthg0byb77HsL8+fPXNPym6k9tqtQXnTTh04wYshMjhmzPiCE7cdKETzc7pD7N5IEkqUftdvD2PVLPvOkXs8+4R7n6qlfYZ9yjzJt+8WrrPDn1Yg5f+Ch7DFrO+w9JfvhD+MAH4cklCzn2orv5xsKnGbd8GbNzJeOWL+MbC5/ukdjVv5w4YSLTp43hxaWzmD5tDCdOmFjTsp6ov1Hbtm63dOlMfvObnRg//vg1DV+SOvTTn/wSOAGYC5xQTmtNmTyQJK0mIk6OiOkRMX3Bgvr+0nnEFybUtbyOLF00jyMOX8nw4XDE4StZumjeausseW4eZ+ZKHlwJRx4Jw4fD+PGwdCnMnLuIh1csZyIwCpgIPLxieY/Erv5lzuwHaGk5HRhFS8vpzJn9QE3LeqL+Rm276nb/wqxZM9Yo9r6gsj19/rXXmh2O1M+sAP5/e/cdH1WV/nH88zCBhA4xEEBgxYYgIT1IFYEglhVBkGbZFUV0d9W14mJD1i72iuKqWGh2xZ8iGkGpoYOYhVV0EZfepSWc3x9zExOSDJOQZFK+79drXrnl3HOfc+/kucmZe8/cjP9KfrM3L8WlzgMREcnHOTfeOZfknEtq1KhRqMMplrrHncDkKdXYsgUmT6lG3eNOyFemXsMTeMSqcXo1mDQJtmyBqVOhbl2IP/k4TqsRzjhgAzAOOK1GeBm3QqqCdu1jCAt7DNhAWNhjtGsfE9S6sth/aW2bd7vHiYtLLFbsFUHufNrA5wt1OCJVTA3gUfxX8ke9eSkudR6IiEildELS9cyefyqX/ymC2fNP5YSk6/OVadH1eqZEncrCQ+F8Nt245BL45GNoUS+K125I5baoZswPr0l7q8b88JrcFtUsBC2Rym7Cm+NI6rCEOnXjSOqwhAlvjgtqXVnsv7S2zd6ubt14OnZcxdSp/ypu+CIihbqw/5nAK8DJwCvevBRXWKgDEBGRsmNmbwPdgSgzWw/c7ZybENqoSkeNmg04tevdRy1zUu/fy6QNTcuzPtIXxkPRLUsjPJEcjaMb8dEXrxd5XVnsv7S2zd4uKuL4Yu23vKhKOVWkInrpzed4KdRBVCLqPBARqUKcc0NCHYOISGWhnCoiVYkeWxARERERERGRgNR5ICIiIiIiIiIBhbzzwMz6mFmGma01s1EFrA83s8ne+vlmdkLZRykiIiIiIiJSdYW088DMfMCzwDlAW2CImbU9othwYLtz7mTgceChso1SREREREREpGoL9YCJKcBa59wPAGY2CegLfJerTF/gHm96GvCMmZlzzpVkIBs3bmTw4L4smLuAsDBHrWngM6i2H3zAYcAXncyhrEO0bnsahrF0ySpq1m9Jjeo7OLD7J7KyjHpRJxLdejAnRd/Arl27qFbNR2Lytzz6zBPcdsP9rFy+gnbtY5jw5jgaR5fdd6dv2riZ4cNuKpX9Z9e9fOkyfL4IDmcdICaufZm3UUREREREREpHqB9bOB74b6759d6yAss45zKBncBxJR3IZZcNJCpqPqef7jjnHHjjDTj3PGgSAU2AA9Rm566L2bt3GYsXbmbRwkSyMlewZ2t39mzbzHnnOSZOPEzXDmv5Lu0pdu8eDKzFueEsW/IjF6ReSvr8ePbsXkr6/HiGD7uppJsQ0PBhN5Xa/rPr/m1vHLu9YxSKNoqIiIiIiEjpCHXngRWw7Mg7CoIpg5mNMLN0M0vfvHlzkQNZvHgpgwbBjz/CoEEQFQWDBsP2w7ANOMhhHLcCzfAvucWbvoWDBw8zcKB/m8GDISvrEM75yzp3KwcPHmbnjh1kZt4INCMz80ZWLl9R5BiPxcrlK0pt/7/XvQK4uVT2ISIiIiIiIqET6s6D9UCLXPPNgQ2FlTGzMKA+/v/e83DOjXfOJTnnkho1Kvqt8gkJcUyeDK1aweTJsGULTJ4EDatBJFCDahgPe+FFAo94049Qo0Y1pk71bzNpEvh81THzlzV7mBo1qlG/QQPCwh4DNhAW9hjt2scUOcZj0a59TKnt//e6Y4BHS2UfIiIiIiIiEjqh7jxYCJxiZq3MrAYwGPjwiPO95NgAACAASURBVDIfApd70wOAL0t6vAOA11+fypYtHVi1yvj0U7jkEpj+CfxvP/wPCGcv9etNoXbtWBKSG5GYvAhfWAx1jkujTmQjPvnEuPTSasyefzJtu19H3bqTgJMxm0BsfCs+nDGRpA5LqFM3jqQOS5jw5riSbkJAE94cV2r7z667Vu2l1PWOUSjaKCIiIiIiIqUjpAMmOucyzeyvwGf4xyV8xTm3yszuBdKdcx8CE4CJZrYW/x0Hg0sjlujoaL76ah7dLpkEQNrQNOZek5anTOuMmXnm+185u9D6vty4CIDIL8eyrcedAHz0xeslGHHRNI5uVGr7L826RUREREREJPRC/W0LOOemA9OPWHZXrun9wMCyjktERERERERE/EL92IKIiIiIiIiIlHPqPBARERERERGRgNR5ICIiIiIiIiIBqfNARERERERERAJS54GIiIiIiIiIBBTyb1sQEZGqJ/srbIu6Lp+3Cv/K3OI4ct+dFncutOzW2j1KdN8iIiIi5Zk6D0REpNSlDU3LmZ57xP/7N43pmWvuYJHqzbttSci7f3UQiIiIiPip80BERErd3GvSCl3X+fiidRiU1LYiIiIiEjyNeSAiIiIiIiIiAanzQEREREREREQCUueBiIiIiIiIiASkzgMRERERERERCUidByIiIiIiIiISkDoPRERERERERCQgdR6IiIiIiIiISEDqPBARERERERGRgNR5ICIiIiIiIiIBqfNARERERERERAJS54GIiIiIiIiIBKTOAxEREREREREJSJ0HIiIiIiIiIhKQOg9EREREREREJCB1HoiIiIiIiIhIQOo8EBEREREREZGA1HkgIiIiIiIiIgGp80BEpIoxsz5mlmFma81sVKjjERGpqJRPRaQqUeeBiEgVYmY+4FngHKAtMMTM2oY2KhGRikf5VESqmrBQByAiImUqBVjrnPsBwMwmAX2B70pyJytXrqTnZRPYs+9FsrKgt4FvP1QHDgNEJ3Mo6xCt256GYWR8/z3t2scw4c1xNI5ulFPPpo2bGT7sJlYuX5GzHsi3LPc2IiJlpEzyaVUyauPPfHOgBnAQqEGX8IM8GN0y1GGJiEd3HoiIVC3HA//NNb/eW1aievfuTHitTM4/H954A845D6IjwAGtqc3OXRezd+8yFi/czKKFiezZvZT0+fEMH3ZTnnqGD7uJ9PnxedYXtExEJATKJJ9WJf6OgyuAtcAV3ryIlBe680BEpGqxApa5fIXMRgAjAFq2LPqnPtu27aJmTRg4EKKiYPBgSPsI9gPfcRjHrUAzYBtwC9CMzMwbWbk8Lk89K5evIDNzYr71BS0TESljRc6n0T796R3YQeBm/NeHm4EXQhtORVC/HhF/G0m1P7TArGQ+F966fTuPvvAcK1avpl6duoRVD+NPAwfRo0uXo27735o+3A/bSiSO0vb+ex/QqXNHGjdunG/dsmXLeOiBhzh48BAHDx2kT5+zufYv1xZ5HxNfn8iAgQOoWbNmSYR8jIzq4T4aRNfGF+YrVg3KYCIiVct6oEWu+ebAhiMLOefGA+MBkpKS8v0xfDSRkfUIC9vF1Kn+DoRJk6BhNf8jC6dSjQU87HUgRAKPALcQFvYY7drH5KmnXfsY0uc/RmbmjXnWF7RMRKSMFTmfnhYeUeR8WrXUAB7F33HwqDcvgUT8bSSNE+JoEFYds4L6s4rGOceIm29k2B/7MvWJpwD4ecMGpqd9xcnR0Ufdvmb96hxu0apI+8zMzCQsrOz/LZ0xYwa9UnvS+rTW+db173cRb016i9jY9mRlZZGR8e8Cyx3N5ElTuOHGG4iKigp6m6ysLHy+4v1zH4hzju3btrNj43aOO75eseoI2WMLZhZpZjPMbI33s2Eh5bLMbKn3+rCs4xQRqWQWAqeYWSszqwEMBko8t37++bcc+C2Mjz+GSy6BTz+Bjfv9H9NlsJf69aZQu3YsCcmNSExeRJ26cSR1WJIzpkG2CW+OI6nDkjzrC1omIhICZZJPq5Iu4QeBV4CTgVe8eQmk2h9alFjHAcDXC+ZTo3p1hl98cc6yls2aMXLoMAD2HzjAyDvvoMNF/eh88UBmLVgAwFnDhrJ67dqcbXr1SGXxosXs3buXq4aPoGOHTiQnpvDhB/5fkddffZ3BFw/hwgv6cW6f8/g67Wt69Uhl0MDBtGsbw2WXXI5z/r62U048lTtG30nXzt04I6UjSxYv4bw+53HaKacx/oXxOfsc9+g4OnboREJcImPuuReAdevWEXN6e0aOuIbYmDjOPftc9u3bxzvT3mVR+iIuu/RykhKS2bdvX57jsHnTZpo2bQKAz+ejbds2AIW2Jysri9tuuY342AQS4hJ59plneebpZ9iwYQOpPXuT2rM3AJPenkx8bAJx7eO5fdQ/cvbXsF4k99w9hs4duzBv7rxjPY0FMjMaRjbk0IGsYtcRyjsPRgEznXMPel9tMwq4rYBy+5xzuidVRKQEOOcyzeyvwGeAD3jFObeqpPfTrl07fn19ON3f6g7AA7PvybO+dcbMoOppHN2Ij754Pd/ygpaJiJSlssqnVYkGRyw6s2oBOw42bd3CNTffxNKMDOJat+b5R8fR+LjCPwVfvfY/xLVpU+j68ZMmATD/nffI+PEHLhx5NUs+/JiL+vTh3c8/IyGxDb/++isbNmwgITGBO0bfSfezuvPShPHs2LGDTmd0pmevnv465s1n0dJ0IiMj+Trta5YuWcrSFUto1qwZZ3btzpxv59C5S2cAWjRvzuxvZ3HzjTcz/Ior+Xp2Gvv37ycuJp4RI0cw4/MZrF2zljnzvsU5R/++/Zk9azYtWrZg7Zq1THxzIi+Mf54hg4by7jvvMeySoTz/3PM89PCDJCYl5mvndTdcR7s2MXQ7sxtnn92bSy+/lIiICB64/8EC2/PG62/w44/rWLhoAWFhYWzbto3IyEiefPwpZsz8nKioKDZs2MDo20czb+FcGjZsyLl9zuOD9z+g74V92bt3L6effjr3jLk7qPNeXP73SvFvgArlgIl9gde86deAC0MYi4hIleGcm+6cO9U5d5Jz7r5QxyMiUlEpn0p5d83NNxGzbBlL9u4lZtkyrrm5aIMM33j/P+k48CLOHDoYgLlLFjPk/D8C0LrVibRo2oy1P62jf++zeW/G5wBMmzqNiwZcBMAXM77gkYcfISkhmV49Ujmw/wA///wzAD179SQyMjJnX8nJSTRv3pxq1aoRG9uedet+yll3/gXnA/4PJ1JSUqhbty6NGjUiIiKCHTt28MWML/hixkySE1NISepARkYGa9f474Ro1eoE4uJiAUhITOCnn36vtzB33DmauQvmkJrai0mTJnP+uX8M2J6ZM79kxNVX5Tx+kbtd2dIXptPtzG40atSIsLAwhgwZzDezvwH8dzf0v6hfEGcktEJ550G0c+5XAOfcr2aWf6QKvwgzSwcygQedc+8XVOhYB/cSERERERGpTJZmZPCvrCyaATdmZRGfkRGwfJuTT+KDmTNy5h/7xx1s2b49p/PAFfKpdbPoaCLrN2D56gymTpnGs88/6y/vHJOnTqJ167zjBSycv5DatWvlWRYeHp4z7fP5yMrMzLeuWrVqhIf/PhZGtWrVyMzMxDnHrbfdwlVXX5WnznXr1lEjT73V2Lcvk2CcdNJJnHTNSQy/ajjNoo9n69athbbHOXfUR0eyH8MoSERERKmMc1DSSvXOAzP7wsxWFvDqW4RqWjrnkoChwBNmdlJBhZxz451zSc65pEaN9H3fIiIiIiJStcW1bs1jPh8bgMd8PuJaBx7078yUDhw4cJCXp0zOWbZv//6c6c4JiUye/gkAa9atY/3/fuWUE/wDJA7o04dHX3yZnTt3EhPTDoDU3qk898xzOf84L1mytCSblyO1dyqvvvoae/bsAeCXX35h06ZNAbepU6cOu3fvLnDd9E+m58S8Zs0afD4fDRo0KLQ9qam9GP/iS2R6HR7btvm/caJO3d/3kdIhhdmzZrNlyxaysrKYPHkyXbt1PcaWl61S7TxwzvVyzrUr4PUBsNHMmgJ4Pws8u865Dd7PH4A0IL40YxYREREREakMnn90HCtiY4mvXZsVsbE8/2jgQYbNjLefeJJv0tNpd04fug8dwtV3jObe6/8OwFWDBpN1OIsOF/XjT7fewgv3/pPwGv47AS5MTWXyR9MZMHBATn2j7/gHhw4dIiEukbj28Yy5655SaWdq71QGDxlE187diI9NYPDFQwrtGMh22eWX8Zdr/1rggIlvvvEWp7dpR1JCMn++/Apem/gqPp+v0PZcceUVtGzZgoS4RBLjk5j0tn9siCuvGs4fz7uA1J69adq0KWPvG0tqz94kxicRFx/PBX0vKJXjUVos0O0Tpbpjs0eArbkGTIx0zt16RJmGwG/OuQNmFgXMBfo6574LVHdSUpJLT08vVlzdLvGf6LShacy9Ji3PuiMH+Op/5exC63n3ZX8vUuSXY9nW485ixSIi5VNUxPHF3tbMFnl3U1UYxc2ph6ePPOYBE0Wkcqtq+fS08Ag3ockJoQ5DKpHaLz8b1FcolpXifFWjlK2M7zNocmLeMRka1WweVD4N5YCJDwKpZrYGSPXmMbMkM3vZK9MGSDezZcBX+Mc8CNhxICIiIiIiIiIlK2QDJjrntgI9C1ieDlzpTc8BYso4NBERERERERHJJZR3HoiIiIiIiIhIBaDOAxEREREREREJSJ0HIiIiIiIiIhKQOg9EREREREREJCB1HoiIiIiIiAgATc5IyTP/xgfvc9P99wEwYcoU3vrow4Dbv/7q61z/t+tLLJ57x4zlhBatSEpIznnt2LEjX7lePVJZlL6oxPYL8OADD+WZ79blzBKpd/68+XTu2IWkhGRiTm/PvWPGFquep558it9++61EYgpGyL5tQURERERERCqO4RdfXOr7yMrKwufz5Vl23Q1/48abbiz1fR/poQceYtTtt+XMz/rm6xKp94o/D+etSW8RG9uerKwsMjL+Xax6nn7yGYYOG0qtWrWC3iYrK6tY+wLdeSAiIiIiIiJBuP/553jytVcBWLRyJWcM6E+PS4cx+rFxpPTvl1Nuw4ZfOf+c82nbui2jbrs9Z/mMz2fQtXM3UpI6MPjiIezZsweAU048lX+OvY/u3c5i2tR3gopl3759DBtyCQlxiQwdPIx9+/blrGtYLzJn+p1p7zL8z1cCsHHjRgb0H0hifBKJ8UnMnTMXgIv6DaBD8hnExsTx8viXAfjH7aPZt28fSQnJXHbJ5Xnqdc4x6tZRxLWPJz42gSmTpwLwddrX9OqRyqCBg2nXNobLLrkc51y+2Ddv2kzTpk0A8Pl8tG3bBoC9e/dy1fARdOzQieTEFD78wH+XR1ZWFrfdchvxsQkkxCXy7DPP8szTz7BhwwZSe/YmtWdvACa9PZn42ATi2sdz+6h/5Dke99w9hs4du7Bs6bKgjm9BdOeBiIiIiIhIJbRp6xZG3HkTS7/LIK5ta8aPHUfj46ICbrPvwAE6XTwgZ377zp2ce2b3fOWuuetOnrrrbs6Ii+OuJx7Ps275smUsWLSA8PBw2rWJ4S9/vZaaNWvywP0P8n+ff0rt2rV55OFHeeLxJ7njztEARESEkzbrqwJjeuqJp3nrzbcBaNiwITNmfs6LL7xIrVq1WLx0EcuXr6BDUoejHo+/X38j3c7syrR3p5KVlZXTefHShPFERkayb98+OnboRL+L+nH/A/fx/LPPk754Yb563nv3fZYtXcaiJels2bKFTh0607VbFwCWLlnK0hVLaNasGWd27c6cb+fQuUvnPNtfd8N1tGsTQ7czu3H22b259PJLiYiI4IH7H6T7Wd15acJ4duzYQaczOtOzV0/eeP0NfvxxHQsXLSAsLIxt27YRGRnJk48/xYyZnxMVFcWGDRsYffto5i2cS8OGDTm3z3l88P4H9L2wL3v37uX000/nnjF3k/F9xlGPU2F054GIiIiIiEglNOLOm2jUfBkvjN9Lo+bLGHHnTUfdpmZ4OHOmTMt5jb72L/nK7Ni1iz2/7eWMuDgALj733Dzrz+rRg/r16xMREUGbNqfx808/M3/efFZ/t5ozu3YnKSGZia9P5Oeffs7ZZuDFAwuN6bob/kb64oWkL17IjJmfAzB71jcMHTYEgPbtY4hpH3PUtqV9lcbVI68G/J/4169fH4Bnnn6WxPgkunTqyvr/rmftmrUB65nz7bcMGjwIn89HdHQ0Xbt1JX1hOgDJyUk0b96catWqERvbnnXrfsq3/R13jmbugjmkpvZi0qTJnH/uHwH4YsYXPPLwIyQlJNOrRyoH9h/g559/ZubMLxlx9VWEhfk/+4+MjMxXZ/rCdLqd2Y1GjRoRFhbGkCGD+Wb2Nzlt7X9Rv3zbFJXuPBAREREREamEln6XwQs3ZBEVBRcNyGLkiOJ/6pybI/+t+LmFh9fImfb5fGRmZuKc83+K/tbEArepXbt2keMws6Mu379/f8A6vk77mi9nfsnsb2dRq1YtevVIPeo2BT2KkC08PDxn2ufzkZWZWWC5k046iZOuOYnhVw2nWfTxbN26Fecck6dOonXr1vn2V1hbg4kpIiIi3zgSxaE7D0RERERERCqhuLateWeajy1b4J1pPuLatj76RkFoWK8+dWrVZsFy//Pz0/7v/466TYczOjB3zlzWrvV/qv/bb7/x738Xb6BAgK7duvD2W5MAWLlyFSuWr8hZFx3dmNWrV3P48GE+eP+DnOVn9TiLF194EfCPI7Br1y527txJgwYNqFWrFt9//z3z583PKV+9enUOHTqUb99dunZl6hT/ow+bN2/mm9nfkJySHHTs0z+ZnvPP/po1a/D5fDRo0IDU3qk898xzOeuWLFkKQGpqL8a/+BKZXkfEtm3bAKhTtw67d+8GIKVDCrNnzWbLli1kZWUxefJkunbrGnRMwVDngYiIiIiISCU0fuw4Nq+PZeSI2mxeH8v4seNKrO5n7xnDdfeOocelw3DOUa9unYDlGzVqxMuvvMSlwy4jIS6RLp26Bv38/VNPPJ3nqxrXrVvH1SOvZs+ePSTEJTLukXF5/nn/5/330e+CfvTudXbOwIQAjz0xjrSvviY+NoEOyWfw3arvOLvP2WRlZpIQl8g9d42hwxm/j50w/KrhJMQl5gyYmO3Cfn2JaR9DYnwSvXudzf0P3k+TJk0I1ptvvMXpbdqRlJDMny+/gtcmvorP52P0Hf/g0KFDJMQlEtc+njF33QPAFVdeQcuWLUiISyQxPolJb/s7Ta68ajh/PO8CUnv2pmnTpoy9byypPXuTGJ9EXHw8F/S9IOiYgmGBbm+oqJKSklx6enqxtu12if9EpA1NY+41aXnWtc6YmWe+/5WzC63n3Zf9vTyRX45lW487ixWLiJRPURHHF3tbM1vknEsqwXBKXXFz6uHpI+n+VncAHph9T551R+ZTEamaqlo+PS08wk1ockKow5BKpPbLz3JydHRI9r3nt9+o431F4LgJL7Nxyxaevv9ODrdoFZJ4JDgZ32fQ5MS8YyY0qtk8qHyqMQ9ERERERESkSD6bNYtxr7xMZmYWLZo15YV7/xnqkKSUqfNAREREREREiuSiPn24qE+fUIchZUhjHoiIiIiIiIhIQOo8EBEREREREZGA1HkgIiIiIiIiIgGp80BEREREREREAlLngYiIiIiIiABQPz6WThcPIKV/Py69+UZ+27evwHIX/eUaduzaVeT6d+zYwQvPv1Dk7V595VXiYxNIiEskrn08H37wYZntW/zUeSAiIiIiIiIA1AwPZ86UaSx49z1qVK/OhKlT8qx3znH48GHeefZ5GtSrV+T6/f/Av1ikbdavX8+DDzxI2qyvWLx0Ed/MmU1M+5gy2XdWVlaR91NZqfNARERERERE8ukUn8gP//0vP/3yC4kXXsDf7/snXQZdzPr//Y/TzzmbLdu3c+fjj/HS5Ek529w7ZiyPP/Y4e/bs4ezUs0lJ6kB8bELOnQKjb7+DH/7zA0kJyYy6dRQA4x4dR8cOnUiIS2TMPffmi2Pzps3UqVuXOnXqAFCnTh1atWoFwH/+8x/OP+d8OiSfwVln9uD7778HYOPGjQzoP5DE+CQS45OYO2duvn075xh16yji2scTH5vAlMlTAfg67WtSe/bm0mGXER+bUHoHuIIJC3UAIiIiIiIiUvI2bd3CpTffzfKM72jfui0THx1D4+Oigto2MzOTz7+dTWrnLgCsWbeO5+8dy+Oj78hTbkCfc7jtkYe4atBgAKZNncbH0z8iIiKCqe9MpV69emzZsoWunbryxwv+yH0P/JNVq1aRvnghADM+n8HaNWuZM+9bnHP079uf2bNm07Vb15x9tI9tT3Tjxpx6UmvO6nEWF/bry/l/PB+Aa0deyzPPPcMpp5zCgvkLuO6v1/P5F5/x9+tvpNuZXZn27lSysrLYs2dPvn2/+857LFu6jEVL0tmyZQudOnSmazd/excuWMiS5YtzOilEnQciIiIiIiKV0qU3382CZclkZr3NgmWPcenNd/PZv54NuM2+AwfodPEAADrFJ3BZv/78umkTLZs2I6V9bL7ysW3asHnbNn7dtIndv+ymYcOGtGzZkkOHDnHn6DuZPfsbqlWrxi+/bGDjxo35tv9ixhd8MWMmyYkpAOzds4e1a9bm6Tzw+Xx8/OnHpC9M58svv+KWm25h8eIl3HjT35k7Zx5DBg3NKXvgwAEA0r5K41+vvZKzff369dm+fXuefc/59lsGDR6Ez+cjOjqart26kr4wnXr16pGckqyOgyOo80BERERERKQSWp7xHZlZbwPNyMy6keUZ+f/5P1L2mAdHqlWzZqHbXNgrlfdnzGDrnm1cPGggAG+/+TabN29h/sJ5VK9enVNOPJX9+/fn29Y5x6233cJVV18VMC4zIzklmeSUZHr16smVw6/ihr9fT4MGDXLuJCgq51yh62rXqlWsOiszjXkgIiIiIiJSCbVv3ZYw32PABsJ8j9G+ddtS2c+APucw7bNPeffTz+h/UX8Adu7aSePGjahevTppX6Xx008/AVC3bl327N6Ts21q71ReffU19uzxL/vll1/YtGlTnvo3bNjAksVLcuaXLV3GH1q2pF69epzQ6gSmTX0H8HcGLFu2HICzepzFiy/4B0fMyspi165d+fbdpWtXpk7xP9awefNmvpn9DckpySV8dCoPdR6IiIiIiIhUQhMfHUNK7ELq1I4lJXYhEx8dUyr7aXPyyezZu5dm0dE0bdoUgCFDh7Bo0WLOSOnI229NovVprQE47rjj6NipI3Ht4xl16yhSe6cyeMggunbuRnxsAoMvHsLu3bvz1H/o0CFuu3UU7drGkJSQzNQpUxn3xDgAXpv4Kq++8i8S45OIjYnjow8/AuCxJ8aR9tXXxMcm0CH5DL5b9V2+fV/Yry8x7WNIjE+id6+zuf/B+2nSpEmpHKPKwALdqlFRJSUlufT09GJt2+0S/0ihaUPTmHtNWp51rTNm5pnvf+XsQut592X/MzqRX45lW487ixWLiJRPURHHF3tbM1vknEsqwXBKXXFz6uHpI+n+VncAHph9T551R+ZTEamaqlo+PS08wk1ockKow5BKpNZLT3NydBPMLNShAFCzfnUOt9A4AeWVc45/Z6yhyYkN8yxvVLN5UPlUdx6IiIiIiIhUQId/+i87Mg8FfHZfBPwdB9u3bad6uK/YdWjARBERERERkQpo/9MvsOlvI9nyhxaYhf5z4eq7fLi9B0MdhhTIqB7uo0F07WLXELLOAzMbCNwDtAFSnHMF3hNrZn2AJwEf8LJz7sEyC1JEpBIJNu+KiEhgyqdSbuzcxf5/PhzqKHLE9GnKwSffCHUYUkpC2T21EugPzCqsgJn5gGeBc4C2wBAzK50hQkVEKr+j5l0REQmK8qmIVDkhu/PAObcaONrgHinAWufcD17ZSUBf4LtSD1BEpJIJMu+KiMhRKJ+KSFUU+gdjAjse+G+u+fXeMhEREREREREpI6V654GZfQEU9EWZo51zHwRTRQHLChxK1MxGACO82T1mlhFclPlEAVt8bxawpmbzoCtplGf7F4sZSpmIAraEOogypjZXDeW1zX8ozcpLIO9m11MSOTUKXtwC0OXINUXIpxVMeX3flaaq1uaq1l4ov22uiPn0QJefM1aWRHwlqDyeX8UUnPIX0/iMKMY3L18x+ZW/Y1W+Ygoqn5Zq54FzrtcxVrEeaJFrvjmwoZB9jQfGH+P+MLP0ivadwceiqrUX1Oaqoiq2GUok72bXc8w5tSqeA7W58qtq7YWq2WYonXxaHo+lYgqOYgpOeYwJymdc5TGmoynvjy0sBE4xs1ZmVgMYDHwY4phEREREREREqpSQdR6YWT8zWw90BD4xs8+85c3MbDqAcy4T+CvwGbAamOKcWxWqmEVEKrLC8q6IiBSN8qmIVEWh/LaF94D3Cli+ATg31/x0YHoZhnbMjz5UMFWtvaA2VxVVsc0BFZZ3S1FVPAdqc+VX1doLVbPNAR1DPi2Px1IxBUcxBac8xgTlM67yGFNA5lyB4w+KiIiIiIiIiADlf8wDEREREREREQkxdR54zKyPmWWY2VozGxXqeEqSma0zsxVmttTM0r1lkWY2w8zWeD8besvNzJ7yjsNyM0sIbfTBMbNXzGyTma3MtazIbTSzy73ya8zs8lC0JRiFtPceM/vFO89LzezcXOtu99qbYWZn51peYd73ZtbCzL4ys9VmtsrMrveWV9rzXJFVpPdWUSifVs7fs6qWU5VPy46ZDfSO8WEzK3RU9bJ87xR2ngsol5Xr/V8qA5Yfrd1mFm5mk731883shNKIo4gx/cnMNuc6NleWQUz5ctQR68v8ehNETN3NbGeu43RXKcdTYF47okyZHqcgYyrT43TMnHNV/gX4gP8AJwI1gGVA21DHVYLtWwdEHbHsYWCUNz0K9v7JsgAADHZJREFUeMibPhf4FDDgDGB+qOMPso3dgARgZXHbCEQCP3g/G3rTDUPdtiK09x7g5gLKtvXe0+FAK++97qto73ugKZDgTdcF/u21rdKe54r6qmjvrSK2Tfm0Ev6eVbWcqnxapse6DdAaSAOSCilTpu+dws5zAeX2lPKxOWq7gWuBF7zpwcDkchDTn4Bnyvh9lC9HHbG+zK83QcTUHfi4DI9RgXktlMcpyJjK9Dgd60t3HvilAGudcz845w4Ck4C+IY6ptPUFXvOmXwMuzLX8dec3D2hgZk1DEWBROOdmAduOWFzUNp4NzHDObXPObQdmAH1KP/qiK6S9hekLTHLOHXDO/Qisxf+er1Dve+fcr865xd70bvzfwHI8lfg8V2AV6r1VApRPK/jvWVXLqcqnZcc5t9o5l3GUYmX93insPJe1YNqdO9ZpQE8zsxDHVOaCyFFlfr0pYt4sdQHyWm5lepyCjKlCUeeB3/HAf3PNr6eCn9gjOOBzM1tkZiO8ZdHOuV/B/8YGGnvLK9OxKGobK0Pb/+rdhvVKrtsQK117vdsW44H5VM3zXN5V5mOsfFq1fs8qfU5VPi0Xyvo4FnaejxRhZulmNs/MSqODIZh255Rx/q9w3wkcVwqxFCUmgIu83DDNzFqUYjzBKq+/ix3NbJmZfWpmp5fVTo/Ia7mF7DgFiAlCdJyKI2Rf1VjOFNSDWZm+hqKzc26DmTUGZpjZ9wHKVvZjAYW3saK3/XlgLP6YxwLjgCsovF0FdR6W+/aaWR3gHeAG59yuAB9AVNbzXBFU5mOsfJpXZf49q/Q5Vfm0ZJjZF0CTAlaNds59EEwVBSw7puMYKKYiVNPSy3cnAl+a2Qrn3H+OJa4jBNPusn6PBbO/j4C3nXMHzGwk/jsjepRiTMEoj7+Li4E/OOf2mH/MmPeBU0p7p0fmtSNXF7BJqR+no8QUkuNUXLrzwG89kLvXsDmwIUSxlDjn3Abv5yb830mcAmzMvk3H+7nJK16ZjkVR21ih2+6c2+icy3LOHQZewn+eoRK118yq40++bzrn3vUWV6nzXEFU2mOsfFp1fs8qe05VPi05zrlezrl2BbyC6TiAUjiOR4mpsPN8ZB3Z+e4H/GM2xB9LTAUIpt05ZcwsDKhP6d4qf9SYnHNbnXMHvNmXgMRSjCdY5e530Tm3yzm3x5ueDlQ3s6jS3GcheS23Mj9OR4spFMfpWKjzwG8hcIqZtTKzGvgHZCmVUWXLmpnVNrO62dNAb2Al/vZlj4p8OZB9gfsQuMwbjfQMYGf2rW0VUFHb+BnQ28waeren9vaWVQhHPLPVD/95Bn97B5t/xOJW+HszF1DB3vfeM44TgNXOucdyrapS57mCqFDvrWApn1at37PKnFOVT8udsn7vFHaec3jnNNybjgI6A9+VcBzBtDt3rAOAL51zpflJ8VFjOiI3XID/OfZQK3fXGzNr4uUazCwF//+dW0txf4XltdzK9DgFE1NZH6dj5srBqI3l4YV/9M1/4x9hdXSo4ynBdp2If6TYZcCq7Lbhf15sJrDG+xnpLTfgWe84rKCQkYHL2wt4G/gVOIS/V3F4cdqI/5bUtd7rz6FuVxHbO9Frz3L8ybFprvKjvfZmAOfkWl5h3vdAF/y3li0Hlnqvcyvzea7Ir4r03ipCm5RPK+nvWVXLqcqnZXqs+3nvqQPARuAzb3kzYHoo3jsBznMS8LI33ck718u8n8NLKZZ87QbuBS7wpiOAqd77awFwYhmcs6PF9AD+a8Ay4CvgtDKIqaAcNRIY6a0v8+tNEDH9Nddxmgd0KuV4CstrITtOQcZUpsfpWF/mBS0iIiIiIiIiUiA9tiAiIiIiIiIiAanzQEREREREREQCUueBiIiIiIiIiASkzgMRERERERERCUidByIiIiIiIiISkDoPRERERERERCQgdR5IhWVme0IdQzDMbJ2ZRXnTc7yfJ5jZ0NBGJiLip3wqIlWFmTUws2tDHMNAM1ttZl+FMo5gZF8fzKyZmU3zpuPM7NzQRiahoM4DKVfMr9K+L51znbzJE4Ai/7FrZr4SDUhEKi3l08CUT0WqrAZASDsPgOHAtc65s0IcR9CccxuccwO82TigSJ0Hlf2aVFXoBErIeZ8arTaz54DFQJaZ3Wdmy8xsnplFe+VamdlcM1toZmODqPcWr+xyMxvjLUv25iPMrLaZrTKzdmbW3cxmmdl7Zvadmb2QneDMrLe338VmNtXM6njL15nZGG/5CjM7zVt+nJl9bmZLzOxFwHLFlP3p3oNAVzNbamZ/N7M/mdkzucp9bGbds7cxs3vNbD7Q0cwSzexrM1tkZp+ZWdNjPgkiUikonyqfishRPQic5OWMqWbWN3uFmb1pZhd4eeQDM/s/M8sws7tzlbnEzBZ4279oAToizWyIl9NWmtlD3rK7gC7AC2b2SCHb+czskVx592pveT8z+8L8mprZv82sSXHi9fJh0NcH7/qy0sxqAPcCg7w6B5nZPWZ2c66yK73yR16TWhR2HZCKQZ0HUl60Bl53zsV78/Occ7HALOAqb9mTwPPOuWTgf4EqM7PewClACv7e0UQz6+acWwh8CPwTeBh4wzm30tssBbgJiAFOAvqb//bYO4BezrkEIB24MdeutnjLnweyk+bdwDdeWz4EWhYQ4ihgtnMuzjn3+FGOTW1gpXOuAzAfeBoY4JxLBF4B7jvK9iJStSifFk75VERGAf9xzsUBzwB/BjCz+kAnYLpXLgUYhj/vDTSzJDNrAwwCOnvbZ3ll8jGzZsBDQA+vjmQzu9A5dy/+/DfMOXdLITEOB3Z6OToZuMrMWjnn3sOfs/8CvATc7ZzLzuFFjbc2xbg+OOcOAncBk728O7mQNmTLfU3aS+DrgJRzYaEOQMTzk3Nunjd9EPjYm14EpHrTnYGLvOmJ+BNyYXp7ryXefB38f/zOwt9buhDYD1yXa5sFzrkfAMzsbfy9wvuBtsC3ZgZQA5iba5t3c8XZ35vulj3tnPvEzLYHiDMYWcA73nRroB0ww4vHB/x6jPWLSOWifFo45VMRyeGc+9rMnjWzxvhzzTvOuUwvJ8xwzm0FMLN38eexTCARWOiVqQlsKqT6ZCDNObfZq+NN/Dnt/SBC6w20N7PsxwTq48+7PwJ/A1bi/8f/7VzbFDXekrg+BCP3NekMAl8HpJxT54GUF3tzTR9yzjlvOou871NHcAx4wDn3YgHrIvH/8VsdiMi17yPrdl49M5xzQwrZz4FjjDNbJnnvBIrINb3fOZflTRuwyjnXsYj1i0jVoXyqfCoiwZuI/9P4wcAVuZYXlsdec87dHkS9dvQiAbf9m3PuswLWHQ8cBqLNrJpz7nCu+HI7WrwlcX3IFijv5r4mHe06IOWcHluQiuRb/IkdCrlFLJfPgCvs9+dpj/d6lQHGA3cCb5K3NzXFe86rGv5bvL4B5gGdzexkr55aZnbqUfY9Kzs+MzsHaFhAmd1A3Vzz64A4M6tmZi3w33pWkAygkZl19OqvbmanHyUeEZEjKZ8qn4pUVUfmjFeBGwCcc6tyLU81s0gzqwlciD9vzgQGZOdAb/0fCtnPfOBMM4vyxhkYAnwdZIyfAdeYWXVvP6eaf2yZMOBf+AeJXU3eW/6PNd5swVwfCsq7Cd4+EoBWhWxXnOuAlCO680AqkuuBt8zsen6/7bRAzrnPvee85nq3Re0BLjGzPkCmc+4tL5HPMbMe+Htw5+IfRCcG/x+s7znnDpvZn4C3zSzcq/4O4N8Bdj/GK78Y/0Xi5wLKLAcyzWwZ/ovWE/hvRVuB/1a0xYW066B3C9tT3rN5Yd62qwoqLyJSCOVT5VORKsk5t9XMvjWzlcCnzrlbzGw1+R8n+Ab/XQknA28559IBzOwO4HOvc/QQ/vEHfipgP7+a2e3AV/g/cZ/unPsgyDBfxv9NMovNn3g34+8QuAn/GC+zzWwp/scRPimJeHMJ5vrwFTDKi+EBr9xl2TFRSF53zm0uxnVAyhH7/W4VkarL/CNx3+ycOz/UsYiIVGTKpyJSkZhZLfydjQnOuZ3esj8BSc65v4YytmBVtHil4tJjCyIiIiIiUuWYWS/ge+Dp7I4DESmc7jyQCs3MYvDfopXbAe9ruEREJEjKpyIixWNm84HwIxZf6pxbcZTtzib/txn86JzrV5LxiZQUdR6IiIiIiIiISEB6bEFEREREREREAlLngYiIiIiIiIgEpM4DEREREREREQlInQciIiIiIiIiEpA6D0REREREREQkoP8Hz7vA6m7HLXgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x720 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the decision surface for pair-wise features, using\n",
    "# the best estimator found by GridSearchCV\n",
    "best_n_estimators = n_estimators=gs_rfc.best_params_['n_estimators']\n",
    "best_max_depth = n_estimators=gs_rfc.best_params_['max_depth']\n",
    "\n",
    "plot_decision_surface(RandomForestClassifier(n_estimators=best_n_estimators,\n",
    "                                             max_depth=best_max_depth),\n",
    "                      n_classes, train_X, train_y,\n",
    "                      \"Decision surface of a Random Forest classifier using paired features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.04444444 0.01111111 0.         0.94444444]\n",
      " [0.03333333 0.1        0.         0.86666667]\n",
      " [0.11666667 0.04166667 0.80833333 0.03333333]\n",
      " [0.8        0.1        0.         0.1       ]\n",
      " [0.         0.08       0.92       0.        ]\n",
      " [0.56666667 0.16666667 0.00833333 0.25833333]\n",
      " [0.         0.3        0.5        0.2       ]\n",
      " [0.71666667 0.16666667 0.00833333 0.10833333]\n",
      " [0.71666667 0.16666667 0.00833333 0.10833333]\n",
      " [0.03333333 0.1        0.         0.86666667]\n",
      " [0.         0.2        0.         0.8       ]\n",
      " [0.04444444 0.89111111 0.02       0.04444444]\n",
      " [0.24       0.4        0.1        0.26      ]\n",
      " [0.         0.2        0.         0.8       ]\n",
      " [0.03333333 0.1        0.         0.86666667]\n",
      " [0.         0.9        0.         0.1       ]\n",
      " [0.         0.95       0.         0.05      ]\n",
      " [0.         0.         1.         0.        ]\n",
      " [0.06666667 0.16666667 0.00833333 0.75833333]\n",
      " [0.94444444 0.01111111 0.         0.04444444]\n",
      " [0.365      0.125      0.         0.51      ]\n",
      " [0.71846154 0.00769231 0.01538462 0.25846154]\n",
      " [0.8        0.1        0.         0.1       ]\n",
      " [0.01666667 0.34166667 0.00833333 0.63333333]\n",
      " [0.         0.8        0.         0.2       ]\n",
      " [0.         0.8        0.         0.2       ]\n",
      " [0.12692308 0.70769231 0.         0.16538462]\n",
      " [0.3        0.7        0.         0.        ]\n",
      " [0.21666667 0.04166667 0.70833333 0.03333333]\n",
      " [0.         0.1        0.8        0.1       ]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.57      0.67      0.62         6\n",
      "          1       0.50      0.40      0.44        10\n",
      "          2       1.00      1.00      1.00         6\n",
      "          3       0.44      0.50      0.47         8\n",
      "\n",
      "avg / total       0.60      0.60      0.60        30\n",
      "\n",
      "[[4 1 0 1]\n",
      " [2 4 0 4]\n",
      " [0 0 6 0]\n",
      " [1 3 0 4]]\n"
     ]
    }
   ],
   "source": [
    "# Train our RFC using the best parameters found\n",
    "# by GridSearchCV\n",
    "rfc = RandomForestClassifier(random_state=42, max_depth=7, n_estimators=10)\n",
    "\n",
    "# fit\n",
    "rfc.fit(train_X, train_y)\n",
    "\n",
    "# get the prediction probabilities\n",
    "# import pdb; pdb.set_trace()\n",
    "pred_prob_y = rfc.predict_proba(test_X)\n",
    "print(pred_prob_y)\n",
    "\n",
    "# predict\n",
    "pred_y = rfc.predict(test_X)\n",
    "\n",
    "# evaluation metrics\n",
    "print(classification_report(test_y, pred_y))\n",
    "cm = confusion_matrix(test_y, pred_y)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## XGBoost Classifier with GridSearchCV\n",
    " \n",
    "Finally, let's try XGBoost on our dataset, to see how well it does.\n",
    "\n",
    "### Setup\n",
    "\n",
    "XGBoost is a separate library from sklearn (https://xgboost.readthedocs.io/en/latest/build.html)\n",
    "\n",
    "```\n",
    "conda install -c anaconda py-xgboost\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### XGBoost and Scikit-learn\n",
    "\n",
    "XGBoost has its own API, but includes an sklearn wrapper for convenience.\n",
    "\n",
    "https://github.com/dmlc/xgboost/blob/master/demo/guide-python/sklearn_examples.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 16 candidates, totalling 48 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.6222222222222222\n",
      "Best parameters: {'max_depth': 8, 'n_estimators': 30}\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.17      0.17      0.17         6\n",
      "          1       0.36      0.50      0.42        10\n",
      "          2       1.00      0.50      0.67         6\n",
      "          3       0.29      0.25      0.27         8\n",
      "\n",
      "avg / total       0.43      0.37      0.38        30\n",
      "\n",
      "[[1 4 0 1]\n",
      " [2 5 0 3]\n",
      " [1 1 3 1]\n",
      " [2 4 0 2]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n",
      "[Parallel(n_jobs=1)]: Done  48 out of  48 | elapsed:    0.8s finished\n",
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\preprocessing\\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.\n",
      "  if diff:\n"
     ]
    }
   ],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "# Use GridSearchCV to select the optimum hyperparameters\n",
    "gs_xgb = GridSearchCV(xgb.XGBClassifier(), {'max_depth': [2, 4, 6, 8],\n",
    "                                            'n_estimators': [5, 10, 20, 30]},\n",
    "                      verbose=1)\n",
    "\n",
    "gs_xgb.fit(train_X, train_y)\n",
    "\n",
    "# select the best estimator\n",
    "print('Best score:', gs_xgb.best_score_)\n",
    "print('Best parameters:', gs_xgb.best_params_)\n",
    "\n",
    "# predict\n",
    "pred_y = gs_xgb.predict(test_X)\n",
    "\n",
    "# evaluation metrics\n",
    "print(classification_report(test_y, pred_y))\n",
    "cm = confusion_matrix(test_y, pred_y)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 16 candidates, totalling 48 fits\n",
      "Best score: 0.6555555555555556\n",
      "Best parameters: {'max_depth': 8, 'n_estimators': 30}\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.20      0.17      0.18         6\n",
      "          1       0.36      0.40      0.38        10\n",
      "          2       1.00      0.83      0.91         6\n",
      "          3       0.33      0.38      0.35         8\n",
      "\n",
      "avg / total       0.45      0.43      0.44        30\n",
      "\n",
      "[[1 4 0 1]\n",
      " [2 4 0 4]\n",
      " [0 0 5 1]\n",
      " [2 3 0 3]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  48 out of  48 | elapsed:    1.8s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "# Use GridSearchCV to select the optimum hyperparameters\n",
    "gs_gb = GridSearchCV(GradientBoostingClassifier(random_state=42),\n",
    "                     {'max_depth': [2, 4, 6, 8],\n",
    "                      'n_estimators': [5, 10, 20, 30]},\n",
    "                      verbose=1)\n",
    "\n",
    "gs_gb.fit(train_X, train_y)\n",
    "\n",
    "# select the best estimator\n",
    "print('Best score:', gs_gb.best_score_)\n",
    "print('Best parameters:', gs_gb.best_params_)\n",
    "\n",
    "# predict\n",
    "pred_y = gs_gb.predict(test_X)\n",
    "\n",
    "# evaluation metrics\n",
    "print(classification_report(test_y, pred_y))\n",
    "cm = confusion_matrix(test_y, pred_y)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Workshop: Regression with Decision Trees\n",
    "\n",
    "In this workshop, we will apply decision tree-related algorithms to a multi-variate linear regression problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Goal\n",
    "\n",
    "Train a regression model to predict the `Lifetime Post Total Reach` of a Facebook post to seven input features (category, page total likes, type, month, hour, weekday, paid)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Dataset\n",
    "\n",
    "#### Facebook metrics Data Set\n",
    "\n",
    "https://archive.ics.uci.edu/ml/datasets/Facebook+metrics\n",
    "\n",
    "The data is related to posts' published during the year of 2014 on the Facebook's page of a renowned cosmetics brand.\n",
    "\n",
    "1. Download dataset from the above URL\n",
    "2. Extract the folder and note the path for use in read_csv below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Page total likes</th>\n",
       "      <th>Type</th>\n",
       "      <th>Category</th>\n",
       "      <th>Post Month</th>\n",
       "      <th>Post Weekday</th>\n",
       "      <th>Post Hour</th>\n",
       "      <th>Paid</th>\n",
       "      <th>Lifetime Post Total Reach</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>139441</td>\n",
       "      <td>Status</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>50128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Page total likes    Type  Category  Post Month  Post Weekday  Post Hour  \\\n",
       "0            139441   Photo         2          12             4          3   \n",
       "1            139441  Status         2          12             3         10   \n",
       "2            139441   Photo         3          12             3          3   \n",
       "3            139441   Photo         2          12             2         10   \n",
       "4            139441   Photo         2          12             2          3   \n",
       "\n",
       "   Paid  Lifetime Post Total Reach  \n",
       "0   0.0                       2752  \n",
       "1   0.0                      10460  \n",
       "2   0.0                       2413  \n",
       "3   1.0                      50128  \n",
       "4   0.0                       7244  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = pd.read_csv('../data/Facebook_metrics/dataset_Facebook.csv',\n",
    "                 delimiter=';',\n",
    "                 usecols=['Page total likes', 'Type', 'Category',\n",
    "                         'Post Month', 'Post Weekday', 'Post Hour', 'Paid',\n",
    "                         'Lifetime Post Total Reach'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Prepare Dataset\n",
    "\n",
    "### Handle NaN values\n",
    "\n",
    "Check for NaN values and decide what to do with them.\n",
    "\n",
    "### Convert string columns\n",
    "\n",
    "There are two options for the `Type` column.\n",
    "  1. Convert to dummy columns.\n",
    "  2. Encode the labels to numbers.\n",
    "\n",
    "Either option is fine because the number of labels is small (4). Since we've demonstrated label encoding, we'll try the dummy column approach.\n",
    "\n",
    "### Balance dataset\n",
    "\n",
    "This is a regression task, so we don't need to balance the `Lifetime Post Total Reach` output, because it is a continuous variable.\n",
    "- Instead, we'll check that the discrete columns (`Type`, `Category`, `Paid`) values are balanced.\n",
    "- It's less critical, but a good idea to also check the `Post Month`, `Post Weekday` and `Post Hour` columns, just in case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handle NaN values\n",
    "\n",
    "Check for NaN values and decide what to do with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 500 entries, 0 to 499\n",
      "Data columns (total 8 columns):\n",
      "Page total likes             500 non-null int64\n",
      "Type                         500 non-null object\n",
      "Category                     500 non-null int64\n",
      "Post Month                   500 non-null int64\n",
      "Post Weekday                 500 non-null int64\n",
      "Post Hour                    500 non-null int64\n",
      "Paid                         499 non-null float64\n",
      "Lifetime Post Total Reach    500 non-null int64\n",
      "dtypes: float64(1), int64(6), object(1)\n",
      "memory usage: 31.3+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Page total likes</th>\n",
       "      <th>Type</th>\n",
       "      <th>Category</th>\n",
       "      <th>Post Month</th>\n",
       "      <th>Post Weekday</th>\n",
       "      <th>Post Hour</th>\n",
       "      <th>Paid</th>\n",
       "      <th>Lifetime Post Total Reach</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>81370</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4188</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Page total likes   Type  Category  Post Month  Post Weekday  Post Hour  \\\n",
       "499             81370  Photo         2           1             4          4   \n",
       "\n",
       "     Paid  Lifetime Post Total Reach  \n",
       "499   NaN                       4188  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Convert string columns\n",
    "\n",
    "Convert `Type` to dummy columns and append to DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Photo</th>\n",
       "      <th>Status</th>\n",
       "      <th>Video</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link  Photo  Status  Video\n",
       "0     0      1       0      0\n",
       "1     0      0       1      0\n",
       "2     0      1       0      0\n",
       "3     0      1       0      0\n",
       "4     0      1       0      0"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_type_dummies = pd.get_dummies(df['Type'])\n",
    "df_type_dummies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Page total likes</th>\n",
       "      <th>Type</th>\n",
       "      <th>Category</th>\n",
       "      <th>Post Month</th>\n",
       "      <th>Post Weekday</th>\n",
       "      <th>Post Hour</th>\n",
       "      <th>Paid</th>\n",
       "      <th>Lifetime Post Total Reach</th>\n",
       "      <th>Link</th>\n",
       "      <th>Photo</th>\n",
       "      <th>Status</th>\n",
       "      <th>Video</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2752</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>139441</td>\n",
       "      <td>Status</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10460</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2413</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>50128</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>139441</td>\n",
       "      <td>Photo</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7244</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Page total likes    Type  Category  Post Month  Post Weekday  Post Hour  \\\n",
       "0            139441   Photo         2          12             4          3   \n",
       "1            139441  Status         2          12             3         10   \n",
       "2            139441   Photo         3          12             3          3   \n",
       "3            139441   Photo         2          12             2         10   \n",
       "4            139441   Photo         2          12             2          3   \n",
       "\n",
       "   Paid  Lifetime Post Total Reach  Link  Photo  Status  Video  \n",
       "0   0.0                       2752     0      1       0      0  \n",
       "1   0.0                      10460     0      0       1      0  \n",
       "2   0.0                       2413     0      1       0      0  \n",
       "3   1.0                      50128     0      1       0      0  \n",
       "4   0.0                       7244     0      1       0      0  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Append to existing dataset\n",
    "df = pd.concat([df, df_type_dummies], axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Balance dataset\n",
    "\n",
    "Check that these discrete columns are balanced\n",
    "- `Type`\n",
    "- `Category`\n",
    "- `Paid`\n",
    "\n",
    "Optionally, check that these discrete columns are balanced\n",
    "- `Post Month`\n",
    "- `Post Weekday`\n",
    "- `Post Hour`\n",
    "\n",
    "Balanced means that the counts for each discrete values are not overly biased to one or two values. As a rule of thumb, overly means 10x or larger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Photo     425\n",
       "Status     45\n",
       "Link       22\n",
       "Video       7\n",
       "Name: Type, dtype: int64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Detect if the `Type` column is balanced\n",
    "df['Type'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Instead of dropping all the columns, let's reduce just the `Photo` rows by a random sample of 90 entries\n",
    "num_photo_rows = 90\n",
    "photos_df = df.loc[df.Type == 'Photo']\n",
    "\n",
    "indices_to_keep = photos_df.sample(num_photo_rows).index # random sample the indices\n",
    "indices_to_drop = list(set(photos_df.index) - set(indices_to_keep))\n",
    "\n",
    "balanced_df = df.drop(df.index[indices_to_drop])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Photo     90\n",
       "Status    45\n",
       "Link      22\n",
       "Video      7\n",
       "Name: Type, dtype: int64"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df['Type'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    64\n",
       "2    56\n",
       "3    44\n",
       "Name: Category, dtype: int64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df['Category'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    125\n",
       "1.0     39\n",
       "Name: Paid, dtype: int64"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df['Paid'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 164 entries, 1 to 495\n",
      "Data columns (total 12 columns):\n",
      "Page total likes             164 non-null int64\n",
      "Type                         164 non-null object\n",
      "Category                     164 non-null int64\n",
      "Post Month                   164 non-null int64\n",
      "Post Weekday                 164 non-null int64\n",
      "Post Hour                    164 non-null int64\n",
      "Paid                         164 non-null float64\n",
      "Lifetime Post Total Reach    164 non-null int64\n",
      "Link                         164 non-null uint8\n",
      "Photo                        164 non-null uint8\n",
      "Status                       164 non-null uint8\n",
      "Video                        164 non-null uint8\n",
      "dtypes: float64(1), int64(6), object(1), uint8(4)\n",
      "memory usage: 12.2+ KB\n"
     ]
    }
   ],
   "source": [
    "balanced_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Label Encode\n",
    "type_le = LabelEncoder()\n",
    "balanced_df.Type = type_le.fit_transform(balanced_df.Type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Select Features\n",
    "\n",
    "Select the features X and y from `balanced_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(164, 11) (164,)\n"
     ]
    }
   ],
   "source": [
    "X = balanced_df.loc[:, balanced_df.columns != 'Lifetime Post Total Reach']\n",
    "y = balanced_df['Lifetime Post Total Reach']\n",
    "\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Train the Decision Tree Regressor\n",
    "\n",
    "1. Shuffle and split the data set into train and test\n",
    "2. Train a `DecisionTreeRegressor`\n",
    "\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE 0.46014398138632684 R2 0.23677646873665914\n",
      "[[ 0.16259957]\n",
      " [-0.02338074]\n",
      " [-0.00974091]\n",
      " [ 0.14830979]\n",
      " [-0.14958954]\n",
      " [-0.39499977]\n",
      " [ 0.90803209]\n",
      " [-0.29069605]\n",
      " [-0.13537843]\n",
      " [ 0.30993971]\n",
      " [-0.37518366]\n",
      " [-0.1005316 ]\n",
      " [ 1.87107439]\n",
      " [ 0.0029524 ]\n",
      " [-0.13101845]\n",
      " [-0.2212297 ]\n",
      " [ 0.33621308]\n",
      " [-0.16939217]\n",
      " [ 0.37700316]\n",
      " [-0.3654733 ]\n",
      " [-0.12667529]\n",
      " [-0.33229481]\n",
      " [-0.34773364]\n",
      " [-0.25913781]\n",
      " [ 2.06732757]\n",
      " [ 0.63305615]\n",
      " [-0.19339233]\n",
      " [-0.15913986]\n",
      " [ 0.23247   ]\n",
      " [ 0.31170553]\n",
      " [ 0.31331977]\n",
      " [-0.41629308]\n",
      " [ 0.04381023]\n",
      " [-0.03655909]\n",
      " [-0.31428595]\n",
      " [-0.15919421]\n",
      " [-0.15107354]\n",
      " [-0.09822534]\n",
      " [ 1.71178267]\n",
      " [ 0.28270942]\n",
      " [ 2.08334637]]\n",
      "MSE 6.24706872545956 R2 -9.36177815110221\n",
      "[-0.33426768 -0.33426768 -0.33426768  4.71358557 -0.33426768 -0.33426768\n",
      "  4.71358557 -0.33426768 -0.33426768 -0.33426768 -0.33426768 -0.33426768\n",
      "  5.25922589  4.71358557 -0.33426768 -0.33426768 -0.33426768 -0.33426768\n",
      " -0.33426768 -0.33426768  4.71358557 -0.33426768 -0.33426768 -0.33426768\n",
      "  5.25922589 -0.33426768 -0.33426768 -0.33426768 -0.33426768 -0.33426768\n",
      " -0.33426768 -0.33426768  4.71358557  4.71358557 -0.33426768 -0.33426768\n",
      " -0.33426768  4.71358557  5.25922589 -0.33426768  5.25922589]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\issohl\\AppData\\Local\\Continuum\\miniconda3\\envs\\mldds02\\lib\\site-packages\\sklearn\\utils\\validation.py:475: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, DataConversionWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "train_X, test_X, train_y, test_y = train_test_split(X, y, random_state=42)\n",
    "\n",
    "X_scaler = StandardScaler()\n",
    "train_X_scaled = X_scaler.fit_transform(train_X)\n",
    "test_X_scaled = X_scaler.transform(test_X)\n",
    "\n",
    "y_scaler = StandardScaler()\n",
    "train_y_scaled = y_scaler.fit_transform(train_y.values.reshape(-1, 1))\n",
    "test_y_scaled = y_scaler.transform(test_y.values.reshape(-1, 1))\n",
    "\n",
    "# comparison: linear regressor\n",
    "lin = LinearRegression()\n",
    "lin.fit(train_X_scaled, train_y_scaled)\n",
    "pred_y_scaled = lin.predict(test_X_scaled)\n",
    "print('MSE', mean_squared_error(test_y_scaled, pred_y_scaled),\n",
    "      'R2', r2_score(test_y_scaled, pred_y_scaled))\n",
    "print(pred_y_scaled)\n",
    "\n",
    "dtr = DecisionTreeRegressor(max_depth=6, max_leaf_nodes=10)\n",
    "dtr.fit(train_X, train_y_scaled)\n",
    "pred_y_scaled = dtr.predict(test_X_scaled)\n",
    "\n",
    "print('MSE', mean_squared_error(test_y_scaled, pred_y_scaled),\n",
    "      'R2', r2_score(test_y_scaled, pred_y_scaled))\n",
    "print(pred_y_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The binary tree structure has 19 nodes and has the following tree structure:\n",
      "node=0 test node: go to node 1 if X[:, 10] <= 0.5 else to node 2.\n",
      "\tnode=1 test node: go to node 5 if X[:, 0] <= 118481.0 else to node 6.\n",
      "\tnode=2 test node: go to node 3 if X[:, 4] <= 2.5 else to node 4.\n",
      "\t\tnode=3 leaf node.\n",
      "\t\tnode=4 leaf node.\n",
      "\t\tnode=5 test node: go to node 7 if X[:, 5] <= 2.5 else to node 8.\n",
      "\t\tnode=6 leaf node.\n",
      "\t\t\tnode=7 test node: go to node 9 if X[:, 6] <= 0.5 else to node 10.\n",
      "\t\t\tnode=8 test node: go to node 11 if X[:, 3] <= 3.5 else to node 12.\n",
      "\t\t\t\tnode=9 leaf node.\n",
      "\t\t\t\tnode=10 leaf node.\n",
      "\t\t\t\tnode=11 leaf node.\n",
      "\t\t\t\tnode=12 test node: go to node 13 if X[:, 2] <= 1.5 else to node 14.\n",
      "\t\t\t\t\tnode=13 leaf node.\n",
      "\t\t\t\t\tnode=14 test node: go to node 15 if X[:, 0] <= 117099.5 else to node 16.\n",
      "\t\t\t\t\t\tnode=15 leaf node.\n",
      "\t\t\t\t\t\tnode=16 test node: go to node 17 if X[:, 4] <= 3.5 else to node 18.\n",
      "\t\t\t\t\t\t\tnode=17 leaf node.\n",
      "\t\t\t\t\t\t\tnode=18 leaf node.\n",
      "\n",
      "Rules used to predict sample 0: \n",
      "decision id node 9 : (X_test[0, -2] (= 1.686342195404005) > -2.0)\n",
      "\n",
      "The following samples [0, 1] share the node [0 1 5 7 9] in the tree\n",
      "It is 26.31578947368421 % of all nodes.\n"
     ]
    }
   ],
   "source": [
    "# http://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html\n",
    "\n",
    "estimator = dtr\n",
    "\n",
    "# The decision estimator has an attribute called tree_  which stores the entire\n",
    "# tree structure and allows access to low level attributes. The binary tree\n",
    "# tree_ is represented as a number of parallel arrays. The i-th element of each\n",
    "# array holds information about the node `i`. Node 0 is the tree's root. NOTE:\n",
    "# Some of the arrays only apply to either leaves or split nodes, resp. In this\n",
    "# case the values of nodes of the other type are arbitrary!\n",
    "#\n",
    "# Among those arrays, we have:\n",
    "#   - left_child, id of the left child of the node\n",
    "#   - right_child, id of the right child of the node\n",
    "#   - feature, feature used for splitting the node\n",
    "#   - threshold, threshold value at the node\n",
    "#\n",
    "\n",
    "# Using those arrays, we can parse the tree structure:\n",
    "\n",
    "n_nodes = estimator.tree_.node_count\n",
    "children_left = estimator.tree_.children_left\n",
    "children_right = estimator.tree_.children_right\n",
    "feature = estimator.tree_.feature\n",
    "threshold = estimator.tree_.threshold\n",
    "\n",
    "\n",
    "# The tree structure can be traversed to compute various properties such\n",
    "# as the depth of each node and whether or not it is a leaf.\n",
    "node_depth = np.zeros(shape=n_nodes, dtype=np.int64)\n",
    "is_leaves = np.zeros(shape=n_nodes, dtype=bool)\n",
    "stack = [(0, -1)]  # seed is the root node id and its parent depth\n",
    "while len(stack) > 0:\n",
    "    node_id, parent_depth = stack.pop()\n",
    "    node_depth[node_id] = parent_depth + 1\n",
    "\n",
    "    # If we have a test node\n",
    "    if (children_left[node_id] != children_right[node_id]):\n",
    "        stack.append((children_left[node_id], parent_depth + 1))\n",
    "        stack.append((children_right[node_id], parent_depth + 1))\n",
    "    else:\n",
    "        is_leaves[node_id] = True\n",
    "\n",
    "print(\"The binary tree structure has %s nodes and has \"\n",
    "      \"the following tree structure:\"\n",
    "      % n_nodes)\n",
    "for i in range(n_nodes):\n",
    "    if is_leaves[i]:\n",
    "        print(\"%snode=%s leaf node.\" % (node_depth[i] * \"\\t\", i))\n",
    "    else:\n",
    "        print(\"%snode=%s test node: go to node %s if X[:, %s] <= %s else to \"\n",
    "              \"node %s.\"\n",
    "              % (node_depth[i] * \"\\t\",\n",
    "                 i,\n",
    "                 children_left[i],\n",
    "                 feature[i],\n",
    "                 threshold[i],\n",
    "                 children_right[i],\n",
    "                 ))\n",
    "print()\n",
    "\n",
    "# First let's retrieve the decision path of each sample. The decision_path\n",
    "# method allows to retrieve the node indicator functions. A non zero element of\n",
    "# indicator matrix at the position (i, j) indicates that the sample i goes\n",
    "# through the node j.\n",
    "\n",
    "node_indicator = estimator.decision_path(test_X_scaled)\n",
    "\n",
    "# Similarly, we can also have the leaves ids reached by each sample.\n",
    "\n",
    "leave_id = estimator.apply(test_X_scaled)\n",
    "\n",
    "# Now, it's possible to get the tests that were used to predict a sample or\n",
    "# a group of samples. First, let's make it for the sample.\n",
    "\n",
    "sample_id = 0\n",
    "node_index = node_indicator.indices[node_indicator.indptr[sample_id]:\n",
    "                                    node_indicator.indptr[sample_id + 1]]\n",
    "\n",
    "print('Rules used to predict sample %s: ' % sample_id)\n",
    "for node_id in node_index:\n",
    "    if leave_id[sample_id] != node_id:\n",
    "        continue\n",
    "\n",
    "    if (test_X_scaled[sample_id, feature[node_id]] <= threshold[node_id]):\n",
    "        threshold_sign = \"<=\"\n",
    "    else:\n",
    "        threshold_sign = \">\"\n",
    "\n",
    "    print(\"decision id node %s : (X_test[%s, %s] (= %s) %s %s)\"\n",
    "          % (node_id,\n",
    "             sample_id,\n",
    "             feature[node_id],\n",
    "             test_X_scaled[sample_id, feature[node_id]],\n",
    "             threshold_sign,\n",
    "             threshold[node_id]))\n",
    "\n",
    "# For a group of samples, we have the following common node.\n",
    "sample_ids = [0, 1]\n",
    "common_nodes = (node_indicator.toarray()[sample_ids].sum(axis=0) ==\n",
    "                len(sample_ids))\n",
    "\n",
    "common_node_id = np.arange(n_nodes)[common_nodes]\n",
    "\n",
    "print(\"\\nThe following samples %s share the node %s in the tree\"\n",
    "      % (sample_ids, common_node_id))\n",
    "print(\"It is %s %% of all nodes.\" % (100 * len(common_node_id) / n_nodes,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAD8CAYAAAAfSFHzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHadJREFUeJzt3Xu8XuOd9/HPV1KRCKGVaUOxq4LGKdh1DlIZilFSOnRMJ8E0L9qpJ+MxaoZR9VTx0I6pdkZTNLRepKGMcapDJeIQ7CRbEl7Oh9ZhPIKGSBwav+ePdW1Zdu6dfd/7Pu61v+/Xa7+s+7qvtdZ17Zt8XWut3D9FBGZmZv3dWs0egJmZWS040MzMrBAcaGZmVggONDMzKwQHmpmZFYIDzczMCsGBZmZmheBAMzOzQnCgmZlZIQxu9gAGko022ija2tqaPQwzs35l3rx5SyJiZG/9HGgN1NbWRkdHR7OHYWbWr0h6oZx+vuRoZmaF4EAzM7NCcKCZmVkhONDMzKwQHGhmZlYIDjQzMysEB5qZmRWCA83MzArBgWZmZoXgQDMzs0JwoJmZWSE40MzMrBAKH2iSZkk6sFvbVEmXS7p2Dfu0N2aEZmZWC4UPNOBq4OhubUcDv4yII5swHjMzq4OBEGjXAn8laQiApDZgY+BFSYtT21BJ10haKGkGMLRrZ0kHSHpA0nxJMyUNT+37S1ogaVFa7Q1p9MTMzGyVwgdaRLwOPAR8OTUdDcwAItftRGB5ROwAnAPsAiBpI+AMYEJE7Ax0ACdLWgeYDhwVEduT1ZU7sf6zMTOznhQ+0JL8Zcej0+u8fYBfA0TEQmBhat8dGAPcJ6kTmARsDmwNPBcRT6Z+V6RjrEbSFEkdkjpee+21Gk3HzMy6GyiBdgOwv6SdgaERMb9EnyjRJuCOiBibfsZExPGpvSwRMS0i2iOifeTIXiuIm5lZHw2IQIuIZcAs4HJWX50B3AMcAyBpO2CH1D4X2EvSlum9YZK2Ah4H2rragW8As+s2ATMz69WACLTkamBH4JoS7/0nMFzSQuBUsntuRMRrwGTg6vTeXGCbiHgXOBaYKWkR8CFwSd1nYGZmPVJEqSttVg/t7e3R0dHR7GGYmfUrkuZFRK9/N3ggrdDMzKzAHGhmZlYIDjQzMysEB5qZmRWCA83MzArBgWZmZoXgQDMzs0JwoJmZWSE40MzMrBAKG2iSTpf0aKpx1ilpt1SpelgZ+5bVz8zMWkchA03SHsBfATunGmcTgD8CU4FygqrcfmZm1iIKGWjAKGBJRLwHEBFLgCPJKlXfLeluAEn/mWqVPSrp+6ntpBL9lnUdWNKRkqan7a9JWizpEUn3NHB+ZmbWTVED7XZgU0lPSvoPSftGxE+Al4HxETE+9Ts9feHlDsC+knbooV9PzgQOjIgdga/UaS5mZlaGQgZaqn+2CzAFeA2YIWlyia5/LWk+sADYlqw6dSXuA6ZL+iYwqFQHV6w2M2uMwc0eQL1ExEqyop6zUs2ySfn3JX0OOAX4YkS8mS4jrtPT4XLbH/WJiBMk7QYcAnRKGhsRr3cbxzRgGmTlY6qalJmZ9aiQKzRJW0sanWsaC7wAvA2sl9rWB94Blkr6NHBQrn++H8Crkr4gaS1gYu48n4+IByPiTGAJsGntZ2NmZuUo6gptOHCxpA2APwNPk11+/Dpwq6RXImK8pAXAo8CzZJcPu0zL9wNOA24ie1JycTo+wAUpOAXcBTxS/6mZmVkprljdQK5YbWZWOVesNjOzAcWBZmZmheBAMzOzQnCgmZlZITjQzMysEBxoZmZWCA40MzMrBAeamZkVggPNzMwKwYFmZmaFUPhAk7RSUmcqxDlT0jBJbZIWV3icyZI2rtc4zcysOoUPNGBFRIyNiO2A94ET+nicyWSVrM3MrAUNhEDLmwNsmbYHSfqFpEcl3S5pKICksZLmSloo6XpJG0o6EmgHrkqrvaGS9pe0QNIiSZdLGtKsSZmZ2QAKNEmDyWqeLUpNo4GfRcS2wJ+AI1L7lcB3I2KH1Pd7EXEt0AEcExFjyQp+TgeOiojtycrwnNjDeV2x2sysAQZCoA2V1EkWSH8ALkvtz0VEZ9qeB7RJGgFsEBGzU/sVwD4ljrl12v/JXvoREdMioj0i2keOHFmD6ZiZWSlFLfCZtyKtqj4iCeC9XNNKYGgFx1QNxmVmZjU0EFZoZYuIpcCbksalpm8AXau1t4H10vbjZCu6LUv0MzOzJhgIK7RKTQIukTQMeBY4NrVPT+0rgD1S+8x0b+5h4JImjNXMzBJFRLPHMGC0t7dHR0dHs4dhZtavSJoXEe299fMlRzMzKwQHmpmZFYIDzczMCsGBZmZmheBAMzOzQnCgmZlZITjQzMysEBxoZmZWCA40MzMrhAEfaJKWlWg7QdLf9bLfZEk/rd/IzMysEv4uxxIiwt/LaGbWzwz4FVopks6SdEraniXpfEkPSXoy9038+f6HSHpA0kaNH62ZmYEDrVyDI2JXYCrwvfwbkiYCpwEHR8SS7ju6YrWZWWM40Mrz2/TPeUBbrn088F3gkIh4s9SOrlhtZtYYDrTydFW3XsnH7zs+S1b0c6uGj8jMzD7GgVadF4CvAldK2rbZgzEzG8gcaDBM0ou5n5Mr2TkingCOIate/fn6DNHMzHoz4B/bj4g1hnpE7JfbXkK6hxYR04HpaXsBMKZOQzQzszJ4hWZmZoXgQDMzs0JwoJmZWSE40MzMrBAcaGZmVggD/inHRlr00lLaTru52cMwM2uo5887pCHn8QrNzMwKwYFmZmaF4EAzM7NCGNCBJmmlpE5JiyXNlDSsl/63SNqgRPtH9dPMzKw5BnSgASsiYmxEbAe8D5ywps4RcXBE/KkxQzMzs0oM9EDLmwNsCSDpBknzJD0qaUpXB0nPd1WllnS6pCck3Qls3Zwhm5lZFz+2D0gaDBwE3JaajouINyQNBR6WdF1EvJ7rvwtwNLAT2e9wPlnxTzMza5KBHmhDJXWm7TnAZWn7JEkT0/amwGjg9dx+44DrI2I5gKQbezpBWuFNARi0vitWm5nVy0APtBURMTbfIGk/YAKwR0QslzQLWKfEvlHOCSJiGjANYMio0WXtY2ZmlfM9tNWNAN5MYbYNsHuJPvcAEyUNlbQecGhDR2hmZqsZ6Cu0Um4DTpC0EHgCmNu9Q0TMlzQD6AReILtcaWZmTTSgAy0ihpdoe4/sAZFS/dty2+cA59RtcGZmVhFfcjQzs0JwoJmZWSEM6EuOjbb9JiPoaFAZBTOzgcYrNDMzKwQHmpmZFYIDzczMCsH30Bpo0UtLaTvt5mYPw2qgUSXlzax8XqGZmVkhONDMzKwQWi7QKq0i3cMxpva0n6RZktpzr9skLa5mzGZm1nwtF2hUWEW6B1OBioOwWpIGNfqcZmaWacVAy8tXkT45rdoWS5qa2taVdLOkR1L7UZJOAjYG7pZ0dyUnk7SOpF9KWiRpgaTxqX2ypJ/m+t2UyswgaZmksyU9COxRk1mbmVnFWvYpx3wV6VQh+lhgN0DAg5JmA1sAL0fEIWmfERGxVNLJwPiIWNLD4a+StCJtrw18mLa/DRAR26fSMbdL2qqXoa4LLI6IM/s2UzMzq4VWXKF1VZHuAP5AVkV6b7IK0e9ExDLgt2RVoxcBEySdL2lcRCwt8xzHpMuaY4GDc+17A78CiIjHyUrD9BZoK4HrenpT0hRJHZI6Vi4vd3hmZlapVlyhlaoirVIdI+LJtHo7GDhX0u0RcXYV5y55HuDPfDz88xWs342IlT0d0BWrzcwaoxVXaKXcAxwuaZikdYGJwBxJGwPLI+LXwIXAzqn/28B6fTzPMQDpUuNmZEU+nwfGSlpL0qbArtVMxszMaq8VV2irSRWipwMPpaZLI2KBpAOBCyR9CHwAnJjenwbcKumViBhfwan+A7hE0iKyVdnkiHhP0n3Ac2SXOBcD86uflZmZ1ZIifBWsUYaMGh2jJl3U7GFYDfirr8waR9K8iGjvrV9/ueRoZma2Rg40MzMrhH5xD60oXLHazKx+vEIzM7NCcKCZmVkhONDMzKwQfA+tgYpYsdqPr5tZq/AKzczMCsGBZmZmheBAMzOzQuhzoElaKakzFdacKaniCtGSppbaT9Jhkm7Ivf5nSU/nXh8q6cY+jnuWpDV+hUr3gp5mZtb6qlmhrUg1xbYD3gdO6MMxpgKlgvB+Pl79eQ/gLUl/kV7vCdzXh/OZmVlB1eqS4xxgSwBJJ6dV22JJU1PbupJulvRIaj9K0knAxsDdku7OHywiXgOWStoyNW1CVkRzz/R6T7LQQ9IBkh6QND+tFIen9l0kzZY0T9LvJI3KnyOVgrlC0g/S62MlPZkqYe+V63eopAclLZB0p6RPp32fkjQyd6ynJW1Uo9+nmZlVqOpAkzQYOAhYlIptHgvsBuwOfFPSTsCXgZcjYse0orstIn4CvAyM76HEy/3AnpK2Bp4C5qbXg4EdgIdTgJwBTIiIncmqXJ8s6RPAxcCREbELcDlwTu7Yg4GrgCcj4owUdt8nC7K/BMbk+t4L7B4ROwHXAKdGxIfAr0m104AJwCMRsaTE78cVq83MGqCav4c2VFJn2p4DXEZWj+z6iHgHQNJvgXHAbcCFks4HboqIOWUc/z6yldgg4AGyWmhnAjsBT0TEu5ImkIXPfamo9dqp79bAdsAdqX0Q8Eru2D8HfhMRXSG3GzArrQyRNAPYKr33WWBGCr21yeqiQRaS/wVcBBwH/LLUJFyx2sysMaoJtBURMTbfoJQe3UXEk2n1djBwrqTbI+LsXo5/P/AdsjD6RUS8LWkdYD9W3T8TcEdEfL3bOLYHHo2I/H247sceL+lHEfFu1zB76Hsx8OOIuFHSfsBZaU5/lPSqpC+RBeIxPexvZmYNUOvH9u8BDpc0TNK6wERgjqSNgeUR8WvgQmDn1P9tYL0ejvUY2T22ccCC1NZJ9vDJ/en1XGCvrntt6bxbAU8AIyXtkdo/IWnb3LEvA24BZqZLmA8C+0n6VLpc+bVc3xHAS2l7UrcxXkp26fE3EbGyl9+NmZnVUU0DLSLmA9PJLg8+CFwaEQuA7YGH0iXK04EfpF2mAbd2fygkHSvSMZZExAep+QFgC1KgpUuEk4GrJS0kC7htIuJ94EjgfEmPkAXhnt2O/2NgPvAr4FWyldcDwJ2pvctZZME3B+h+j+xGYDg9XG40M7PGUZYb1hfp77P9W0SMK6f/kFGjY9Ski+o8qsbydzmaWb1JmhcRa/z7w+AvJ+4zSaeRPQTje2dmZi3AX33VRxFxXkRsHhH3NnssZmbmFVpDbb/JCDp8ic7MrC68QjMzs0JwoJmZWSH4kmMDFbFitVmen3q1ZvIKzczMCsGBZmZmheBAMzOzQmhKoNWz2nV6b5akP+S/LFnSDZKWVTHmj52vmmOZmVntNWuFVs9q113+RCrUKWkDYNQa+tbifGZm1kStcMmxptWuc64Bjk7bXwV+2/WGMhek4y2SdFRq3y+t7q6V9Likq1LfkueTdE4a11xJn67x78XMzCrQ1ECrY7VrgLuAfSQNIgu2Gbn3vgqMBXYkqzZ9QSrgCVkB0alkhUO3APbq4XzrAnMjYkeysjnfrPLXYWZmVWhWoHVVu+4A/kBWn2xvUrXriFhGtqIaBywCJkg6X9K4iFha5jlWAvcCRwFDI+L53Ht7A1dHxMqIeBWYDXwxvfdQRLwYER+SlZ1p6+H47wM3pe15PfWTNEVSh6SOlcvLHbqZmVWqWX+xut7VrrtcA1xPqjKdP90a9nkvt72Snn9HH8Sq2js99ouIaWR13xgyarRr9ZiZ1Ukr3EPrUstq113mAOcCV5c411GSBkkaCexDVpR0Tco5n5mZNUnLfPVVRMyXNJ1VwXJpRCyQdCDZPa4PgQ/IapDBqmrXr/R0Hy2toC4s8db1wB7AI0AAp0bE/0jaZg1D7PV8ZmbWPK5Y3UBFrFhtlufvcrR6KLdidStdcjQzM+szB5qZmRVCy9xDGwhcsdrMrH68QjMzs0JwoJmZWSE40MzMrBB8D62BFr20lLbTbm72MJrGj3SbWT15hWZmZoXgQDMzs0Jo+UCT9BlJ10h6RtJjkm6RtFUPfTeQ9K1Gj9HMzJqvpQMtfQP/9cCsiPh8RIwB/gXoqZjmBkDdAy3VcTMzsxbS0oEGjCcr03JJV0NEdAILJN0laX6qOH1Yevs84POSOiVdACDpnyQ9LGmhpO93HUfSv6aq1HdIulrSKal9bKpAvVDS9ZI2TO2zJP1Q0mzgdEnPSfpEem99Sc93vTYzs8Zr9ZXGdmTFM7t7F5gYEW9J2giYK+lG4DRgu65aa5IOAEYDu5LVQLtR0j7AcuAIsurUg4H5ufNcCXwnImZLOhv4HlkFa4ANImLfdOw24BDgBrKK2NdFxAc1nLuZmVWg1QOtJwJ+mMLpQ2ATSl+GPCD9LEivh5MF3HrAf0XECgBJ/53+OYIstGan/lcAM3PHm5HbvhQ4lSzQjgW+WXKg0hRgCsCg9UdWNEkzMytfq19yfBTYpUT7McBIYJe0GnsVWKdEPwHnRsTY9LNlRFzGmitWr8k7XRsRcR/QJmlfYFBELC61Q0RMi4j2iGgfNGxEH09rZma9afVA+z0wRNJHqx9JXwQ2B/5fRHwgaXx6DatXlf4dcJyk4WnfTST9BXAvcKikddJ7hwBExFLgTUnj0v7fAGbTsyvJqmH/ssp5mplZlVr6kmNEhKSJwEWSTiO7d/Y8cBbwE0kdQCfweOr/uqT7JC0Gbo2If5L0BeCB7IFJlgF/GxEPp3tujwAvAB3A0nTaScAlkoYBz5JdTuzJVcAPyELNzMyaaMBWrJY0PCKWpeC6B5gSEfMrPMaRwGER8Y1y+g/0itX+6isz64tyK1a39AqtzqZJGkN27+2KPoTZxcBBwMH1GJyZmVVmwAZaRPxNlft/p1ZjMTOz6g3YQGsGV6w2M6ufVn/K0czMrCwONDMzKwQHmpmZFYLvoTXQQK9Y3V/4rxeY9U9eoZmZWSE40MzMrBAcaGZmVggD6h6apE8Bd6WXnwFWAq+l17tGxPtNGZiZmVVtQAVaRLwOdBX/PAtYFhEXNnVQZmZWE77kCEg6V9K3c6/Pl/QtSRMk3S3pBkmPSfqZ0tf2SzpI0gOS5kuaIWnd5s3AzMwcaJlLgckAkgYBX2NVSZjdgKnA9sAXgMNSTbXTgP0jYmdgIfC/Sh1Y0hRJHZI6Vi5fWqqLmZnVwIC65NiTiHhG0tuSticrFvpQRLyZFmNzI+J5AEnXAHun3cYA96c+a5MVDS117GnANMjKx9RzHmZmA5kDbZXLyFZpbcDPc+3dQygAAbeVWwfNzMzqz5ccV7kOOJTsoZE7c+27S9osXYr8a7KV2P3AvpK2AJC0rqTRjR6wmZmt4hVaEhHvSroH+J+I+DD31v3Aj4BtgVnAjRERko4HZkhaO/X7F+CpRo7ZzMxWGbCBFhFn5V9LWgvYFTi8W9d3IuJrJfa/A7ijbgM0M7OK+JIjkB4GeYbsvtizzR6PmZlVbsCu0PIiYhHwuRLtd/Lx+2lmZtaiHGgNtP0mI+hwaRIzs7rwJUczMysEB5qZmRWCLzk2kCtWWzlcMdusb7xCMzOzQnCgmZlZITjQzMysEHoNNEkrJXVKWixppqRh9RqMpLGSDi6j336S9iyj32RJP11Tu6QTJP1d2p4lqb0vYzczs+YqZ4W2IiLGRsR2wPvACXUcz1ig10AD9gN6DbRyRMQlEXFlLY5lZmbNU+klxznAlgCpivM8SY9KmtLVQdLxkp5Mq51f5FZCIyVdJ+nh9LNX/sDpS37PBo5KK8KjJH0ynWehpLmSdpDURhaq/5j6jZN0qKQHJS2QdKekT5c7IUlnSTqlW9takq6Q9IP0+oBcdeqZkoan9vNSJeuFki6s8HdpZmY1VPZj+5IGAwcBt6Wm4yLiDUlDgYclXQcMAf4V2Bl4G/g98Ejq/+/Av0XEvZI2A35HVgEagIh4X9KZQHtE/EM658XAgog4XNKXgCsjYqykS4BlEXFh6rchsHv6Fvy/B04F/neffiPZ7+QqYHFEnCNpI+AMYEJEvCPpu8DJKagnAtuk827Qx/OZmVkNlBNoQyV1pu05ZIUwAU6SNDFtbwqMBj4DzI6INwAkzQS2Sn0mAGNShWeA9SWtFxFvr+HcewNHAETE7yV9StKIEv0+S1bKZRRZ9ejnyphXT34O/CYizkmvdyerTn1frjr1A8BbwLvApZJuBm4qdbC0ep0CMGj9kVUMy8zM1qScQFsREWPzDZL2IwuoPSJiuaRZwDpklZx7slbqv6KC8ZU6XvcK0gAXAz+OiBvT2M6q4Bzd3Q+Ml/SjiHg3jeGOiPj6aoOTdgX2B44G/gH40mqDjZgGTAMYMmp0qbGbmVkN9PWx/RHAmynMtiFbxQA8RFbJecN0ifKI3D63k/2hD2RPNJY47tvAernX9wDHpP77AUsi4q0S/UYAL6XtSX2cU5fLgFuAmWkOc4G9JHXdOxwmaat0H21ERNwCTCV7oMXMzJqkr4F2GzBY0kLg/5D9oU9EvAT8EHiQrOzKY8DStM9JQHt6gOIxSj8teTfZZclOSUeRrbTa03nOY1VY/TcwseuhkNRvpqQ5wJI+zukjEfFjYD7wK+B1YDJwdRrHXGAbskC9KbXNBv6x2vOamVnfKaK2V8EkDY+IZWl1cz1weURcX9OT9FNDRo2OUZMuavYwrMX5uxzNPk7SvIjo9e8I1+ObQs5KD5EsJns444Y6nMPMzOxjav5t+xFxSu+9zMzMasvlYxrIFavNzOrHX05sZmaF4EAzM7NCcKCZmVkhONDMzKwQHGhmZlYIDjQzMysEB5qZmRWCA83MzArBgWZmZoVQ8y8ntp5Jeht4otnjqNJG1KCiQZN5Dq3Bc2gN/WEOm0dErxWS/dVXjfVEOd8Y3cokdXgOzec5tAbPobX4kqOZmRWCA83MzArBgdZY05o9gBrwHFqD59AaPIcW4odCzMysELxCMzOzQnCg1YCkL0t6QtLTkk4r8f4QSTPS+w9Kasu998+p/QlJBzZy3N3G2Kc5SGqTtEJSZ/q5pNFjz42xtznsI2m+pD9LOrLbe5MkPZV+JjVu1KuNsZo5rMx9Djc2btSrjbG3OZws6TFJCyXdJWnz3Hv95XNY0xxa4nNIY+ltHidIWpTGeq+kMbn3WuLPpopEhH+q+AEGAc8AWwBrA48AY7r1+RZwSdo+GpiRtsek/kOAz6XjDOpnc2gDFveTz6EN2AG4Ejgy1/5J4Nn0zw3T9ob9aQ7pvWX95HMYDwxL2yfm/l3qT59DyTm0yudQwTzWz21/BbgtbbfEn02V/niFVr1dgacj4tmIeB+4BjisW5/DgCvS9rXA/pKU2q+JiPci4jng6XS8RqtmDq2i1zlExPMRsRD4sNu+BwJ3RMQbEfEmcAfw5UYMuptq5tAqypnD3RGxPL2cC3w2bfenz6GnObSScubxVu7lukDXQxWt8mdTRRxo1dsE+GPu9YuprWSfiPgzsBT4VJn7NkI1cwD4nKQFkmZLGlfvwfagmt9lf/oc1mQdSR2S5ko6vLZDK1ulczgeuLWP+9ZLNXOA1vgcoMx5SPq2pGeA/wucVMm+rcbfFFK9UquU7o+O9tSnnH0boZo5vAJsFhGvS9oFuEHStt3+z68Rqvld9qfPYU02i4iXJW0B/F7Sooh4pkZjK1fZc5D0t0A7sG+l+9ZZNXOA1vgcoMx5RMTPgJ9J+hvgDGBSufu2Gq/QqvcisGnu9WeBl3vqI2kwMAJ4o8x9G6HPc0iXJF4HiIh5ZNfat6r7iFdXze+yP30OPYqIl9M/nwVmATvVcnBlKmsOkiYApwNfiYj3Ktm3AaqZQ6t8DlD57/MaoGtF2SqfRWWafROvv/+QrXKfJbtx2nXjddtufb7Nxx+o+E3a3paP33h9luY8FFLNHEZ2jZns5vNLwCdbcQ65vtNZ/aGQ58geRNgwbfe3OWwIDEnbGwFP0e0BgFaZA9kf8M8Ao7u195vPYQ1zaInPoYJ5jM5tHwp0pO2W+LOp4jk3ewBF+AEOBp5M/4KfntrOJvs/N4B1gJlkN1YfArbI7Xt62u8J4KD+NgfgCODR9C//fODQFp7DF8n+z/Md4HXg0dy+x6W5PQ0c29/mAOwJLEqfwyLg+Baew53Aq0Bn+rmxH34OJefQSp9DmfP49/TfbydwN7nAa5U/myr58TeFmJlZIfgempmZFYIDzczMCsGBZmZmheBAMzOzQnCgmZlZITjQzMysEBxoZmZWCA40MzMrhP8Ptlt5jCPO1M0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_features = len(X.columns)\n",
    "\n",
    "plt.barh(range(n_features), dtr.feature_importances_)\n",
    "plt.yticks(range(n_features), X.columns);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if you want to silence the warning about int64, you can use something like this to convert to int32:\n",
    "\n",
    "https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.astype.html\n",
    "    \n",
    "```\n",
    "df['Page total likes'] = df['Page total likes'].astype('int32')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Evaluate Metrics: Regression\n",
    "\n",
    "We'll now plot the learning curve for regression to see how well the decision tree performed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "def plot_learning_curve(axis, title, tr_sizes, tr_scores, val_scores):\n",
    "    \"\"\"Plots the learning curve for a training session\n",
    "    Arg:\n",
    "        axis: axis to plot\n",
    "        title: plot title\n",
    "        tr_sizes: sizes of the training set\n",
    "        tr_scores: training scores\n",
    "        val_scores: validation scores\n",
    "    \"\"\"\n",
    "    tr_scores_mean = np.mean(tr_scores, axis=1)\n",
    "    tr_scores_std = np.std(tr_scores, axis=1)\n",
    "    val_scores_mean = np.mean(val_scores, axis=1)\n",
    "    val_scores_std = np.std(val_scores, axis=1)\n",
    "    \n",
    "    axis.fill_between(tr_sizes, tr_scores_mean - tr_scores_std,\n",
    "                    tr_scores_mean + tr_scores_std, alpha=0.1,\n",
    "                    color=\"r\")\n",
    "    axis.fill_between(tr_sizes, val_scores_mean - val_scores_std,\n",
    "                    val_scores_mean + val_scores_std, alpha=0.1, color=\"g\")\n",
    "    axis.plot(tr_sizes, tr_scores_mean, 'o-', color=\"r\",\n",
    "            label=\"Training score\")\n",
    "    axis.plot(tr_sizes, val_scores_mean, 'o-', color=\"g\",\n",
    "            label=\"Cross-validation score\")\n",
    "    axis.set(title=title,\n",
    "           xlabel='Training examples',\n",
    "           ylabel='R2 Scores')\n",
    "    axis.grid()\n",
    "    axis.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Generate the learning curve for decision tree classifier, using 3-fold Cross Validation\n",
    "train_sizes, train_scores, validation_scores = learning_curve(\n",
    "    DecisionTreeRegressor(random_state=42), train_X, train_y,\n",
    "    random_state=42)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15, 8))\n",
    "\n",
    "plot_learning_curve(ax, 'Learning Curve: Decision Tree Regressor',\n",
    "                    train_sizes, train_scores, validation_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The Training R2 Score is constant. Can you think of a reason why?\n",
    "\n",
    "hint: consider how a decision tree would be created on the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Validation R2 score isn't very good.\n",
    "\n",
    "Ways to improve:\n",
    "- Try Random Forest or XGBoost (typically will perform better)\n",
    "- Use a non-decision tree algorithm (Linear Regression, anyone?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Visualizing the Decision Tree\n",
    "\n",
    "Finally, we'll visualize the decision tree.\n",
    "\n",
    "This tree will look very complicated, because:\n",
    "1. There are many features\n",
    "2. We didn't place any limits on max_depth (which we should to make the tree more robust and less likely to overfit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "feature_names = list(balanced_df.columns)\n",
    "feature_names.remove('Lifetime Post Total Reach')\n",
    "target_names = 'Lifetime Post Total Reach'\n",
    "\n",
    "filename = '../data/Facebook_metrics/facebook_likes_regressor_maxdepth6_maxleaf10_noscale.dot'\n",
    "\n",
    "visualize_tree(dtr, feature_names, target_names, filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Optional Exercises\n",
    "\n",
    "This is a loooong workshop, but if you desire to learn more about Decision Trees, try the following:\n",
    "\n",
    "1. XGBoost regression\n",
    " - Replace `DecisionTreeRegressor()` with `xgb.Regressor()` with optional `GridSearchCV()`\n",
    " - https://github.com/dmlc/xgboost/blob/master/demo/guide-python/sklearn_examples.py\n",
    " \n",
    "2. Random Forest Regressor\n",
    " - Replace `DecisionTreeRegressor()` with `RandomForestRegressor()` with optional `GridSearchCV()`\n",
    "\n",
    "3. Try other Decision Tree algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree Algorithms\n",
    "\n",
    "ID3: basic algorithm (Iterative Dichotomiser 3)\n",
    "\n",
    "C4.5: ID3 + if-then rules and continuous variable support\n",
    "\n",
    "CART: (what sklearn uses). C4.5, adapted for classification and regression trees.\n",
    "\n",
    "(CART = Classification And Regression Trees)\n",
    "\n",
    "Reference: http://scikit-learn.org/stable/modules/tree.html#tree-algorithms-id3-c4-5-c5-0-and-cart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ID3\n",
    "\n",
    "https://www.cise.ufl.edu/~ddd/cap6635/Fall-97/Short-papers/2.htm\n",
    "\n",
    "Iterative algorithm\n",
    "- Select a feature and threshold (e.g. rnd_expenditure <= 1.416)\n",
    "- Split the nodes\n",
    "- Repeat for children until tree is created\n",
    "- Compute Entropy (how much information is gained from the split)\n",
    "   \n",
    "   $$Entropy(S)=\\sum _{x\\in X}-p(x)\\log _{2}p(x)$$\n",
    "   \n",
    "   - S: data set\n",
    "   - X: classes or discrete values\n",
    "   - p(x): number of elements in class x, divided by number of elements in S\n",
    "\n",
    "\n",
    "   - Entropy(S) = 0: all elements are of the same class / discrete value\n",
    "   - Entropy(S) = 1: random (not classified / regressed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "livereveal": {
   "autolaunch": false,
   "overlay": "<div class='logo'><img src='assets/Stackup_Logo_Small.png' width='90%'/></div>"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}