lectures/070_scikit_learn/100-scikit-learn-method_behind_the_madness.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext postcell\n",
    "%postcell register"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-Learn\n",
    "\n",
    "The scikit-learn library is the premier, non-deep learning, machine learning library in the Python ecosystem. It contains algorithms for classification, regression, clustering and utilities such as test/train data set creation, various transfomers for scaling, imputation algorithms, etc.\n",
    "\n",
    "An extremely important feature of this library is that it is designed as a cohesive whole. Allmost all algorithms fit in this framework like lego blocks. In other words, this is _not_ just a collection of algorithms which you can download as one packages.\n",
    "\n",
    "While other classes in though program will teach the specific of algorithms, such as _linear regression_ or _decision trees_, this lecture will describe how the API works at a higher level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick classification models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download a simple dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load a simple dataset\n",
    "import seaborn as sns\n",
    "titanic_df = sns.load_dataset('titanic')\n",
    "\n",
    "#only keep numeric columns (for simplicity)\n",
    "titanic_onlynum_df = titanic_df.drop(['sex', 'embarked', 'class', 'who', 'deck', 'embark_town', 'alive'], axis=1)\n",
    "\n",
    "#remove na, nan, etc.\n",
    "titanic_onlynum_noempty_df = titanic_onlynum_df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_onlynum_noempty_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_onlynum_noempty_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set up imports and test/train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pre-training\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#post training \n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_noempty_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_noempty_df.survived\n",
    "    , random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import tree\n",
    "\n",
    "model = tree.DecisionTreeClassifier(splitter='best', criterion='gini')\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "y_predict = model.predict(X_test)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import ensemble\n",
    "\n",
    "model = ensemble.RandomForestClassifier(n_estimators=100, min_samples_split=2)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "y_predict = model.predict(X_test)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import svm\n",
    "\n",
    "model = svm.SVC(C=1.0, kernel='rbf')\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "y_predict = model.predict(X_test)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA9IAAAC6CAYAAABC3HV0AAAgAElEQVR4Ae2dTY7cyNGGeRh9h7BO0IY3dtuCLuDZC4b2A437AqNdbccuLWfQ0Ala45EW41XvDWs8augIDQgY5If8CWZkZCQzWUWyyKpXgMC//Il8MpiMl5ms7h4fH80x///zn/8Y/AMBEAABEAABEAABEJiXwDHx2hR5EfPN278oHQRAYFsEumMHVgyq2+pwWAsCIAACIAACILBNAsfGbMfmR8y3Tb+B1SAAAvMQgJCehytKBQEQAAEQAAEQAIFJCRwrhI/NDyE9aXeiMBAAgY0TgJDeeAfCfBAAARAAARAAgcsgcKwQPjY/hPRl+BlaCQIg0EYAQrqNE1KBAAiAAAiAAAiAwEkJHCuEj80PIX3S7kflIAACKyMAIb2yDoE5IAACIAACIAACIKAROFYIH5sfQlrrFZwDARC4VAIQ0pfa82g3CIAACIAACIDApggcK4SPzQ8hvSl3gbEgAAIzE4CQnhkwigcBEAABEAABEACBKQgcK4SPzQ8hPUUvogwQAIFzIQAhfS49iXaAAAiAAAiAAAicNYFjhfCx+SGkz9q90DgQAIGRBCCkRwJDchAAARAAARAAARA4BYFjhfCx+SGkT9HrqBMEQGCtBAaF9MO7vdntdmb/7sGUBt/hQfW9ueluzHvb+l/35pr2T0Hjw43pus7cfDii8inKOKJ6ZAUBEAABEAABELhcAqVYbIrziPmEXyHmE0BwCAIgIAkUhPSDudtbAX1nbmcW0u+/6Uz3jZPa0rZpj6cYEKcoY9pWoTQQAAEQAAEQAIELITCFYM7LQMynug9iPhULToIACEQCqpC2byVv7x/N4+P9cUKaz0LbAenZ3nyOdbu9xYS0qBeHIAACIAACIAACILAlArkItrHacf8R823JA2ArCIDAmgioQjoOyocK6c9m/6xzS6ntcmr5//rNZ+MEtHKt48u/eyFul4hTOWGpeE9R1nVt9r/2F8OScsqrLe22Zds8vI4ZyjDGfH5znbFI2svMxi4IgAAIgAAIgAAIcAIxPjtOPOvlIOZzrF3seWTciJiPuy32QeBsCcwkpAMvNgvtRKSyhHtwRpoNZvRts0z/+c1NIpzddWXm2xgvlKmc2KMkoKN4nqUMt0Qo1mHcsXwpEK3CHgiAAAiAAAiAAAhwAroAnkpUHyqkg4WI+eKqS8R83G2xDwJnS2BWIc3FsxWndiZa/pPCOLkehDQXv65MVSiHnEWBOiykeR1lkXt4GZyFt9SWxYR10nAcgAAIgAAIgAAIgEBKYM1Cmsc5iPmuxe//IOZLPRlHIHAeBOYR0mwmWS7rlkuZ60K6IjbVurSZ3sNFcOzqI8oQbyfdA4cvY4+VYA8EQAAEQAAEQAAEMgKrFNJqHEZLo9N4DDFf1qU4AQIgsGEC8wjpACS+kfTfMSezvixN8Ve73eA8JKS9sE1mulc6I+1nuenBYrdD7dqwR8F0EAABEAABEACBWQisUkiHliLmYy8NXCyKmG+WmwCFgsCKCMwupL14LgvpwZnZmpAOb0GjQPfCWs56e95HzCb3HXZ4GfEB0xeGHRAAARAAARAAARBoJrB2IY2Yz3clYr5ml0ZCENg0AV1I39+a3W6X/397n/2Zhf/85z8FAFw8WwFamoGVv7rN3ujVhHT2q4jXZv9hb67Zkmkv1PlbQb8fZ7EVcSxmtacowwTRL5e6x5cABYw4DQIgAAIgAAIgAALGZDHYJMIaMZ/hEzCI+XCrgQAItBLQhfSIv0lYFtKtJlxCOv+yIIp33+bB2fhLwII2ggAIgAAIgAAINBOYRDiPiPFkfYj5WroKMV8LJaQBgXMgACG9SC/6WW8ppO3Sn27oF8gXsQ2VgAAIgAAIgAAIbIGAFLZLH0NIt3gJYr4WSkgDAudAAEJ6qV7UlnZDRC9FH/WAAAiAAAiAwOYJLC2cZX0Q0o0uhJivERSSgcC2CUBIb7v/YD0IgAAIgAAIgMCFEJDCduljCOkLcTQ0EwRAoIkAhHQTJiQCARAAARAAARAAgdMSWFo4y/ogpE/b/6gdBEBgXQQgpNfVH7AGBEAABEAABEAABFQCUtgufQwhrXYLToIACFwogUmE9JePHw3+gwF8AD4AH4APwAfgA/CB+XxgaeEs67NC+pcvH/EfDOAD8AH4wJePBkIaLwHwEgQ+AB+AD8AH4APwgQ34gBS2Sx9DSOMlAl6kwAfgA9EHIKQ38ODE2/153u7/9uKFWcN/9O88/Quu4AofgA+cmw8sLZxlfRDSMYCGmNgWixe/vTBz/4dPbMsnpugvCGkI6YudhViDiLY2nFugh/ZAvMAH4APwgXl8QArbpY8hpC9PKEwhNtZQxtwi2pa/hnbChmXvUQhpCOmLFXIQ0vMEegigwRU+AB+AD8zjA0sLZ1kfhPSyQTpE0XS8IaSnYwm/jCwhpCGkIaRPvMQbAec8ASe4git8AD5wbj4ghe3SxxDSMYCGmNgWCwjpbfXXVu6vopC+f7szux39vzX3j49GG7DtoFp7UP3v9ZXpui78vzLffdjYw/2Hl6brXpp3A6L73d+ofZ3p/vCt+d9A2hqvVVz/8K35Y9eZP77+udq/q7D3AN6Ykd7YfXhAH2/VN2E3fBM+AB/QfECLw6Y4NybmqwW4//5HGvO9/u/GAvgffcz3w8CvMv/wNYv5/vyt+fdA2hqvVVz/77fmquvM1T9+PtvlyRDSG7sPN3JP6UL6/tbs3t73wtkNsPs786CI6aqQdiJ0g+KZB+0NQpoeeO6lAYT0YuLbvcD42/cH1UdC+l+/60z3u+cn++Ex8h1sETjDB+AD8AH4wJAPTCGaszJGxnyDws+J0CuzOfHMg/YGIU0M3EsDCOnFxLd7gfH19wfVByENIU337ZRbXUhLwfzpzux3e3P3KZ+VrgnpsxCWlyik+YuEFe9DSCPoHAo6cQ3+AR+AD5yTD2QiWMZrUxxXYr6hIPQshOUlCmn+ImHF+xDSEMND488prp1USFsRZJcO86Xfcikxv2aXh/Pr7trfvjXf/cEvsXn1w/fmlVtCPnYG/Oe+DFqC/uoHFnxIIe2OO9MpM6GDLw4oX1jmntRhxWpYTk02yCXivr3fmy+8HG6DPW9nw/n10bPjxNAz5bxdMBLq+C4sZf/j6+97dn17NDuypfFpPdnSecfCLqfn6eLy+mQpff/ZgLU5pqkFT24mOslLS7Wemn/Rd9N/vTJ/6uzxc/P3Pi277tJ9Zf75f5S3M93/XZkHyk/b50/Z5w2d+fvz+Ke3anbiOrsXV/xSB/2EfoIPwAfm9oEtC2krguzSYb70Wy4l5tdsLMSvu2tff2te/9k/b1/++L156Z7LY2fAf+7LoHjr5Y9MoEgh7Y470ykzoYMvDihfiB2SOqxYDcupyYZOzGz79n5vfuHlcBvseZuHXxdl1IUFMfRMOW+XN9TxOixlv/rH9z27vj2aHd1Lky6NT+uxsVpy3bGw53i6mCZZSt/HYtbmmKbWVj8j/ZW5+guL17rOPP2J/izWc/O068yT/VfJn8n6av/EdN1T8/y3F8btv7rqy3j6k8/TdU/M1S/41e5aH5zj9SYhPX5pdy5M+4GCCb9eDJHYc+IpimAvorkw8qKKxJ2/3hkr4HxZNq+vm9K0PNRcXmZXlscJ02CH2482yrRFIS3zBdHci8+PP5vv/sa/rU7bauuh9kax6NP0Zbg6+DfaeRnS3vJxgWOow/Ile3r+xDCzI/gDXQ/imPeRL4v1NXupQO3T+kk7V25TGuQ1Le12QtoPuiR+06XgQUSzpeHuOhfTTkQ/Mf/8axDPoUwqr9VepEv7DzzAAz4AH7g0H1hCSNdivjwYzoVpH/Mx4deLIRJ7TjxFEexFNBdGXlSRuPPXO2MFnC/L5vV1U5rcNiaQw0yry8vsyvI4YRrscPvRRpm2KKRlviCae/H55Wfz+mv+bXXaVlsPtTeKRZ+mL8PV0Xkx7dqWlyHtLR8XOIY6LF+yp+dPDDM7gj/Q9SCOeR/5slhfs5cK1D6tn7Rz5TalfW+F9PNXnelePU+EMl/y7YTyX67MV/3fnPbCm8S1F9VefLuynICOaVptQbq0b7bMoy6k7bczu525vc+XddsB/Zil3U4EkYh2M01cGOpCjgtVvs8Fld3nIq32oHV5h2YynTB8ab5zP5pWFtG2Hm4Tr1ezidvM09K+vJ6JTSu+/8DaGuzkP4omy6Cy61tRNs0Esjp4W90+CWWWpq/HnqO+1q4HcU2imWbn++MC28Pb97H/JjoVxnGm2AltIXrtuYffP4mzzv2MNcvnzkXhbMv/0++/6uuzZfA6e0bEGNuDvnkHR4gq+AB84Nx9YHYh3RDzDQW9RWH5JYhfEtFM+HnhpAs5Xh7f54LK7nORNmSfvebyDs1kOmH40rx2P5pWFtG2LG4Tr1ezidvM09K+vO7KTuwUjIKdfGZXlkFl17ei7PDSwc+Ge8HL2+r2SSgrdrh81Nfa9SCuSTTT7Hx/XGB7ePs+OvHsxa+fXeYCut//5co8CbPL7pw7jum50Oai3O5bsV3nfD4CEm31fTkspMOAun/30P/wmBzEjxbSJL4y8cBFNQsOmAjjQo4LKrs/RkjbB6/LT8tFSPCRTa7OsBSkaK+3kdsUH+hhRpbK51tWnsvLr9l9eV3aRjbaLWNDdXMudK5tO4OQDi8rhhj1/SZWJ5RsPrx9Y4R0FMU0i91vxZLt/i18R3nEsm/ev2EWu9Q2nGf3Pfdz7ONFA3wAPnChPiBjsEmPG2O+oQCaiy2ZblgEidnWipDjZdn9MULa2uXy0/OYBF9SZ4j5SCzSNbHV2xtmZKl8vmXlubz8mt2X16VtvH5FoHIukv/w8QxCOrwEGGLU95tYnVCy9fD2eSFtxbEX06F/k9lnv8SbRLFN65dyxxlsCGm8CJC+WRbS7scmdmZIRNsBfD4hrQs5LsL4PhdUdr8XZKMftkH0MgEbBapyTZTPbeJCqGqTE8HpbDdvky2rVHZfz4qFdGK7YucXdUY65dG3kzGXjLQ0pXMkhvnsMJ3rt2J2uT9P3z5rM9J0LWy1GWleTsk+nIeQhg/AB+AD8AHuA5MKZ/7DZCNiPhlI8mNdNPnge1gE6UKOl8f3eVl2vxdkXGg27QfRywRsnIVVrokyuU2cQ9UmJ4LT2W7eJltWqey+nhUL6cR2xc5f1BnplEffTsZcMtLSlM71s85i2Xa21Punp6ZzAtsu2fbfPlNeCGkIaelfBSF9b27t35BmfwKrNHjPJ6Tpm+D8u1la7svFGRdUVdHKhBh/QNE+L8udS4SfnynnM8WUz265Tfy8F+NlYejy8eXlrs5zmZGW32rL47AigM+2N85IZ9wqfcv7hMSsW6rtflCMLc8mMVwT0i/yb6Sp3H4rv5GmssOW24R9BM3wAfgAfAA+UPKBUix23PlxMZ8MJPlxIqCYALJpaiLI5eVLmcV3xbxsXpbdP1xIK3Ylws/PlPOZ4qb2KkI5y8fb6tKfy4y0/FZbHgfmfLa9cUY68xHhY5yx3CcxzLd8eXY8HwT0T1fmiZixhpCGkJZ+pQpp90MTVkjL/4qwnlNI2weJE7Vs6QuJaHuNi1YufscJ6TDLzOrov+UlUZYIaf7r2iTyg7jmZbh9uh6CAhLHLF1sj7DjD9+ad/abbDYzzturPmSlncSPlaHmo3barWKjW65MZbA6uD1uP0kTls2EtsZ2Bhbsx8Rc+VxEWzsahfSX8J14XFItmPO2if1e6JIY7vuF/Sp3VUhb8a0s3+Y/NmYFs7IEHD82hmC5ej8Kn0V6+Ax84LJ94DjBrP/WzdiYTwaS/JiLXX7e7nPxK6/RsUvTP4v9D4vRNV42L2uckA6zzKwO+WvZcUY6iIb+h7Dox7GCuOZluH26HvKROGbp4jfAwo4/f2t+sN9ks5lx3l5ikGwTwe/r5FyStCXBqdjo4imyg9XB7XH7SZo05ovtlAxDOi6irW2NQvoX+yNt4VfbfdwnmJfa+cUu7fY/ChbjxS7MPNOvdsetE8zJL3r7axDSoT8HODf53RnlV4X0mIG6JqTx0L/Ahz4T22vu/yiklZloMWs8Z9o1M4JtF3j/Qrzj+2P4wGp9YEx8NkdaG/NdWqCM9lbEExPba2YVZ5yjYJ763JrbD9sqfnyguIeQRsAwfcAAIZ38QndNhEOsQqzCB+AD8AH4QIsPzCGOx5QJIT1PML5pkQMh3f85rU3344FC8tLbDCENIQ0hveDssyaqW4InpEGQDR+AD8AH4ANjRO8caSGkIaQz4QQhDSF9wSIcQhpCenohvRGmmqg9xTkExwiO4QPwAfgAfKDFB+YQx2PKhJCGkM6E9EZE1NTLuLXytsoGdh9+X0NIb0T0tTxgkQaBGHwAPgAfgA/AB87XB8aI3jnSQkgfHnBDrIAdfOD8fABCGkL6YmekEWyeb7CJvkXfwgfgA+foA3OI4zFlQkifnxCAuEOfwgcO94FJhLTBPxAAARAAARAAARAAgVkJjBG9c6S1Qhr/QAAEQAAEPAEIaXgCCIAACIAACIAACGyAwBzieEyZENIbcBKYCAIgsBgBCOnFUKMiEAABEAABEAABEDicwBjRO0daCOnD+w45QQAEzo8AhPT59SlaBAIgAAIgAAIgcIYE5hDHY8qEkD5Dp0KTQAAEDiYAIX0wOmQEARAAARAAARAAgeUIjBG9c6SFkF6ur1ETCIDA+glASK+/j2AhCIAACIAACIAACJg5xPGYMiGk4YQgAAIgEAnoQvrTndnvdmbX/9+bu0+P6gCOQTXCxB4IgAAIgAAIgAAIzEVgjOhtTouYb67uQrkgAAJnTkAX0o+paH54tze7/Z15EOftIA0hfeYeguaBAAiAAAiAAAisgkCzOFbitda8iPlW0dUwAgRAYAMEmoT04/2t2e1uzb0yMENIb6CXYSIIgAAIgAAIgMDmCbSK4aPSIebbvJ+gASAAAssQaBLS9293Zvf2Hku7l+kT1AICIAACIAACIAACGYGjBLIyGaKVh5gvw44TIAACIKASKAtp90YyfCddWNZtB2DMSKtccRIEQAAEQAAEQAAEJiWgCd9JziHmm7SfUBgIgMBlECgLaf7m0g2w+g+OQUhfhqOglSAAAiAAAiAAAqclMIlo5vGdto+Y77SdjNpBAAQ2Q6BNSD8+mLv9zuzfPWTLuyGkN9PXMBQEQAAEQAAEQGDDBBYR0oj5NuwhMB0EQGBJAqOE9O19+mvedkCHkF6yu1AXCIAACIAACIDApRJYUkgj5rtUL0O7QQAEWgk0CWn3pxDwq92tTJEOBEAABEAABEAABCYnsISQRsw3ebehQBAAgTMloAppP4iGHxrb7Yp/Qxoz0mfqFWgWCIAACIAACIDA6gjMIaQR862um2EQCIDARgioQnrMQI2l3RvpaZgJAiAAAiAAAiCwaQJj4rM50iLm27T7wHgQAIGJCUBITwwUxYEACIAACIAACIDAHATmEMdjyoSQnqNXUSYIgMBWCUBIb7XnYDcIgAAIgAAIgMBFERgjeudICyF9Ue6GxoIACFQIQEhXAOEyCIAACIAACIAACKyBwBzieEyZENJr8ALYAAIgsBYCENJr6QnYAQIgAAIgAAIgAAIDBMaI3jnSQkgPdA4ugQAIXBwBCOmL63I0GARAAARAAARAYIsE5hDHY8qEkN6i18BmEACBuQhMIqS/fPxo8B8M4APwAfgAfAA+AB+AD8znA2NE7xxprZD+5ctH/AcD+AB8AD7w5aOBkMZLALwEgQ/AB+AD8AH4AHxgAz4whzgeUyaENF4i4EUKfAA+EH0AQnqBB+dvL16YNfyvzRKswUbYsA5fOfd+wL0wzs8uiVetrS3Xz/3+mbp9LUxraaa26dDyanYee32M6J0jLYR0DKBLYuLFby/MEv9L9dP5NdhgbanZQfYObWtlTHF9qP7Wa1PYsZYyWts8lG6JtgzVv8Q1CGkI6X4W4tDAAfnGiRLwOj2vWjCLPkr76JJ41drach3+k/pPjUcL01qaWh1LXa/Zeez1OcTxmDIhpCGkuThqESo8vbY/RRlauWPPtdhRSzO2zjWnr7W15foS7WuxY840ENIQ0hDSK1kxsFSgh3pe9D5fCmrBKBVCJU50/px4UZuO2Z4TjyXacgxryruEnS11kD1zbceI3jnSQkhDSHNx1CJQeHptf4oytHLHnmuxo5ZmbJ1rTl9ra8v1JdrXYsecaSCkIaR7UdESJCBNKjDAY5s88iD3e/Oqe2nehfEA/Zr2a84r/TGlc+JVa2vL9XPisURbWpjW0ixhZ0sdNTvj9XTMiefTe0uen0McjykTQhpCmosjVaD8+NJ0X3/f/xAVT6/tq2WIH/LS8k19rsWOWpqpbTplebW2tlxfwv4WO3ya783L7qX5QfhWe3793q8K6Yd3e7Pb7cz+3YPRBls7qMqBfr5j++DpzKsfhh80Sf0fvjV/7DrThf+j8k4kslsevkukSbgobbM2/Ot3kZVn9sT8869pUD2lrb6+p+ZfB88Kf2X++X+d6f7vyjwcXEapfc/N35nvkA/57TE2l+qrnP/rlfkTt+d3z9m396mtf/r9V+xapdzJub0wvwVbh+1Ibe6S9rwwvz1/6u7bvz+f3v70XvDjyh9f/9yPZZP5uOizclvm9OPj+aW88vE34VXse9/GYZ843tbElkHf1pnX2tpyvd2GxvbOeC+MsfX48VpvbwvTmOZn890fOtP94VvzP/YcG9OO317ofT9FGdHOj+bLDy/dGFaKO/73+iprR5KftY/Oa3HYlOdaYr5jA8/2/Dbw7czLH/UAVi3nv9+aK/acHJV3ogB7CfFg61Dbz9pAdjx/JeK6V8/9d8s/2WfsE3P1i/ym+ytz9ZfOPNl/Fb9v/uXKPGFcu1BGZoMV0d2Vef3f2Gdkx4vWMlgbqPy+jN+em6fOjqfm+UHfovu29fHcX67MV6EcquuYrbPTcT3UPtkXcx4TS+8fSX83+FfCKdx3V//4OfHL2G8D7RjDS/iQtTmxw/lfecz49z+uTPfnb82/FR9LyhlxfVhIf7oz+93e7PcbFtL9g+gAEd7nzQNHeqi1bPuHswsw5xWmfV1KAFmz1eZ1gRIXNDMHcMcHZlMERHpwl7A8cd+pwvT5U5MLs5WIlaKYItah37ivSZ+d0ff4vfDub53p/vZ9L6LttaTvpV0HHfuXBnl/CR6FF0IPv39SeVlUK5/qOWzLeWn7KS9vSy6YF/TNxvvVjT+Cuda+sedSHocxT8oYvBfm7Xtux/Hjtc5iLF93z25BSIcX+SUh/eVjeCkgxp8hHlOK5qysxpjv0IBzfL4DhHQfAB+TN4rA8TbXf1yrSVw0iMSabS9+C6KRicW0bi+kpIDygpcJ7CBeknQ/PTVPf5JiXmfu6mwuQ2fvygiC62r/xHTdYULVvVSgFwnEp/RSoPcl3SaNP7fzMKE/IDgbfCLt39aylBcnY4X0F9/38wpp3V+TfgiCvvwC7Wfz+s9dsmIiyX9Anw8K6fu3XkDTNht0Hx/N6mekezEMIT30cLbXbMCUCekXL4wL4rsTzMAeJFj0QI0HgwftNwbmB5VdbWeD6OzLWFCs9HUewny54F/rk/5ecIHulfnuQ/qyTMtz3Lnj2rtFIZ2/NFjQNxvvV228632jf3akvtFy/ThfGXs/Hedby9qqt62FKU+zzMsv3dYaL27nl8L4MjoN80UtDpvqHMV6tNXKXXZpty7M2gLfY/K2CyfNlsPETKvoiem0uvm5F2H21grekk1OWAqh/ZUVqv25IMZ78ZmWxesrzfb1gr6hDF4e33dtCTY5+w4R0k7MsxcEVpiyWXle36H7ly6kpYAt+V1yvnVGWus/KfidkE5XRGR92ZJmhKAuC+n7W7Pb3Zr7x0dTG1STh4Ib8P1b1levaVn1S/MuLHFKl2N5cdsvsWDfKPZlUj62pCR9uxve6NJ18Za6L+fjCYV0mJ2L7YzLbOKsjQ2G7Gx1EE2uPfnstZ8RoPyKuBV1xfLbfmBJCyz9bGhqS9UOEuXUL11nuC1enA+0w4q0MAvTcxMzR4kN4hoFO2k9qQ2/vSDmPhD19aTtpHI0BnTN2mHbxuvibXXpRFsykTHQb0N1kw1xOyxWEmZdl89oCzvsW1+57F6WkbaVs5S8eWDq02UcbL8LG9Q0tDST/Ev2vyvD2s7tiW2hccEtrVRmg3qeot/STwhs2dZfeB0KU/fCodzehKdoB/ep/j5Ixoa07iSNmO1P6sn6ldqijz/Eq7TteQ2+XCHffO4/xwh9l/pP3v/Zddn33I+F73AeeTncH+N+qY1jzkcenKcf77g/2z6xdvF+TuwU7eF5/fhFY6jYTtj3ti3cPm1McO2V90pig+fw9+cpD2rPGLaltMQ8tZW4xHs/uQ/E/da39XfP0zYnbRGfQIkySvaVz/sYhn9aUk77Uf3EThO8o8+NiPmywPSLn+V5+Q9aVv3S/BCWWKbLKL24jfel8r0i5aOxPVvaHWaU6HpxmeYKhLQTCOSDXV2cMlHnBN5frsxVWJb9ZP/cLbe27EgY5/2QvgAgIR1FcSqCS6LPiut+9rkgXkgIRRuCDyjL8LMZbjGzGstI7efnqT67PVRIpy8ILIu4zDufXS/bwu2S+ypT8oPkRUKs290P/YsL30fOVpue8lp/5/nteZuHXxdlWFvSZf2lWfwpZqR1Xn1/1dpiX4zwtmgvSgq+KPugfux9Vc6e1/PpbSwI6Qdzt9+Z2/tHN2gfKqQ7J4yDWLYCN3k768/zh4cLarmYdiKazxZJMZwvi9KWfPmHkszbPsvgyqRBm2+VAFx7ANIDflgQxYCYAikXELCHtAsA2LEPGGKAQAKEghMKtKg8zTZ+ztrp6hBBgyy3ageVw2ztGchA2wVfrA103QWQBVFLacJWcqK6Mj5BVBEP4mO/EaLvwGXbqKyhvnN5rF9Qe6Xtro2xDskzO5Z2yvyi/b2N7jyJFXlIDJIAACAASURBVOUbabEUvMQn+k8UF1RHiTVdj9uCHTLY7u8nxqdvX0l8hmCc+WnWb0yAUHtcmpDH+305gHXtkP3Y20Vc6J6NtudM07RkS+RE14NYIR8SddW5l1j58iWf3E5qS3z5wev8El5ExgA4Bof2xabWnvxcFFE9B+nbod/66/JeoLGF9X1eD4nx2C9qGsGY0vAx8dB9X1bup1QHbatjR2/jUP8OXQtjOvOrsX1Ptrqt6y9lvJb9SC87+n7K+57bYTn753/0q97Xii/H02d4tI/1e8leejnAuFA7vV2d6X+zIfPJ4Xv2EJ8pvdDTyhotkB99HDecb1zMlwecJG6tMA5i2QrcZObHn+fBq5vB5D/+k31fK8VwqIf9kNUPX3eFbx5lXj0Iztvy0bgy+2cT80lWr5aPzjkxVQj63bV+JjQVNsmy4yAqrKD1wtELaJ6G6ittox2hDYrQIrFN4jw7dnaIWVwmhPu6k75OWfNZ3779oowac57vKCFNYtT1j2UaRWTflhGzkTKPs9MxC32r8gsimmwhwcv6h/o8LmH3y5r7fgr+EV+S5Muena9kZaY+57lGBpzzL3QvK/eCnHmWHOjYlje+LSmfmJ/di84m3xaqa8zWjT2N93OtXFVIux+b2N+ZhzAAHyqkvUjmAtbuB2HsRHL8lVz/wOBpPxorYLnQpmCun5F2wlyUkYh1/qBNy9YeUHOdowf0kBgjURdFHs3IUtBigyUWILgAKw2gXADQBy7+Ye/OhWCh1j5rJxcbqd0001a3Y7idMQhx5ZcCnRC8JDz6oDItg7extzkEcjJ/mjblV7dH8vd2SJFCfUliwF6XdnDOtX7zM/N63bG9xMQHrLK+PJ0mNjyPPoBUeDtbMz+kuvm2ZofCPquvkMb5Bt0XoU53jjFSgl/e99p4w+8Px6vqg5p99hyzo2+TlpbzmlNIazZJe/xx4jfs3uRstH3Vv/q2Uzs1n/Dn6F6p3gs0RmUz6lRHwR8yW0R6dl1r39hzjgfjV+JTGztiPtlf3P7aNemPMv1w30cb5DMp2sDHsz590n6l79k9O5avlt7Wm/uPbZtsv7ebjwe9zVRGIrAV24O/aGVottXOOSHd+MJgWBC3iOY8zdiYLw8s+QwPF7B2PyyzdCJZzkDztF7AcqFNgXwftDuxJsooCri07NzmVOxNed2JkSDU+pldJhy9WBHCyC3DZoKViTEnJIIgcvtBgNVs5qKI/9CXtImLcy96mdhShWCc2e5tUPvXM24R0n05BRHL2+KFFbMxYxvty/K9ei5mtL1om2NG2n/LzfqU7HS+IewXL17yNgqxy/yD2pj0o/QnV7f0OeIkyg521vqk5bq17ZC2OJ9hLwFcGwUjaneLHTJN6TMEma7lWBHS9+Z2tzd3n+JgO4eQ1h8cfIaI75MgFmJYWfbt32LzWexCXvbNUe0ht+SMNAWU/MHu9kNQ37+lZ2+IKI8LZtj5Pu2xQtoFRCEYabDDCz8hdFigmrQtCbZiYKa1mdqZ5FcDH1uODBZD2Ul9SprkOrOHBX2yfjWI7Nvrg7C+L3j/hJcetX4b9WKi8ALB2+zbm9oig8w0TSKsePDYt6PUz+Xgk9tS6tPBNK6P5JtJe8zaMtBftmx/z4vxhI0Jvn562RDrSu1V/Kfkd8Xz0ce0oJzsGLo2yMr2Wcs9W7GPXmKmvuO5jJ2RTn0q9ZPqvRB8MEmXiJ7As9L/xFXb1p4HLddtufU+K7y47MeO6BvF8cyl1fyQc4j+y/sv+vJAfmmLOj6mfdgzTfpAScOuW6ZzzEi7Pii8dCn1T35esT1wydPS2EJxR9tWj4f0vNML6fExXx5c1oW0HrjyfHyfRK4Qw8qyb+/T2jeRIm9BoOVtmWhG2oqRIKbpvutnE4NQ4aI4Ew5MKLl0xwrpUKcqalldqRijNihiUIqtISFdEEBcCC02I21jFzYTzGfgNV8Ye861yfEM425SVxCu/HofS9n0kTPvc+KUbFmf0fmk74TvkQ/arfRDWuIuX7DQiyyel/b7l1uV+8radkhbvJ8Ov2ygdo/tI5teH49o3Bm3zYW0+05m5/7klf2zV+n/VGDbAV3/sTEugnmwavdbZ6R5GfRA4WV9NP6HPMSMNAuG04BH5C2mo7qm2+qBBQ+S7H4tmLHXmVCQAQ4FbmJGuq+7Fw/ldtm0mih05/pgtW7HKOGnBmaSDc2E6O3Xghn68yZp0C6DW4V5yR4W9HGmJWY8jeUn7eDXnf0D/Ua+MVRGLK8U9PnzyWzzQJtcee46rURQ+oT8pfcNnqZkB6VR2Gc+XUjj7CoJ+FB+pW1+bNDGGH9/RJ5kr+aDin3FepW0or26H/v6h655W4fKt9f0eye2cyh/XRzEchgv0T79nkzrrd8LsnzFr229xX6Q+fPj9LlRHi+H0jkepbGEcdHGW51lyilNU7t2XN8ndRXapLYjSauMB6yPhli2XovM+YuDcttL91R+XrE99GGetn6vaO1xQrrxU7HJhfQBMV8euHIRzAWs3W+dkeZlUCDLy/poftFmpIuBvMhbTEd1TbelAD/ZOuEThZK75sSOFwtWBCUChwklLkTc/iEz0iSktR8gC6Lr6U/ajGW+ZJi3q/eF4soA+yvmjWUM9BGv0zHQvqHt20gzrWKrCXrWB31bBuyopXF29n2XLlHu28Dq7M8J23mfq2n6OmIbEyGtzkjHtGmZC8xIy9ll3l6lLWr7tf6TPzbW2HezL+2WA/UcM9I0w8GXbsvvm9O3tF4I27ch/dLuUX86YgVCOohlXRANBUM+2EsFbR4A0sxTnGlI08QHOLFMZ+5tICIDIhcoiCC8agd9H6cKrNSm5tlrFnQlgR29QFDq8rYzweXK4MJQYZ4Ef8zWgfolM2lfdWl2Zherlwds/EeV7HnxzbOvtxT0+bZG3wsCRPRtarvCh4kAm1YLJIftoLbVy6YXCLk/B9uHXj4M9Je1j+6FdIyJoinlEGzOyszbUL438rSyjjJLTcQTR9oOMynbRfmH7SNepa1si36c+2b9HiX7ylv9/vPtif5ezi9t5W2k1UjxmRN9hKeT+75Mb0Py8krcP7rtmq1D/TNv3yd8SuOjO89Fq+Sf9z1/2SH5HXJs7bQ8W/u8dL/l5xXb+bgsnj3jbS+/0NPKkvHZHMe1mC8XE1wEcwHLhLTy53Hk983pLJEvx8Z8cfbL19M1fdvI7ZhOJOdtz8tORUoQLwUR4AW0FZpi9o2JCy4q3L4Q0o6bndkUXDQ7XH5ZF/vRrfhNbhRdPk8u9NPl0NwHUibWjrYy0nycNW+L3oZorxOUHfvBtF60BWHbi7pU6PL6aIY8+l7ZNp7P2cn6jl4ipLPgab28bbTv2tjbGdtG17UZ21RIhx8aGypDcMlnpNvazNsv962949tSePFSuIdknfXjsq/W8+ZM8hlp5YcpaoNqPtjzBwMXsHafiTf3PTN7e5x9I+TLoaUEr37wx2lQk6ZxaVk5LlBOlk74+riAz+1vC5Za8+WBSGxzfOgPBUoxsHKBV9IeJhTtwz2IMmJmt1RHtLcipIfKDwFE1Q4S06wssoNmpriNfp8FYi4wi5zs9VRQhSCVle/LSHlIO7UyknOlQDETUaJPhkSdZVZrz0C/9f4jy+B1ymvEZSDN359bhox5gw2SZ/YLvi12OB8q+7sXV2nf276N/mPZB/FA7bRbHtAO9Jfl2d8Lhd9UcMyVtiS+El6Meb8L9nLe9KKB2xj2Y1va/Njak7Jn/RbuyezeF7ak+a29/F4p90fCq7CSx/EiOwa2mQ28zyjfoB9W+p3KsFvRf5F5vHc1u3vfsG2lz4fYMyW5XuWR9y/3IcdD9BO3qe1eUMZ9UWbGfUTfq/ea8+PUB6WtKW9FjLJ7tIVpLY3jpvhO+uzI+0M+N+pCeriMmp3Z9cIYlKULvjaHcJZl1mK+PNjkgSkXsFxI04wyG9uzX9wOQjmMky9/9MepmEnTuP5j5fSiUoy76bfXeTCct+nwNE7sODHF2qouqaU/v6SIPibGuBBx+61C2gmP1AZNKPdixzLTliJbwSXbI2yw/NIXIZFfKv6YPUoZpX6g5cfJ8zaMQ1e/pEKzLKRtuiBiyT9Ye5O66TMC5lvJ9cLMZ+x79mKk7wd2Ttph7WGil/d5z68XvtQfvLwgnFl7bD5iEbmxPLJPBZOW9tbSWBuqben5RN9IVmdQu6cS0gOrJ2rt0a43CWk5yPJjfWn3tEK09EDZynkeGJ1yv8brlLah7uHgHnym5cPvBbkSxl5r4z0sPtvKmLZdc9XJeWn7c9V7inLT9oWXjo3LbinvKezecp3E7ZgtCf5UwIdVM8mLg3nvuXFt4JMObXETj79Osb/s35GOYkwLYNd6ThU+JAYm3Nbav4QdqQ3+JYp8aVGzIy1D7/NaGVNcT+3wbZGz/Gma3NYp7FhLGbW2tlxfoi0tdsQ0/KVf3n8xXfs1COnCjMK4h+HwA3AtAU6tTWuxE3bMG+iBL5uRdve/F0t8lUobIwhpGlPaeG3Dr6lNdksrmtJVUMPjvc13TjyWaAtnfug+fQoihbSbjddWPvDVCxPuj7G/9GnJUBmnEM+8TgjpeoC9hHiwddSC/iXsyGxwM7npj7/V7MjKUGZ7a2VMcZ3bQSsb0tUQ6+n7KdpbK4PzOHS/VscU18fYVlo1MaYMmRZCGkK6X+a6RMCEOrYhJs69n/JA1Yrp+MOFbe2HkCaObby24fu+TfrnL9Te2vaceCzRlhrPluvOTm1p94Ii2trQYqtPk445rfm4qD3FPoT0esSUDOjl8RQipVaGrNMdWzHNvtc+qAwhpmtlTHHdtyXMRNOP5Ak71PayNFPYsZYyam1tub5EW1rs8Gls34o/n8f6rr2cdAyAkIaQ7h/8SwRMqGMbYuLc+6kWtJ57+8e275J41dracn0s30tP38K0lmYtDGt2Hnv9FOKZ1wkhnQbRWvC9hHiwdWh183NL2MHrK+3X7Cjl4+drZUxxndd36P4UdqyljEMZ8HxLtIXXd4p9COkFhPSxD07kry+lBCMwgg/AB+AD8IFz9wEuak+xDyFdF9KnCOZRJ/oFPnAaH4CQhpDuZ6TPPQBB+xBkwwfgA/AB+MCWfeAU4pnXCSF9mmAdIgnc4QPr9IFJhLTBPxAAARAAARAAARAAgVkJcFF7in0rpPEPBEAABEDAE4CQhieAAAiAAAiAAAiAwAYInEI88zohpDfgJDARBEBgMQIQ0ouhRkUgAAIgAAIgAAIgcDgBLmpPsQ8hfXjfIScIgMD5EYCQPr8+RYtAAARAAARAAATOkMApxDOvE0L6DJ0KTQIBEDiYAIT0weiQEQRAAARAAARAAASWI8BF7Sn2IaSX62vUBAIgsH4CENJr7aMPN6b75v1arYNdayQAn1ljr8AmEAABEJiMwCnEM68TQnqyrkwLwvM75YGjOgH4TJ3RAil0If3pzux3O7NL/t+a+8dHwwdUu7/soPre3HSdufkwgsyve3PddaYL/0flVap5/00sq+tuTCp1P5v9s850z/bms5K3+ZS9Obprs/815vj85jo7Z4znsVrBHdhfvzmKRoRw7ntH8/L+B97n7ihoHwiAwKUSkDHYJMeI+YruhJiviCa/cHQMkxd51meO5oWYbw3+MSCkdeEsB+3VC+me8gEivM/rd7yYleKZJ6oIaXfTpAKZ5/b7ZTvdgM5EujzOy9LPuHawcvRUE5w9epCYwIYpimjqtwkqmoJXKOPYF0YTtAZFgAAIgAAITExAxmCTHDshjZhPdhViPkmkcjxFDFOpYpHLiPkWwXwulUBIj+hJJ1yPWW7dcHMOilwukvj+iDbYpIN1jCzrIpI39NuaOKB/19QbsAUEQAAEpiMwiXCWqwshpNUOQsynYjn/k4j5zr+PJ2zhTELaz8zevKFl1TfmvVuuLJc9h6XJ/dJrZbaX8vVp5NLuMAtM14szreWZ3laeQ4Oqu1ayIYheWl7Ot+ky3MBtYOm6F0k35sYuIR8p6l1esjHZ8llyy8kec678uqeVtDdbbp/2a9pGm5/q4OnyOlr7RU/ny5Z1jxKZjf1mWdh6OF9Zr5F+nPQd5+DLSttEfsH7RN4HLMfGHgLMcuyCAAiAAAgMEFinkA7PKMR8vucaYwdj6Nle7nDEfGU26RXEfPxz0JQNjuYkMCCk02+kb+/z76PtgK4v7aaA3wrjIBKswE0C/NzpvRBhYjr7VtjnictWQz1MlDiBp4ppmbcVa7A/EZ7+O+lMLNVme5P2K/XXrrssxPZw4TksJmN7qX1Z+g83yXfqWb/1TfO2Ujn9afIJ9h14ud9irrF7md2h3ug/jSVW+qV/qUB+l6V/b26Yjxp3XRPCJV7U5zFPmbltk08/up2NOJAMBEAABEDgNATmE9KI+XyPxhiIT3rY/TyWqazwy2IB4TO16y45Pf8R8wl62SFivgwJTixAQBfSctnP/a374TFNTA8JaT/ocAFr98Ng4EQyE82usTytMTTTFzmk170gEWUUByaRNxbavOcEExdESs78RmaJiraFNCoTlt/uujKskJ93UE0eGDW7iu0qCUOlL2p1CAxNh9KuQ+uQ5YjK85cASvuSPCUuI84P2lQqJzECByAAAiAAAhsjMIuQRsynegFiPhVLiEO1GLQUeygx0aHxWMEkd1rGRYfWIcsRdSLmE0Au/LBNSD8+mLv9zuzfPTT+aje/mfgNZPf9zacLTp6P71Mv8bKMyZbL9rPG2g0u8lKRI7ZrGFTpBt7bXw+nGdARbbBJdfZUSAsnnyZ9W6sx1/rQ1qPUceiAR2YXtpYXvRTg+4Xk+umWQbXygsX1W++fpRUNJV7K+UGblPR6y3AWBEAABEBgQwQWEdKI+ZxHuOd25dk+GE8NPqcphhWTQcIXnQ3P9gYxnwBTOORxHt8vJNdPV/qtxS9cGsR8Ot8zO9sopO/N7W5aIe1FsBxAuMDSxAC/TrOzsoxSD4m8pWQD51tunqMG1crNSy8O3JJdlzaKxAGzs0uDNmoiNynB90vyfXbRbq0PbWFKX8wkpB0z98LB1tnqK0mDB96++nQ1v3C8k7pLXEacLzK3NvlysLRb9CMOQQAEQGDjBJYR0oj5rJvUnu02zWA8NficphhWm4QITuriovBJF2K+tjvXMkPM18YKqSYh0CSk79/ab2f0P41w8NLuIKZottC2xg1abJY1HaC8+LKzoFEgeMGQiLoiFkW88bRhkMr/NnRMdPSgqrQ5lm73SkLKXgvtZ29HvUAbGITTwuORG5xL+SqcsjaEPlCXmpfao9RRENKOedLnsRlte8GGZ9f9zHRbPp7K28t9NblqVwewfuHX7L7q1+r3ViVeyvmhB/TQNWkcjkEABEAABDZDYAkhjZjPuwNiPstBxj8+HtE/L1RiFYcSMZ/+jX2Jl3J+KK4buraZkW27hupCOnwTvduFH594e58t6abB/HAhTW/j/DJXt0yYiWiPlG5Yn+bmgz+OQtqmStPIcrzYZHWEpRaZKHKOaNOVZy3Lg6ofJFzdyVIOpazwhpHSSjvSlwfRsbyglOWFtmfcYr7Sni+PuHBRrQx4shDRhpsPNg8rQ1yntkaxqdRRENI0C3/oMnZr+sEvHHi7RZt4v5X9ggoQ/vHN+/T7f1F2zsv3M6/TfyvPmFNV1N4DfIIVgV0QAAEQAIEVEqDYa9ItYj61p8vPdvFM7+M+GaPR8m2KtfJVhIj5BHqKh46IYRDzCaY4nJWALqTlD08MHOtCelabz7xwP0AnounMWzzcvPDAGpjxHc5fWXpVy7y163gzubUeg70gAAIg0ExgUgE9ENuV6kHM19xVjQkR86WgEPOlPCpHiPkqgOa/DCE9P+PxNbg3cvps4/jCtp3Dv1nky/lHtueiBhll5nokLiQHARAAARBYL4GSwF3qPIT0DL6BmK+HipivR9Gwg5ivAdLsSSCkZ0d8YAV2YB0xC0uDT78suF9qFJYUHbFM5sAWHJktvJXkS8bHlEjLg0rfV/dL+eOSq5TdBl9kjPSZMTiRFgRAAARA4PQElhLMpXogpGfygZHPb8R8oh8Q8wkgOFyKAIT0UqRRDwiAAAiAAAiAAAgcQaAkcJc6DyF9ROchKwiAwNkRgJA+uy5Fg0AABEAABEAABM6RwFKCuVQPhPQ5ehXaBAIgcCgBCOlDySEfCIAACIAACIAACCxIoCRwlzoPIb1gZ6MqEACB1ROAkF59F8FAEAABEAABEAABEDDFP0UKIQ3vAAEQAIHlCUBIL88cNYIACIAACIAACIDAaAJLCeZSPZiRHt1lyAACIHDGBCCkz7hz0TQQAAEQAAEQAIHzIVASuEudh5A+H19CS0AABI4nACF9PEOUAAIgAAIgAAIgAAKzE1hKMJfqgZCevYtRAQiAwIYIQEhvqLNgKgiAAAiAAAiAwOUSKAncpc5DSF+u76HlIAACOQEI6ZwJzoAACIAACIAACIDA6ggsJZhL9UBIr84lYBAIgMAJCQwL6ftbs9vtwv+9ufv0mP1iJAbVE/YeqgYBEAABEAABELgYAiWBO8l5xHwX40doKAiAwDQEikL6/q0V0Lfm/jEXz3zAhpCepiNQCgiAAAiAAAiAAAgMEeDx15T7iPmGqOMaCIAACOgEdCH96c7sG0S0HcQhpHWwOAsCIAACIAACIAACUxKYUjz3ZSHmm7KLUBYIgMAFEVCF9MO7vdm9vc+WcfeDLpulhpC+IG9BU0EABEAABEAABE5GQIvDjj2HmO9k3YmKQQAENk5AFdJ2ic/+3b2529P30buisIaQ3rgHwHwQAAEQAAEQAIFNEDhWNGv5EfNtouthJAiAwAoJFIX0bsd/XOze3O6suH7IZqkhpFfYqzAJBEAABEAABEDg7AhoQvjYc/77aMR8Z+csaBAIgMDsBIpCWopmt/Rnf2ce2LJuO3hDSM/eR6gABEAABEAABEAABLLJjGNFtM3vZ6TTiRLEfHA2EAABEKgTUIW09r2Mdg5Cug4YKUAABEAABEAABEBgCgJTCGdZhhbfaecQ803RgygDBEDgnAioQvrR/YJjvszn9j7/U1iYkT4nd0BbQAAEQAAEQAAE1kpAiuBJjhHzrbW7YRcIgMDKCehC2i7fvr81u138sTFNROPt5Mp7F+aBAAiAAAiAAAicDYFJhLP4RM+ViZjvbHwEDQEBEFiOQFlIawOtcg4z0st1FmoCARAAARAAARC4XAKzCWklvtPqQsx3ub6HloMACOQEIKRzJjgDAiAAAiAAAiAAAqsjoInbJc9BSK/OJWAQCIDACQlASJ8QPqoGARAAARAAARAAgVYCS4pmrS4I6daeQjoQAIFLIAAhfQm9jDaCAAiAAAiAAAhsnoAmbpc8ByG9eRdCA0AABCYkACE9IUwUBQIgAAIgAAIgAAJzEVhSNGt1QUjP1bMoFwRAYIsEIKS32GuwGQRAAARAAARA4OIIaOJ2yXMQ0hfncmgwCIDAAAEI6QE4uAQCIAACIAACIAACayGwpGjW6oKQXosnwA4QAIE1EICQXkMvwAYQAAEQAAEQAAEQqBDQxO2S5yCkKx2EyyAAAhdFAEL6orobjQUBEAABEAABENgqgSVFs1YXhPRWPQd2gwAIzEEAQnoOqigTBEAABEAABEAABCYmoInbJc9BSE/coSgOBEBg0wQgpDfdfTAeBEAABEAABEDgUggsKZq1uiCkL8XT0E4QAIEWAhDSLZSQBgRWReCz2T+7NvtfV2XUpMZ8fnNtrt98nrRMFAYCIAACWyegidslz0FIb92DYP/2CCDmW3OfKUL6wdztd2a3U/6/vTdywD7HQdUG8d2zvcnC+A83pus6c/NhzV16AtsclxvzfkTV77/pHEvLU2U9oqxlk743N6oP2IFuibaEer7htL1NXXLOGOfH3VYFd4nzsr2N2kAABEBgTQRkDHb8MWI+xHwjPRwxnzEGMd9Ir6kk327Mpwjpx0wsPz76gXb/7iG7BiFd8Q1++de9ud6ssOENEfsHDKpUQvEBRglWty3d7McOqqVyUwBFXs632EseeZwWM3DUZsdAAU2X3IsUIfyzjM6vtvoiIGsNToAACIDA0QSOF85ajCfPIeZzHeWeQey5Orb3EPNlxIoxTJZyLSdKMRFivjE9dM4xX5uQvr81u92tuX+Ug+2juSghPcZrtLQYVDMq5zOoZk0beaI0WPNihtO4gSqspOD7vIT6/nAd9fxtKZoGVWNMa7q2WpEKBEAABLZNYBEhjZhvGidBzJdxRMxHSFpireE0PM7j+1RD23a4jrYy6qlaY7nWdPUal0vRIKTDsh9lWbcd0HUhrS0/NcaMnLl0N9w378MS1bAUWCy5ttDtt5QurV0m3PnjBGF4q+iWEQ8tyw35s+XGYXavnD/UJurplyyL/FSO3S76Hai179ne7MOy6us37/1yZMlEtkOZOXTOznl1Ymm3aHOpnYcPqvbmt7OVfhAgpsmye2eDtYunEXbS8hxqi/Av17OSh+CVsNDyu0LCPUH19GVw29hyd5tOcg/9l31y0Du7L+vmG/sJwtiZ3HY7kvbKfre2iL7vWJo0L2+v7JfQKMe+cK1vN3ZAAARA4DIIzC+kEfPJZ1gSV3A3k7EBPf+zZ2B81pViIV7sZPshZkDMx4mKWIz6jCeR/drHaz5REsdo+V0yUU9fRnus5TRTsXxbCWI+3m2n2q8L6U93Zr/bm7tP+Wx0WUhrotk71ZhBpBfHvaDwTsPL6B2anM0NYExEuBuCHYcBLg6Mwdn7OsK3pVRe0jPBabVvpGU9Sb5wIG3T0hTPlW++2JZiZn8hDA6WH7G1eR1Dan/WjpyPS8/5uDxM7GSM834jS48T0vbhFPvWtym3w4psYpS0lUQ0tZ1mQLO2xTr6gUvxgXJbcobU/rgd8K2QKLU95kz2Qh9nIjxJNHQwbIfs+4w5DewKH15rU1tcBmsP589LwT4IgAAIXBaBWRW5YQAAEV5JREFU2YU0Yj7mUAPPwyxWYtloFzEfkei35TipT1LY8X2BmE/gQcwngCx/WBXS9293ZleYjR4U0jKgPmBA0W44d25I+Ih6bcDOhbdFnATxzi4mvuxPCJR+bEyUHbur8SXBAQxiHRPsMcHL28iZJmyoSpbPC0khbJLrgR/rI1sMr4+KHTrP0+j72gNOiC7HO4rorD6l7/2b6Ni+3H+0er2FpTa2rcQolxtKd6sHpC9LNq7/tNlsmbB4PGSH4OvKkOn9cU3Iq36m2iTLVxPhJAiAAAhcBIG5hTRiPu5GpecPYj6+0ozHkESvFA+VzlO+8lbrC3suxmu0koAmTmxZSX2I+RS8GldKJvi60zK9P77kmG9YSFfeTA4L6VRQaTcadVVpm9wAIZE8NxyQh5lAtpyWlgD3nS5EoK1G1hHtkw5EV0rn6XrYHiWkg7MqbeGDhqgxPWRt5W10+074Fh4O3G6+T6Wzcu2pXsxJW/lMb8jL7aDi2rYac3FOs5UX7uyOS6563+gHZo2HqIOVV2pL6TzLGpfoFGdyNVvSEvqHyBv7o3bpCwSRcuCw3D4qP3KK7FIf9GVQOk38D9+33LwBe3gy7IMACIDABRCYVUgj5hMeVHr+lM6L7LUYRCRPD9PnKD1P7TZ93qa5kiMWm/E4xO0j5ot/OQYxn+5Tzn9jnFf2wdRXLy3mGxTS7s3k/s48KD8yRoO5/o10uJVdJ9jZXisC2Fuj5E4vH/Abn1LJAFweUzra2utap9J1bbZQq9enLw2eDSLHFnDUoNpbfPhOdVAVs/VUE8untoFfpxcRYkaaipLbMmuZUh4rfSH5ymNZRO+f8gIda/2q1BuSF9si+FDp6bZcLqUb9nVvK32X79Kyb5OpjPp2yA57beR97BjnD/7htnArD6iTZ8c+CIAACJwRAYq95tgi5pOOUnoearGBzIuYTyEyMFGlpebnlL6QMZ485tntPmI+SaQyiXNA/HWBMd+AkL43t7udub3Xv42mQXxQSIfZyetn1/kPJyndKU9lwkTpoGpA7kTMUPCf3pyuTjuTqsyeVr+PrYoMX9egsJcQpjxmgo6zdfskfDNe0mbxAHHp7Rsrtjxe6adSM7gdWhovCLWXIWm/2bwuLe83Z8dQ3wfxSW1XDEjt83WW3ginaXlhId9APf3fJBxKw/qPl273Xd2J/7XUKUtxJfkfoCvYkTHWikjO5f3karGfT3CfSfKwg4E2s1TYBQEQAIGLIECx1/RbxHz5D3nqzy/raPkzV3M/nx8xX2RTjpN8GsR8kZU20UdXc/9DzEdslt4WhXTLm0k7kNeEtHeEfEaqpaHeUfiyglwUuZuuEPT3dfRiL5aVLI3h160Qs8dMkOV2+HLk4JilY2WUbJFl9Onm2GGixNka7HP7jKFsR2ZjEMp+mceNee+OmZC2tidpJK9ww8ul34qwqg2qfKlJv1yf2FWFtE0YxDS3Jem39PrNB38c/ae1LXm6WEYwWDJjfeJT+DJK+bJ+Cn6dpSc+pW3FDuqTyJ71vcxb/GX6lGtJVDfd36V24DwIgAAInBmB6QW0nyxBzLc3JKRlDETPOvmMzdIlsUNwPB5fFp+HMzkpYj4FrIw95MRZeh0xn/a5JmI+7lhFId06WLcJaQad117Zd4OUNjBV8uHyJRAoiMozb7p/cB92P20ODQsCNmc7DAYBEACBGQi0xmZzpUPMN0OnosgGAoj5GiBtO8lGY76ZhfRxjg8hve17Yl7rj/OteW2bs/TwtjSbrZ6zzlOUfan9ewrWqBMEQGArBOYSyK3lDgvp48ZtxHxb8cJT2Hmcb53C4mnqRMw3Dcf5SplHSLPlnXIpjGsKu05LZtKtX8KNQXW+jt9+yZc6qNqeswNr/plDuU89q/Qei5852POjl3+XK5vkir331bFjktJRCAiAAAhsk0Cr4J0rnSqkWUynjtvsuv4cQsy3TW9c0mrEfPtfW3kj5mslNUW6eYT0FJahDBAAARAAARAAARAAgZ7AXAK5tVxVSPfWYQcEQAAELosAhPRl9TdaCwIgAAIgAAIgsFECrYJ3rnQQ0ht1HJgNAiAwCwEI6VmwolAQAAEQAAEQAAEQmJbAXAK5tVwI6Wn7E6WBAAhsmwCE9Lb7D9aDAAiAAAiAAAhcCIFWwTtXOgjpC3E0NBMEQKCJAIR0EyYkAgEQAAEQAAEQAIHTEphLILeWCyF92v5H7SAAAusiACG9rv6ANSAAAiAAAiAAAiCgEmgVvHOlg5BWuwUnQQAELpQAhPSFdjyaDQIgAAIgAAIgsC0Ccwnk1nIhpLflL7AWBEBgXgIQ0vPyRekgAAIgAAIgAAIgMAmBVsE7VzoI6Um6EYWAAAicCQEI6TPpyNM147PZP7s27X8o/nSWoubtEPj85tpcv/m8HYNhKQiAAAgsQGAugdxaLoT0Ap286ioQ8626ezZq3JZjvqKQfni3N7vdLv5/e2+0gRaD6ka9dhKz7YDame6b95OUhkLWRsD3ryZo33/Tma67MfP1/Htz03Xm5sPamMAeEAABEDgdAS0Om+IcYr7T9el2akbMt52+OsRSxHyHUFOFtBtQ93fm4fExiOd7c7vbmf27h0xMQ0gfgv088tg3SN2zvZlt3vDXvbnu5p/tdqIQLwMUp5xvUG1i/uHGdAv0v9JwnAIBEACBVRKYQjTLMhDzrbKrV2cUYr7VdcnEBiHmOwSoKqTv3+7MLpmBfjB3ewjpQwCfb54FZgwhpE/sPuVB9VjDmoS0MaY13bH2ID8IgAAIbIGAFMFTHCPm20LPn9pGxHyn7oH560fMdwhjVUg/froze7usO4hpv+Tn1tz3M9Q0U/1oBmeknRCyS0DDf2320s06DaXxHduXkSz3VG5sVx5bcupssMc+rS+HXXfUZB3aLKhME5edurd02TJXxbZaD0leWZnGmFoayVPMtDpb7TmeTqSRdWhLe11+rT9N4CTLlP0yxCJrY/SP1BbRJ5o9vJ3WD1kaJ9LIN5Ot9I8hY2vXvB+kdhsz7s2ub+fNh7S9aZm2Huu3PE3ux2mblXYq7Hk93tepP5T8AUdaT9d/7yzPx/u6UNYYv6l1Ba6DAAiAwMYJTCGcszIQ8ym/88Kfpf6ZR58aIebjN1HghJgvQEHMZ+M6ihsvIebThbQTzH4W2n0nnSzzjiLaDsZlIf3Z7L/hy34V53JBch7sx1u0cIP2CXyZNLi50zLwZsKA0rmOZTf95zc3ySDqrjPBZUricMCOcULJFqS0pS8/7IS2UDvkZS+OOc+cXxRCJFxEvVkdSr/VZgplHwR+dGNldpdOOFt4e3jCvG1Zvw3mj2VJf4hXptnLfUEwr1YT2spfImX3ji+TD2CyXsnH+wL5QXxJE/3L16v2W9bHsRGynngl7rUzt+0q+UAsD3sgAAIgcAkEMhGsTHAclgYxX/xMLY8vUt/Kn+HyeZum147yMrJUWTwmUmRxQG63f87bFwH0rBf1ZnX46/K5P/jMzuKBgdhBNCE5HIzZ8rZlscZg/ljTYFtisoP3cl8QzKslh7Yi5quSWkOCgpD230T7GWkaXPfm7lMqooeFdN681HkbbrTs5pRlKs4p82SDRMNsoCxDHkszMmHZ0LasDN+WoR/uSvllBejLYIXtmXgSItddZy8ZbC35oFBrn+iXxsEta9FQPneNHgwhp0zvjuObsaz8cKLGtZSv+by0S/RJvRyNtz8XRa9nnjz8knrsdSlI037K+16rN1iblM1aINvKLvHdduapjbwM7IMACIDApRE4TCTnsVtaDmK+KDRNWLEn4gvhaOkzbOBZKfLFQ/9sQ8wXifjVkDJOCdddbCH6RMYb7hgxn/8RWMR8zLNm3VWFtPteJpmFDmI6+W7aD8zlGekgwJIls/wXnusBci7gJAulDBngyxtNFmGPw80Xl5nyt3eakFQKcWWEm5zvK0nLp8LAGpglokgI3ryMwkAu2l9j6h4Oss/ssTJDn9qXWsRFGd9PU1WOhO1JatfPtMSYb8Ug7MqI16PwjKWlD8R4fso9Wwfx4vttdWh9K88p9wIvXHDgvk5Mcg6yDlagvM/oUuk8XQ/bvC6RoD+stKtPhx0QAAEQOH8CqQCuCeS264j5EPPxmMDtI+Y7ajDhcR7fbytUi73kuUpshJivDfUEqRQhrYvm/McoKkLaBdSpqEmDZ+kUSmuqQbniSDLPkBhzVfoySOS4U7IMeayYSsu/rSixwjEpT01fORluAhI5NnXKL8+vXhe214R0q+hV6+ImOfvtiwXbz6kf8GSD+0N915c/WEJ60bHIbam2JS3lsCNbt3swWX8Tb1WrJWr3ivR9eSwLtdfztvNUOQet3pBD+FVfzlCf9YnqvhyT1u2OabEHAiAAAudNYHohjZjPfxbHnsul51viWv75iJgvQOljMsR8jghivuRuyePL5DI72F7MpwjpR5P9uFj4IYoxf/7KCTIuGNzAxGekacZ6KLj34qC89EUE+u5GTt8sDi4VsV0X8kTBGurkttMPlYklz6zn/a67ca7NdZI3S9V4wtsR7aLlRvFHzrKCHGPO05fBRX1NSOc8slr8CVcXe/AoyeyNc/3s+oi/M53bH6vxfV/2jZiy3yuIvMxX+wxxxw0C/HuVeKlxL/jqs0Nesgg/p+X2iZ8p/iIsc21I3jKLBEmfBr7sRyOS1ElafiXkG6pHtZ+XwfaL9bA02AUBEACBCyEwvZBGzHfjVuHxeMY/T6vxhX0+2Wd68iw+1BGVZ7h7/iHm80RDbFGLwzl+xHxiNSmHQ5qC/B4xn6DTfKgKaTtQuxlo+8vd4b8mom268tLu2Cm0TOS9/bvD4ibwIiYuvU2XENt2hAGNLTfWxKVflnJj3rsbhxyDhDIXlzmb1IZrs/9g/34xK8NlqdjB04g25jUqZ4Kg58truADuc4SBNaZL7UzbEpcTU353vSJySEzHOvJyqF+SvqBKaFt7CFC6oa1ob8pE+Jhcgi7y2vbo9spyUqbOPCqrxm6gLb5vhn1Rzy7tk0vtbS7lIawURi8EYt+mbeXXLWt7HJkrdrj7UrYpTxfLIKNkmtQOSuXsOeR+ogKwBQEQAIEzIjCHkEbMh5jvmsXYNj7In9kNMQZivmSkQczHcZxvzFcU0q2DdVlIc4CXst8w0JwRCj9I6ALINdMNqgPXN8UivEg5QtQ1vcRQmfgBKH+wqYnP5+RZ+c/5dAtaAgIgcDoCrbHZXOkQ8/G+R8zHaWRL5JOLWztAzLd4j2005oOQntBTLm/2LLxhUsXleT1g/EuD0ox2gxMVlhg15LQLoc3+mfaGuC33NlOdl/9ssw9gNQiAwNoIzCWQW8uFkI4egZgvsmhdFcdzrHkfMd/SvbPdmA9C+mhfYcsVCst+3WArls3EpbXpd+NHm7N4Abb9bGkvW6KuzqCy6wmDng8ra/G2aBWGt5KVH+rScrpzYamTbau2rJwGa50FLeG+PCFtuaj+UwSNCyAAAiBw/gRaBe9c6SCkEfMh5hsYZxDzDcApX9pyzAchXe5XXAEBEAABEAABEACB1RCYSyC3lgshvRpXgCEgAAIrIAAhvYJOgAkgAAIgAAIgAAIgUCPQKnjnSgchXeshXAcBELgkAhDSl9TbaCsIgAAIgAAIgMBmCcwlkFvLhZDerOvAcBAAgRkIQEjPABVFggAIgAAIgAAIgMDUBFoF71zpIKSn7lGUBwIgsGUCENJb7j3YDgIgAAIgAAIgcDEE5hLIreVCSF+Mq6GhIAACDQQgpBsgIQkIgAAIgAAIgAAInJpAq+CdKx2E9Kk9APWDAAisiQCE9Jp6A7aAAAiAAAiAAAiAQIHAXAK5tVwI6ULH4DQIgMBFEoCQvshuR6NBAARAAARAAAS2RqBV8M6VDkJ6ax4De0EABOYkACE9J12UDQIgAAIgAAIgAAITEZhLILeWCyE9UUeiGBAAgbMg8P/UhJ1SUHBfPwAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that changing models requires only **two lines to change!** \n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "This is an example of a framework which extracted a common api, rather than a packages which throws together a bunch of algorithms, written by people who are not communicating with each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try different transformations\n",
    "On my machine, the `SVC` (the svm classifier) produces the worst result. Let's try to improve it by cleaning up the data a bit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_df.survived\n",
    "    , random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notice how similar the transformations are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import preprocessing, impute\n",
    "\n",
    "X_train_transformed = X_train\n",
    "\n",
    "#Fill missing values\n",
    "imputer = impute.SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)\n",
    "X_train_transformed = imputer.fit_transform(X_train_transformed)\n",
    "\n",
    "#Scale values\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_train_transformed = scaler.fit_transform(X_train_transformed)\n",
    "\n",
    "#You don't even have to know how a transformer works, in order to use it!\n",
    "pt = preprocessing.PowerTransformer()\n",
    "X_train_transformed = pt.fit_transform(X_train_transformed)\n",
    "\n",
    "model = svm.SVC(C=1.0, kernel='rbf')\n",
    "model.fit(X_train_transformed, y_train)\n",
    "\n",
    "model.score(pt.transform(scaler.transform(imputer.transform(X_test))), y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformers, Estimators and Predictors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn has three main interfaces: **Transformers**, **estimators** and **predictors**.\n",
    "\n",
    "Objects are created using a constructors, such as `StandardScaler(..)`, `SimpleImputer(..)` or `SVC(..)`.  Once created, objects will have some standard methods.\n",
    "\n",
    "**Estimators**\n",
    "```\n",
    "model = SVC(...) # DecisionTreeClassifier() or RandomForestRegression(), etc.\n",
    "```\n",
    "Standard methods:\n",
    "\n",
    "The `fit` method does the actual training\n",
    "```\n",
    "model.fit(train_inputs, train_target)\n",
    "```\n",
    "\n",
    "The `score` method returns the accuracy of trained model\n",
    "```\n",
    "model.score(test_inputs, test_target)\n",
    "```\n",
    "\n",
    "**Predictors**\n",
    "\n",
    "Predictors are objects which, given an input, generate output based on the learned model. Hence, the most important method is `fit`. Given a list of inputs, perhaps the X_test matrix, this method will return corresponding (predicted) classes. The degree to which these classes match y_test is the accuracy score.\n",
    "```\n",
    "model.predict(target)\n",
    "```\n",
    "\n",
    "If you are not interested in the predicted classes, the probabilities attached to each class, you should use the predict_proba function (note that this function is not always implemented). Given a list of inputs, perhaps the X_test matrix, thsi method will return the probabilities associated with each guess. For example, if the expected result is male or female, each line in the input will correspond to two entries, one for the probability that the correct value is male and one for the probability that the correct value is female.\n",
    "\n",
    "```\n",
    "model.predict_proba(target)\n",
    "```\n",
    "\n",
    "\n",
    "**Transformers**\n",
    "\n",
    "As we have already seen, even transformations, such as imputations, scalers, vectorizers, etc. have a standard interface.\n",
    "\n",
    "```\n",
    "transformer = StandardScaler() # or OneHotEncoder() or SimpleImputer() or PCA()\n",
    "```\n",
    "\n",
    "Although transformers have the `fit` method, the `fit_transform` method combines the fitting and transforming steps and is used much more often.\n",
    "```\n",
    "transformer.fit(train_data)\n",
    "transformer.transform(data)\n",
    "```\n",
    "\n",
    "`fit_transform` processes the input data, such as finding the min and max values of a dataset, which requires reading the whole input, and transforming the data in the same step:\n",
    "```\n",
    "transformer.fit_transform(data)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model performance metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a model has been trained, scikit-learn provides many ways of assessing the quality of the model. Let's bring back an earlier model. Notice the estimators come with a `score` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_noempty_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_noempty_df.survived\n",
    "    , random_state=1)\n",
    "\n",
    "#training\n",
    "\n",
    "model = ensemble.RandomForestClassifier(n_estimators=100, min_samples_split=2)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this was a classification problem, you can generate a confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "\n",
    "y_predict = model.predict(X_test)\n",
    "confusion_matrix(y_test, y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Bonus: Visualizing confusion matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.heatmap(confusion_matrix(y_test, y_predict), annot=True, fmt=\"d\", cmap=\"YlGnBu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A recent scikit-learn visualization library Yellowbrick looks very useful:\n",
    "https://www.scikit-yb.org/en/latest/gallery.html\n",
    "\n",
    "They have created a fourth important interface: **Visualizers** (similar to Estimators, Transformers and Predictors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full classification report can be generated via:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, regression matrix, such as `means_absolute_error`, `mean_squared_error`, `r2_score`, etc. are also available from `sklearn.metrics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Tuning\n",
    "\n",
    "Of all the models we tried, SVM performed the worst (at least on my laptop). However, we have not done any tuning of the model. Scikit-learn provides two main methods of hyper parameter optimization: `GridSearchCV` and `RandomizedSearchCV`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {'C':[.01, .1, 1, 10], 'kernel':['rbf', 'linear', 'poly']}\n",
    "\n",
    "grid = GridSearchCV(estimator=svm.SVC(), param_grid=params)\n",
    "\n",
    "grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(grid.best_score_)\n",
    "print(grid.best_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "\n",
    "Notice in an earlier cell how several transformers worked _very_ similarly:\n",
    "\n",
    "```python\n",
    "#Fill missing values\n",
    "imputer = impute.SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)\n",
    "X_train_transformed = imputer.fit_transform(X_train_transformed)\n",
    "\n",
    "#Scale values\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X_train_transformed = scaler.fit_transform(X_train_transformed)\n",
    "\n",
    "#You don't even have to know how a transformer works, in order to use it!\n",
    "pt = preprocessing.PowerTransformer()\n",
    "X_train_transformed = pt.fit_transform(X_train_transformed)\n",
    "```\n",
    "\n",
    "The code is so similar that we shouldn't have to repeat ourselves. `Pipeline` class in sklearn will take care of the common parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import pipeline\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_df.drop(['survived'], axis=1)\n",
    "    #titanic_onlynum_noempty_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_df.survived\n",
    "    #, titanic_onlynum_noempty_df.survived\n",
    "    , random_state=1)\n",
    "\n",
    "pipe = pipeline.make_pipeline(\n",
    "    # Transformers\n",
    "    impute.SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)\n",
    "    ,preprocessing.StandardScaler()\n",
    "    ,preprocessing.PowerTransformer()\n",
    "    ,svm.SVC(C=1.0, kernel='rbf') # <== Classifier\n",
    ")\n",
    "\n",
    "\n",
    "pipe.fit(X_train, y_train)\n",
    "pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gridsearch over whole pipeline (include transformers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import pipeline\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_noempty_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_noempty_df.survived\n",
    "    , random_state=1)\n",
    "\n",
    "pipe = pipeline.make_pipeline(\n",
    "    # Transformers\n",
    "    impute.SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)\n",
    "    ,preprocessing.StandardScaler()\n",
    "    ,preprocessing.PowerTransformer()\n",
    "    ,svm.SVC() # <== Classifier\n",
    ")\n",
    "\n",
    "\n",
    "params = {'svc__C':[.01, .1, 1, 10], 'svc__kernel':['rbf', 'linear', 'poly'], 'standardscaler__with_mean':[0,1,10]}\n",
    "optimized_pipe = GridSearchCV(pipe, param_grid=params, n_jobs=-1)\n",
    "\n",
    "optimized_pipe.fit(X_train, y_train)\n",
    "optimized_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_pipe.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column Transformer\n",
    "There are times when you need to apply a transformation to just a few columns of a dataframe. `ColumnTransformer` helps you do exactly that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "\n",
    "col_transformer = make_column_transformer(\n",
    "    (preprocessing.StandardScaler(), ['age', 'sibsp','fare']),\n",
    "    remainder='passthrough'\n",
    ")\n",
    "\n",
    "pd.DataFrame(col_transformer.fit_transform(X_train), columns=['age', 'sibsp','fare']+['pclass', 'parch', 'adult_male', 'alone'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_transformer.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "\n",
    "col_transformer = make_column_transformer(\n",
    "    (preprocessing.StandardScaler(), ['age', 'sibsp','fare']),\n",
    "    (preprocessing.OneHotEncoder(), ['adult_male', 'alone']),\n",
    "    remainder='passthrough'\n",
    ")\n",
    "\n",
    "pd.DataFrame(col_transformer.fit_transform(X_train), columns=['age', 'sibsp','fare','adult_male1','adult_male2', 'alone1', 'alone2']+['pclass', 'parch'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline with transformers (whole thing optimized with grid search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn import pipeline\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    titanic_onlynum_noempty_df.drop(['survived'], axis=1)\n",
    "    , titanic_onlynum_noempty_df.survived\n",
    "    , random_state=1)\n",
    "\n",
    "col_transformer = make_column_transformer(\n",
    "    (preprocessing.StandardScaler(), ['age', 'sibsp','fare']),\n",
    "    (preprocessing.OneHotEncoder(), ['adult_male', 'alone']),\n",
    "    remainder='passthrough'\n",
    ")\n",
    "\n",
    "pipe = pipeline.make_pipeline(\n",
    "    col_transformer\n",
    "    ,svm.SVC() # <== Classifier\n",
    ")\n",
    "\n",
    "\n",
    "params = {'svc__C':[.01, .1, 1, 10], 'svc__kernel':['rbf', 'linear', 'poly'], 'columntransformer__standardscaler__with_mean':[0,1,10]}\n",
    "optimized_pipe = GridSearchCV(pipe, param_grid=params, n_jobs=-1)\n",
    "\n",
    "optimized_pipe.fit(X_train, y_train)\n",
    "optimized_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_pipe.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save/load models\n",
    "Once a model is trained, it can be saved to disk. The saved file contains the weights, as well as details about Python classes which can load them and allow clients to `predict` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from joblib import dump\n",
    "\n",
    "dump(optimized_pipe, 'model.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load model from disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from joblib import load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trained_model = load('model.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trained_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(y_test.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [notebook](110-scikit-learn-run_saved_model.ipynb) for a \"clean room\" demonstration of code which loads a scikit-learn model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End notes\n",
    "\n",
    "#### PyCaret\n",
    "http://pycaret.org\n",
    "A higher level machine learning library\n",
    "\n",
    "#### SKORCH\n",
    "https://github.com/skorch-dev/skorch\n",
    "Scikit-learn compatible deep learning library\n",
    "\n",
    "#### awesome-scikit-learn\n",
    "https://github.com/fkromer/awesome-scikit-learn\n",
    "General links to the scikit-learn ecosystem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}