15 Ensembles y Optimización de Hiperparámetros.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "  Según distintos benchmarks (papers, kaggle.com) los **algoritmos de uso general** que tienen más veces la mejor performance son: **Gradient Boosted Trees, Random Forest y SVM**.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../vol/img/benchmark_algs.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "model = DecisionTreeRegressor(max_depth=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = pd.read_csv('../vol/intermediate_results/X_opening.csv')\n",
    "y = X['worldwide_gross']\n",
    "X = X.drop('worldwide_gross',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,\n",
       "           max_leaf_nodes=None, min_impurity_decrease=0.0,\n",
       "           min_impurity_split=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "treedot = export_graphviz(model,\n",
    "                         out_file=None,\n",
    "                         feature_names=X.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'digraph Tree {\\nnode [shape=box] ;\\n0 [label=\"opening_gross <= 41613376.0\\\\nmse = 4.4919943637e+16\\\\nsamples = 1665\\\\nvalue = 141540319.054\"] ;\\n1 [label=\"opening_gross <= 22074048.0\\\\nmse = 1.33338221931e+16\\\\nsamples = 1506\\\\nvalue = 92999937.199\"] ;\\n0 -> 1 [labeldistance=2.5, labelangle=45, headlabel=\"True\"] ;\\n2 [label=\"mse = 4.9236662412e+15\\\\nsamples = 1257\\\\nvalue = 64781848.271\"] ;\\n1 -> 2 ;\\n3 [label=\"mse = 3.147813102e+16\\\\nsamples = 249\\\\nvalue = 235450289.735\"] ;\\n1 -> 3 ;\\n4 [label=\"opening_gross <= 70351576.0\\\\nmse = 1.10398118716e+17\\\\nsamples = 159\\\\nvalue = 601300162.289\"] ;\\n0 -> 4 [labeldistance=2.5, labelangle=-45, headlabel=\"False\"] ;\\n5 [label=\"mse = 4.06753884592e+16\\\\nsamples = 92\\\\nvalue = 440868287.554\"] ;\\n4 -> 5 ;\\n6 [label=\"mse = 1.22264857987e+17\\\\nsamples = 67\\\\nvalue = 821594676.851\"] ;\\n4 -> 6 ;\\n}'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treedot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"923pt\" height=\"269pt\"\n",
       " viewBox=\"0.00 0.00 922.50 269.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 265)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-265 918.5,-265 918.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"567,-261 328,-261 328,-193 567,-193 567,-261\"/>\n",
       "<text text-anchor=\"middle\" x=\"447.5\" y=\"-245.8\" font-family=\"Times,serif\" font-size=\"14.00\">opening_gross &lt;= 41613376.0</text>\n",
       "<text text-anchor=\"middle\" x=\"447.5\" y=\"-230.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 4.4919943637e+16</text>\n",
       "<text text-anchor=\"middle\" x=\"447.5\" y=\"-215.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1665</text>\n",
       "<text text-anchor=\"middle\" x=\"447.5\" y=\"-200.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 141540319.054</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"439,-157 200,-157 200,-89 439,-89 439,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"319.5\" y=\"-141.8\" font-family=\"Times,serif\" font-size=\"14.00\">opening_gross &lt;= 22074048.0</text>\n",
       "<text text-anchor=\"middle\" x=\"319.5\" y=\"-126.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1.33338221931e+16</text>\n",
       "<text text-anchor=\"middle\" x=\"319.5\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1506</text>\n",
       "<text text-anchor=\"middle\" x=\"319.5\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 92999937.199</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M405.942,-192.884C394.205,-183.531 381.316,-173.259 369.154,-163.568\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"371.289,-160.794 361.287,-157.299 366.927,-166.269 371.289,-160.794\"/>\n",
       "<text text-anchor=\"middle\" x=\"364.096\" y=\"-178.441\" font-family=\"Times,serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"696,-157 457,-157 457,-89 696,-89 696,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"576.5\" y=\"-141.8\" font-family=\"Times,serif\" font-size=\"14.00\">opening_gross &lt;= 70351576.0</text>\n",
       "<text text-anchor=\"middle\" x=\"576.5\" y=\"-126.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1.10398118716e+17</text>\n",
       "<text text-anchor=\"middle\" x=\"576.5\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 159</text>\n",
       "<text text-anchor=\"middle\" x=\"576.5\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 601300162.289</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M489.383,-192.884C501.211,-183.531 514.201,-173.259 526.458,-163.568\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"528.713,-166.247 534.386,-157.299 524.371,-160.756 528.713,-166.247\"/>\n",
       "<text text-anchor=\"middle\" x=\"531.484\" y=\"-178.43\" font-family=\"Times,serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"209,-53 0,-53 0,-0 209,-0 209,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"104.5\" y=\"-37.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 4.9236662412e+15</text>\n",
       "<text text-anchor=\"middle\" x=\"104.5\" y=\"-22.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 1257</text>\n",
       "<text text-anchor=\"middle\" x=\"104.5\" y=\"-7.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 64781848.271</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.303,-88.9485C220.782,-78.6102 194.987,-67.2722 172.028,-57.1809\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"173.276,-53.9064 162.713,-53.0867 170.459,-60.3147 173.276,-53.9064\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"427.5,-53 227.5,-53 227.5,-0 427.5,-0 427.5,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"327.5\" y=\"-37.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 3.147813102e+16</text>\n",
       "<text text-anchor=\"middle\" x=\"327.5\" y=\"-22.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 249</text>\n",
       "<text text-anchor=\"middle\" x=\"327.5\" y=\"-7.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 235450289.735</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M322.298,-88.9485C323.003,-80.6238 323.762,-71.6509 324.478,-63.2027\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"327.965,-63.4963 325.321,-53.2367 320.99,-62.9058 327.965,-63.4963\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\"><title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"678.5,-53 460.5,-53 460.5,-0 678.5,-0 678.5,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"569.5\" y=\"-37.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 4.06753884592e+16</text>\n",
       "<text text-anchor=\"middle\" x=\"569.5\" y=\"-22.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 92</text>\n",
       "<text text-anchor=\"middle\" x=\"569.5\" y=\"-7.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 440868287.554</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M574.052,-88.9485C573.442,-80.7153 572.785,-71.848 572.165,-63.4814\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"575.636,-62.9508 571.406,-53.2367 568.655,-63.468 575.636,-62.9508\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\"><title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"914.5,-53 696.5,-53 696.5,-0 914.5,-0 914.5,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"805.5\" y=\"-37.8\" font-family=\"Times,serif\" font-size=\"14.00\">mse = 1.22264857987e+17</text>\n",
       "<text text-anchor=\"middle\" x=\"805.5\" y=\"-22.8\" font-family=\"Times,serif\" font-size=\"14.00\">samples = 67</text>\n",
       "<text text-anchor=\"middle\" x=\"805.5\" y=\"-7.8\" font-family=\"Times,serif\" font-size=\"14.00\">value = 821594676.851</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M656.593,-88.9485C681.868,-78.5183 709.609,-67.0705 734.226,-56.9121\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"735.588,-60.1367 743.496,-53.0867 732.917,-53.666 735.588,-60.1367\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7f28bea5c048>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graphviz.Source(treedot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Virtudes de los arboles de decision:\n",
    "\n",
    "- Metodo poderoso y probado\n",
    "- Interpretable\n",
    "- No necesita escalar los datos (clasificación), y menos preprocesamiento de variables\n",
    "\n",
    "Sin embargo en la practica existen modelos que obtienen mejor rendimiento. Como mejorar el modelo de arboles de decisión?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Concepto General**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest y Gradient Boosted Trees, forman parte de una familia de algoritmos que se denominan ensembles.\n",
    "\n",
    "$$ Ensemble = Submodelos \\rightarrow Entrenamiento \\rightarrow Predicciones_{Intermedias} \\rightarrow Voto \\rightarrow Prediccion_{final}$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Cómo funciona el algoritmo Random Forest?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a generar cientos de modelos de arboles de decisión que serán entrenados sobre **conjuntos de datos bootstrapeados** del conjunto de datos original y donde para cada etapa de separación el **conjunto de features elegibles** sera un subconjunto aleatorio del conjunto original de features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../vol/img/rf_tree.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cada uno de los arboles entrenados luego podrá votar por su predicción y promediaremos estos votos."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../vol/img/random_forest.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ensembles del pobre (\"Poor man's ensembles\")**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Entrenar diversos modelos a mano\n",
    "- Promediar el resultado\n",
    "- Owen Zhang, número 1 de Kaggle.com durante un largo tiempo, ocupaba esta estrategia promediando diversos modelos XGBoost.\n",
    "- ``from sklearn.ensemble import VotingClassifier`` sirve por ejemplo para hacer un ensemble manual de clasificación\n",
    "\n",
    "En general los ensembles del pobre funcionan ya que cada uno de los modelos que votarán en conjunto son bastante fuertes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Porqué RF es poderoso?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "  **Leo Breiman** creador del Random Forest demostró que un ensemble podía tener buen poder de generalización sí:\n",
    "  <ol>\n",
    "    <li>Los submodelos tienen buen poder de predicción</li>\n",
    "    <li>Los submodelos están descorrelacionados</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Así el algoritmo de Random Forest compromete un poco de poder de predicción de cada uno de los decision trees que arma, pero la forma aleatoria de generarlos hace que esten **fuertemente descorrelacionados**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "forest = RandomForestRegressor(200)\n",
    "results = cross_validate(forest,X,y,cv=5,scoring='r2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.965239108957\n",
      "0.514504965042\n"
     ]
    }
   ],
   "source": [
    "test_scores = results['test_score']\n",
    "train_scores = results['train_score']\n",
    "print(np.mean(train_scores))\n",
    "print(np.mean(test_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mejor resultado que Lasso! Ya no tenemos Bias y tenemos un mejor score r2. Sin embargo tenemos una diferencia importante entre score de entrenamiento y de test (overfit)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Boosted Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "ensemble = GradientBoostingRegressor()\n",
    "results = cross_validate(ensemble,X,y,cv=5,scoring='r2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.915139214355\n",
      "0.525193924287\n"
     ]
    }
   ],
   "source": [
    "test_scores = results['test_score']\n",
    "train_scores = results['train_score']\n",
    "print(np.mean(train_scores))\n",
    "print(np.mean(test_scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cómo optimizamos los parametros de este último modelo?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimización de hiperparametros"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Fijar un learning rate alto\n",
    "- Fijar parametros de los arboles\n",
    "- Fijados estos parametros, elegir el mejor numero de estimadores que conforman el ensemble\n",
    "- (Tarea) Con el learning rate dado y el numero de estimadores óptimo, optimizar los parametros de los arboles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Grid Search**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Por ahora dijimos que:\n",
    "    \n",
    "- train_test_split servia para evaluaciones rapidas, testeos y prototipaje\n",
    "- cross_validate es un método más robusto para poder estimar el rendimiento de tu algoritmo\n",
    "\n",
    "Sin embargo una vez que hemos finalizado nuestra etapa de prototipaje y ya queremos establecer un modelo definitivo deberiamos seguir el flujo siguiente."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../vol/img/grid_search_crossval.png\" width=700>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_test1 = {'n_estimators': range(20,501,20)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[20,\n",
       " 40,\n",
       " 60,\n",
       " 80,\n",
       " 100,\n",
       " 120,\n",
       " 140,\n",
       " 160,\n",
       " 180,\n",
       " 200,\n",
       " 220,\n",
       " 240,\n",
       " 260,\n",
       " 280,\n",
       " 300,\n",
       " 320,\n",
       " 340,\n",
       " 360,\n",
       " 380,\n",
       " 400,\n",
       " 420,\n",
       " 440,\n",
       " 460,\n",
       " 480,\n",
       " 500]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(param_test1['n_estimators'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = GradientBoostingRegressor(learning_rate=0.1,\n",
    "                                     min_samples_split=500,\n",
    "                                     min_samples_leaf=50,\n",
    "                                     max_depth=8,\n",
    "                                     max_features='sqrt',\n",
    "                                     subsample=0.8,\n",
    "                                     random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "gsearch1 = GridSearchCV(estimator,\n",
    "                       param_grid = param_test1,\n",
    "                       scoring='r2',\n",
    "                       cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,\n",
       "             learning_rate=0.1, loss='ls', max_depth=8,\n",
       "             max_features='sqrt', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=50, min_samples_split=500,\n",
       "             min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "             presort='auto', random_state=10, subsample=0.8, verbose=0,\n",
       "             warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': range(20, 501, 20)},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='r2', verbose=0)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch1.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.5/site-packages/sklearn/model_selection/_search.py:747: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: 0.65534, std: 0.05764, params: {'n_estimators': 20},\n",
       "  mean: 0.71947, std: 0.06256, params: {'n_estimators': 40},\n",
       "  mean: 0.73472, std: 0.06360, params: {'n_estimators': 60},\n",
       "  mean: 0.73893, std: 0.06236, params: {'n_estimators': 80},\n",
       "  mean: 0.74205, std: 0.06271, params: {'n_estimators': 100},\n",
       "  mean: 0.74593, std: 0.06236, params: {'n_estimators': 120},\n",
       "  mean: 0.74954, std: 0.06335, params: {'n_estimators': 140},\n",
       "  mean: 0.75082, std: 0.06305, params: {'n_estimators': 160},\n",
       "  mean: 0.75257, std: 0.06344, params: {'n_estimators': 180},\n",
       "  mean: 0.75349, std: 0.06447, params: {'n_estimators': 200},\n",
       "  mean: 0.75457, std: 0.06342, params: {'n_estimators': 220},\n",
       "  mean: 0.75531, std: 0.06489, params: {'n_estimators': 240},\n",
       "  mean: 0.75517, std: 0.06572, params: {'n_estimators': 260},\n",
       "  mean: 0.75389, std: 0.06495, params: {'n_estimators': 280},\n",
       "  mean: 0.75460, std: 0.06569, params: {'n_estimators': 300},\n",
       "  mean: 0.75250, std: 0.06545, params: {'n_estimators': 320},\n",
       "  mean: 0.75350, std: 0.06492, params: {'n_estimators': 340},\n",
       "  mean: 0.75354, std: 0.06623, params: {'n_estimators': 360},\n",
       "  mean: 0.75259, std: 0.06542, params: {'n_estimators': 380},\n",
       "  mean: 0.75254, std: 0.06469, params: {'n_estimators': 400},\n",
       "  mean: 0.75186, std: 0.06477, params: {'n_estimators': 420},\n",
       "  mean: 0.75205, std: 0.06508, params: {'n_estimators': 440},\n",
       "  mean: 0.75157, std: 0.06449, params: {'n_estimators': 460},\n",
       "  mean: 0.75051, std: 0.06352, params: {'n_estimators': 480},\n",
       "  mean: 0.75096, std: 0.06327, params: {'n_estimators': 500}],\n",
       " {'n_estimators': 240},\n",
       " 0.7553059694284987)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_results = cross_validate(gsearch1.best_estimator_,X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.81831693665\n",
      "0.741387612516\n"
     ]
    }
   ],
   "source": [
    "test_scores = final_results['test_score']\n",
    "train_scores = final_results['train_score']\n",
    "print(np.mean(train_scores))\n",
    "print(np.mean(test_scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = GradientBoostingRegressor(learning_rate=0.1,\n",
    "                                     min_samples_split=500,\n",
    "                                     min_samples_leaf=50,\n",
    "                                     max_depth=8,\n",
    "                                     max_features='sqrt',\n",
    "                                     subsample=0.8,\n",
    "                                     random_state=10,\n",
    "                                     n_estimators=240)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,\n",
       "             learning_rate=0.1, loss='ls', max_depth=8,\n",
       "             max_features='sqrt', max_leaf_nodes=None,\n",
       "             min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "             min_samples_leaf=50, min_samples_split=500,\n",
       "             min_weight_fraction_leaf=0.0, n_estimators=240,\n",
       "             presort='auto', random_state=10, subsample=0.8, verbose=0,\n",
       "             warm_start=False)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8092888852563106"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reflexiones de cierre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Recursos **\n",
    "\n",
    "- Reddit /machinelearning y /learnmachinelearning\n",
    "- Analytics Vidhya y KD Nuggets\n",
    "- Kaggle.com y \"There is no Free Hunch\" Blog\n",
    "- Arxiv, papers\n",
    "- Libros: \"Pattern Recognition and Machine Learning\" C.Bishop y \"Elements of Statistical Learning\".\n",
    "\n",
    "** Próximos pasos **\n",
    "\n",
    "- Matemáticas\n",
    "- Praxis: Feature Engineering, Model Selection y Tuning\n",
    "- Deep Learning para NLP y Computer Vision\n",
    "- Machine Learning Bayesiano"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}