03-DSA-Cap09-Metodos-Ensemble-Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='blue'>Data Science Academy - Machine Learning</font>\n",
    "\n",
    "# <font color='blue'>Capítulo 9 - Métodos Ensemble</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bagging é usado para construção de múltiplos modelos (normalmente do mesmo tipo) a partir de diferentes subsets no dataset de treino.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Um classificador Bagging é um meta-estimador ensemble que faz o fit de classificadores base, cada um em subconjuntos aleatórios do conjunto de dados original e, em seguida, agrega suas previsões individuais (por votação ou por média) para formar uma previsão final. \n",
    "\n",
    "Tal meta-estimador pode tipicamente ser usado como uma maneira de reduzir a variância de um estimador (por exemplo, uma árvore de decisão), introduzindo a randomização em seu procedimento de construção e fazendo um ensemble (conjunto) a partir dele."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.preprocessing import scale\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Carga de dados\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAALrUlEQVR4nO3d34tc9RnH8c+na4K/YhaqFTGSrVADIuQHEioByQ+VWCXxohcJKERa0otWEloQ7U31H9D0ogghagPGiEYjRVprQBcRWm1+rDW6sWjY4DbqKpJNtNCgPr2Yk5Km2+7Z9XzPzO7zfsGQ2d3ZeZ7dzWe+58ycOY8jQgBmt291uwEA5RF0IAGCDiRA0IEECDqQAEEHEuiJoNtea/td2+/Zvq9wrcdsj9k+XLLOWfWusv2K7WHbb9veUrje+bbfsP1mVe/BkvWqmn22D9l+oXStqt6I7bdsD9neX7hWv+09to9Uf8MbCtZaVP1MZy4nbW9t5M4joqsXSX2S3pd0taS5kt6UdG3BejdKWibpcEs/3xWSllXX50n6W+Gfz5Iurq7PkfS6pO8X/hl/LulJSS+09DsdkXRpS7V2SvpxdX2upP6W6vZJ+kjSwiburxdW9OWS3ouIoxFxWtJTktaXKhYRr0r6rNT9T1Dvw4g4WF0/JWlY0pUF60VEfF59OKe6FDsqyvYCSbdJ2lGqRrfYvkSdheFRSYqI0xFxoqXyayS9HxHHmrizXgj6lZI+OOvjURUMQjfZHpC0VJ1VtmSdPttDksYk7YuIkvW2SbpX0tcFa5wrJL1k+4DtzQXrXC3pE0mPV7smO2xfVLDe2TZI2t3UnfVC0D3B52bdcbm2L5b0rKStEXGyZK2I+CoilkhaIGm57etK1LF9u6SxiDhQ4v7/jxURsUzSrZJ+avvGQnXOU2c375GIWCrpC0lFn0OSJNtzJa2T9ExT99kLQR+VdNVZHy+QdLxLvRRhe446Id8VEc+1VbfazByUtLZQiRWS1tkeUWeXa7XtJwrV+reIOF79OyZprzq7fyWMSho9a4tojzrBL+1WSQcj4uOm7rAXgv4XSd+z/d3qkWyDpN91uafG2LY6+3jDEfFQC/Uus91fXb9A0k2SjpSoFRH3R8SCiBhQ5+/2ckTcWaLWGbYvsj3vzHVJt0gq8gpKRHwk6QPbi6pPrZH0Tola59ioBjfbpc6mSVdFxJe2fybpj+o80/hYRLxdqp7t3ZJWSrrU9qikX0XEo6XqqbPq3SXprWq/WZJ+GRG/L1TvCkk7bfep80D+dES08rJXSy6XtLfz+KnzJD0ZES8WrHePpF3VInRU0t0Fa8n2hZJulvSTRu+3eiofwCzWC5vuAAoj6EACBB1IgKADCRB0IIGeCnrhwxm7Vot61Ot2vZ4KuqQ2f5mt/uGoR71u1uu1oAMooMgBM7Zn9VE411xzzZS/Z3x8XPPnz59WvdOnT0/5e06dOqV58+ZNq97IyMi0vg+9ISL+641iBH0aBgcHW63XdvA2bdrUaj00a6Kgs+kOJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkQdCCBWkFvc2QSgOZNGvTqJIO/UecUtNdK2mj72tKNAWhOnRW91ZFJAJpXJ+hpRiYBs1Wd87rXGplUvVG+7ffsAqihTtBrjUyKiO2Stkuz/91rwExTZ9N9Vo9MAjKYdEVve2QSgObVmr1WzQkrNSsMQGEcGQckQNCBBAg6kABBBxIg6EACBB1IgKADCRB0IAEmtUxD25NTFi5c2Gq9th07dqzVegMDA63WaxuTWoCkCDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpBAnZFMj9kes324jYYANK/Oiv5bSWsL9wGgoEmDHhGvSvqshV4AFMI+OpBArfO618HsNaB3NRZ0Zq8BvYtNdyCBOi+v7Zb0J0mLbI/a/lH5tgA0qc6QxY1tNAKgHDbdgQQIOpAAQQcSIOhAAgQdSICgAwkQdCABgg4k0Nix7pmcOHGi1Xptz14bHx9vtd7g4GCr9fr7+1ut1/b/l4mwogMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkQdCCBOieHvMr2K7aHbb9te0sbjQFoTp1j3b+U9IuIOGh7nqQDtvdFxDuFewPQkDqz1z6MiIPV9VOShiVdWboxAM2Z0j667QFJSyW9XqIZAGXUfpuq7YslPStpa0ScnODrzF4DelStoNueo07Id0XEcxPdhtlrQO+q86y7JT0qaTgiHirfEoCm1dlHXyHpLkmrbQ9Vlx8U7gtAg+rMXntNklvoBUAhHBkHJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkQdCABZq9Nw8jISKv1Fi9e3Gq9+fPnt1pvaGio1Xq9MAutbazoQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSKDOWWDPt/2G7Ter2WsPttEYgObUOdb9n5JWR8Tn1fndX7P9h4j4c+HeADSkzllgQ9Ln1YdzqgsDGoAZpNY+uu0+20OSxiTtiwhmrwEzSK2gR8RXEbFE0gJJy21fd+5tbG+2vd/2/qabBPDNTOlZ94g4IWlQ0toJvrY9Iq6PiOsb6g1AQ+o8636Z7f7q+gWSbpJ0pHRjAJpT51n3KyTttN2nzgPD0xHxQtm2ADSpzrPuf5W0tIVeABTCkXFAAgQdSICgAwkQdCABgg4kQNCBBAg6kABBBxJg9to03HHHHa3WW7lyZav1lixZ0mq9hx9+uNV6bdu2bVu3W2BFBzIg6EACBB1IgKADCRB0IAGCDiRA0IEECDqQAEEHEiDoQAK1g14NcThkmxNDAjPMVFb0LZKGSzUCoJy6I5kWSLpN0o6y7QAooe6Kvk3SvZK+LtgLgELqTGq5XdJYRByY5HbMXgN6VJ0VfYWkdbZHJD0labXtJ869EbPXgN41adAj4v6IWBARA5I2SHo5Iu4s3hmAxvA6OpDAlE4lFRGD6oxNBjCDsKIDCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIEHUiA2WszwODgYLdbmFUGBga63ULrWNGBBAg6kABBBxIg6EACBB1IgKADCRB0IAGCDiRA0IEECDqQQK1DYKtTPZ+S9JWkLzmlMzCzTOVY91UR8WmxTgAUw6Y7kEDdoIekl2wfsL25ZEMAmld3031FRBy3/R1J+2wfiYhXz75B9QDAgwDQg2qt6BFxvPp3TNJeScsnuA2z14AeVWea6kW25525LukWSYdLNwagOXU23S+XtNf2mds/GREvFu0KQKMmDXpEHJW0uIVeABTCy2tAAgQdSICgAwkQdCABgg4kQNCBBAg6kABBBxJg9to0rF+/vtV64+PjrdZ74IEHWq3Xtueff77bLbSOFR1IgKADCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJ1Aq67X7be2wfsT1s+4bSjQFoTt1j3X8t6cWI+KHtuZIuLNgTgIZNGnTbl0i6UdImSYqI05JOl20LQJPqbLpfLekTSY/bPmR7RzXI4T/Y3mx7v+39jXcJ4BupE/TzJC2T9EhELJX0haT7zr0RI5mA3lUn6KOSRiPi9erjPeoEH8AMMWnQI+IjSR/YXlR9ao2kd4p2BaBRdZ91v0fSruoZ96OS7i7XEoCm1Qp6RAxJYt8bmKE4Mg5IgKADCRB0IAGCDiRA0IEECDqQAEEHEiDoQALMXpuGVatWtVpvy5YtrdZr286dO1utNzg42Gq9XsCKDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJDBp0G0vsj101uWk7a1tNAegGZMeAhsR70paIkm2+yT9XdLewn0BaNBUN93XSHo/Io6VaAZAGVMN+gZJu0s0AqCc2kGvzum+TtIz/+PrzF4DetRU3qZ6q6SDEfHxRF+MiO2StkuS7WigNwANmcqm+0ax2Q7MSLWCbvtCSTdLeq5sOwBKqDuS6R+Svl24FwCFcGQckABBBxIg6EACBB1IgKADCRB0IAGCDiRA0IEECDqQgCOaf/+J7U8kTec965dK+rThdnqhFvWo11a9hRFx2bmfLBL06bK9PyKun221qEe9btdj0x1IgKADCfRa0LfP0lrUo15X6/XUPjqAMnptRQdQAEEHEiDoQAIEHUiAoAMJ/Aur4YK84t2cEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline\n",
    "plt.gray() \n",
    "plt.matshow(digits.images[5]) \n",
    "plt.show() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pré-processamento\n",
    "# Coloca todos os dados na mesma escala\n",
    "data = scale(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variáveis preditoras e variável target\n",
    "X = data\n",
    "y = digits.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construção do Classificador\n",
    "bagging = BaggingClassifier(KNeighborsClassifier(), max_samples = 0.5, max_features = 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingClassifier(base_estimator=KNeighborsClassifier(algorithm='auto',\n",
       "                                                      leaf_size=30,\n",
       "                                                      metric='minkowski',\n",
       "                                                      metric_params=None,\n",
       "                                                      n_jobs=None,\n",
       "                                                      n_neighbors=5, p=2,\n",
       "                                                      weights='uniform'),\n",
       "                  bootstrap=True, bootstrap_features=False, max_features=0.5,\n",
       "                  max_samples=0.5, n_estimators=10, n_jobs=None,\n",
       "                  oob_score=False, random_state=None, verbose=0,\n",
       "                  warm_start=False)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "?cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Score do modelo\n",
    "scores = cross_val_score(bagging, X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Média do score\n",
    "mean = scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.95016611 0.93656093 0.94798658]\n"
     ]
    }
   ],
   "source": [
    "print(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9449045416765015\n"
     ]
    }
   ],
   "source": [
    "print(mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extremely Randomized Trees (ExtraTrees)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.preprocessing import scale\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Carregando os dados\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pré-processamento\n",
    "data = scale(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variáveis preditoras e variável target\n",
    "X = data\n",
    "y = digits.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.73920266 0.8163606  0.76845638]\n",
      "0.7746732115493016\n"
     ]
    }
   ],
   "source": [
    "# Cria o classificador com uma árvore de decisão\n",
    "clf = DecisionTreeClassifier(max_depth = None, min_samples_split = 2, random_state = 0)\n",
    "scores = cross_val_score(clf, X, y)\n",
    "mean = scores.mean()\n",
    "print(scores)\n",
    "print(mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.90863787 0.9015025  0.90436242]\n",
      "0.904834264678386\n"
     ]
    }
   ],
   "source": [
    "# Cria o classificador com Random Forest\n",
    "clf = RandomForestClassifier(n_estimators = 10, max_depth = None, min_samples_split = 2, random_state = 0)\n",
    "scores = cross_val_score(clf, X, y)\n",
    "mean = scores.mean()\n",
    "print(scores)\n",
    "print(mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.89202658 0.90984975 0.91610738]\n",
      "0.9059945700686877\n"
     ]
    }
   ],
   "source": [
    "# Cria o classificador com Extra Tree\n",
    "clf = ExtraTreesClassifier(n_estimators = 10, max_depth = None, min_samples_split = 2, random_state = 0)\n",
    "scores = cross_val_score(clf, X, y)\n",
    "mean = scores.mean()\n",
    "print(scores)\n",
    "print(mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Um classificador AdaBoost é um meta-estimador que começa ajustando um classificador no conjunto de dados original e depois ajusta cópias adicionais do classificador no mesmo conjunto de dados, mas onde os pesos das instâncias classificadas incorretamente são ajustados para que os classificadores subsequentes se concentrem mais em casos difíceis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import\n",
    "import numpy as np\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.datasets import fetch_openml\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Carregandos os dados\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_openml.html\n",
    "# https://archive.ics.uci.edu/ml/datasets/Heart+Disease\n",
    "heart = fetch_openml(\"heart\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': <270x13 sparse matrix of type '<class 'numpy.float64'>'\n",
       " \twith 3378 stored elements in Compressed Sparse Row format>,\n",
       " 'target': array([ 1., -1.,  1., -1., -1., -1.,  1.,  1.,  1.,  1., -1., -1., -1.,\n",
       "         1., -1., -1.,  1.,  1., -1., -1.,  1., -1., -1., -1., -1., -1.,\n",
       "        -1., -1.,  1., -1.,  1., -1., -1.,  1.,  1.,  1.,  1.,  1., -1.,\n",
       "        -1.,  1., -1., -1., -1.,  1., -1.,  1.,  1.,  1.,  1.,  1., -1.,\n",
       "        -1., -1., -1., -1.,  1., -1.,  1.,  1., -1.,  1., -1., -1., -1.,\n",
       "         1., -1.,  1., -1.,  1.,  1., -1., -1., -1., -1.,  1., -1., -1.,\n",
       "        -1., -1.,  1.,  1.,  1., -1., -1., -1., -1., -1., -1.,  1., -1.,\n",
       "         1.,  1.,  1.,  1.,  1., -1.,  1., -1., -1., -1.,  1., -1.,  1.,\n",
       "         1.,  1., -1.,  1.,  1., -1.,  1., -1.,  1., -1., -1., -1.,  1.,\n",
       "         1., -1.,  1.,  1.,  1.,  1., -1., -1., -1.,  1., -1., -1.,  1.,\n",
       "         1.,  1., -1.,  1., -1., -1., -1.,  1., -1., -1.,  1., -1.,  1.,\n",
       "        -1.,  1.,  1.,  1.,  1.,  1., -1., -1., -1., -1., -1., -1., -1.,\n",
       "         1., -1., -1.,  1.,  1.,  1., -1.,  1., -1., -1., -1., -1., -1.,\n",
       "         1., -1.,  1.,  1., -1., -1.,  1.,  1.,  1.,  1., -1., -1.,  1.,\n",
       "         1., -1., -1., -1.,  1., -1., -1.,  1., -1.,  1., -1.,  1., -1.,\n",
       "        -1., -1., -1., -1.,  1., -1.,  1.,  1.,  1.,  1., -1., -1., -1.,\n",
       "         1., -1.,  1., -1., -1.,  1., -1., -1., -1., -1., -1., -1.,  1.,\n",
       "         1., -1.,  1., -1., -1.,  1.,  1., -1., -1.,  1.,  1., -1.,  1.,\n",
       "        -1.,  1., -1.,  1., -1., -1.,  1., -1., -1.,  1., -1.,  1.,  1.,\n",
       "        -1.,  1.,  1.,  1., -1.,  1., -1., -1., -1., -1.,  1.,  1., -1.,\n",
       "        -1.,  1.,  1., -1.,  1., -1., -1., -1., -1.,  1.]),\n",
       " 'feature_names': ['att_1',\n",
       "  'att_2',\n",
       "  'att_3',\n",
       "  'att_4',\n",
       "  'att_5',\n",
       "  'att_6',\n",
       "  'att_7',\n",
       "  'att_8',\n",
       "  'att_9',\n",
       "  'att_10',\n",
       "  'att_11',\n",
       "  'att_12',\n",
       "  'att_13'],\n",
       " 'DESCR': '**Author**: Laboratory of Artificial Intelligence and Computer Science of the University of Porto (LIACC)  \\nlibSVM\",\"AAD group  \\n**Source**: [original](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html) - Date unknown  \\n**Please cite**:   \\n\\n#Dataset from the LIBSVM data repository.\\n\\nPreprocessing: scaled to [-1,1]\\n\\nDownloaded from openml.org.',\n",
       " 'details': {'id': '1574',\n",
       "  'name': 'heart',\n",
       "  'version': '1',\n",
       "  'format': 'Sparse_ARFF',\n",
       "  'creator': 'Laboratory of Artificial Intelligence and Computer Science of the University of Porto (LIACC)',\n",
       "  'contributor': ['libSVM', 'AAD group'],\n",
       "  'upload_date': '2015-06-02T11:29:13',\n",
       "  'licence': 'Public',\n",
       "  'url': 'https://www.openml.org/data/v1/download/1594038/heart.sparse_arff',\n",
       "  'file_id': '1594038',\n",
       "  'default_target_attribute': 'class',\n",
       "  'visibility': 'public',\n",
       "  'original_data_url': 'http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html',\n",
       "  'status': 'active',\n",
       "  'processing_date': '2018-10-03 21:40:10',\n",
       "  'md5_checksum': 'f50ad5f44a5aa587e7f3794007182eb2'},\n",
       " 'categories': {},\n",
       " 'url': 'https://www.openml.org/d/1574'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = heart.data\n",
    "y = np.copy(heart.target)\n",
    "y[y == -1] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<270x13 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 3378 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 0., 1., 0., 0., 0., 1., 1., 1., 1., 0., 0., 0., 1., 0., 0., 1.,\n",
       "       1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1.,\n",
       "       1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 1., 0., 1., 1., 1., 1., 1.,\n",
       "       0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 1., 0., 0., 0., 1., 0., 1.,\n",
       "       0., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0.,\n",
       "       0., 0., 0., 0., 1., 0., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1.,\n",
       "       0., 1., 1., 1., 0., 1., 1., 0., 1., 0., 1., 0., 0., 0., 1., 1., 0.,\n",
       "       1., 1., 1., 1., 0., 0., 0., 1., 0., 0., 1., 1., 1., 0., 1., 0., 0.,\n",
       "       0., 1., 0., 0., 1., 0., 1., 0., 1., 1., 1., 1., 1., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 1., 0., 0., 1., 1., 1., 0., 1., 0., 0., 0., 0., 0., 1.,\n",
       "       0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 1.,\n",
       "       0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0., 1., 0., 1., 1., 1.,\n",
       "       1., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1.,\n",
       "       1., 0., 1., 0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 0., 1., 0., 1.,\n",
       "       0., 0., 1., 0., 0., 1., 0., 1., 1., 0., 1., 1., 1., 0., 1., 0., 0.,\n",
       "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 1.])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Datasets de treino e de teste\n",
    "X_test, y_test = X[189:], y[189:]\n",
    "X_train, y_train = X[:189], y[:189]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construindo o estimador base\n",
    "estim_base = DecisionTreeClassifier(max_depth = 1, min_samples_leaf = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construindo a primeira versão do modelo Adaboost\n",
    "ada_clf_v1 = AdaBoostClassifier(base_estimator = estim_base, \n",
    "                                learning_rate = 0.1, \n",
    "                                n_estimators = 400, \n",
    "                                algorithm = \"SAMME\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME',\n",
       "                   base_estimator=DecisionTreeClassifier(class_weight=None,\n",
       "                                                         criterion='gini',\n",
       "                                                         max_depth=1,\n",
       "                                                         max_features=None,\n",
       "                                                         max_leaf_nodes=None,\n",
       "                                                         min_impurity_decrease=0.0,\n",
       "                                                         min_impurity_split=None,\n",
       "                                                         min_samples_leaf=1,\n",
       "                                                         min_samples_split=2,\n",
       "                                                         min_weight_fraction_leaf=0.0,\n",
       "                                                         presort=False,\n",
       "                                                         random_state=None,\n",
       "                                                         splitter='best'),\n",
       "                   learning_rate=0.1, n_estimators=400, random_state=None)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Treinamento do modelo\n",
    "ada_clf_v1.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.78571429 0.77777778 0.73076923]\n",
      "0.7647537647537647\n"
     ]
    }
   ],
   "source": [
    "# Score\n",
    "scores = cross_val_score(ada_clf_v1, X_test, y_test)\n",
    "print(scores)\n",
    "means = scores.mean()\n",
    "print(means)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construindo a segunda versão do modelo Adaboost\n",
    "ada_clf_v2 = AdaBoostClassifier(base_estimator = estim_base, \n",
    "                                learning_rate = 1.0, \n",
    "                                n_estimators = 400, \n",
    "                                algorithm = \"SAMME\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME',\n",
       "                   base_estimator=DecisionTreeClassifier(class_weight=None,\n",
       "                                                         criterion='gini',\n",
       "                                                         max_depth=1,\n",
       "                                                         max_features=None,\n",
       "                                                         max_leaf_nodes=None,\n",
       "                                                         min_impurity_decrease=0.0,\n",
       "                                                         min_impurity_split=None,\n",
       "                                                         min_samples_leaf=1,\n",
       "                                                         min_samples_split=2,\n",
       "                                                         min_weight_fraction_leaf=0.0,\n",
       "                                                         presort=False,\n",
       "                                                         random_state=None,\n",
       "                                                         splitter='best'),\n",
       "                   learning_rate=1.0, n_estimators=400, random_state=None)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Treinamento do modelo\n",
    "ada_clf_v2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.75       0.85185185 0.76923077]\n",
      "0.790360873694207\n"
     ]
    }
   ],
   "source": [
    "# Score\n",
    "scores = cross_val_score(ada_clf_v2, X_test, y_test)\n",
    "print(scores)\n",
    "means = scores.mean()\n",
    "print(means)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obrigado - Data Science Academy - <a href=\"http://facebook.com/dsacademybr\">facebook.com/dsacademybr</a>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}