Modelos-Otimizados-com-RandomizedSeach-out-22.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classificação de risco de crédito otimizando os modelos Ensemble com Randomized Search (Parte 2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objetivo\n",
    "Nesta segunda etapa, o objetivo será avaliar o quanto a acurácia dos modelos de classificação com **métodos ensemble** melhorará fazendo ajustes com o Randomized Search nos hiperparâmetros dos algortimos selecionados."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algoritmos utilizados na Parte 1 que serão testados nesta etapa:\n",
    "- 1º Gradient Boosting: acurária anterior de 82,03%\n",
    "- 2º Random Forest = acurácia anterior de 81,64%\n",
    "- 3º Extra Trees (Extremely Randomized Forest) = acurácia anterior de 81,25%\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pacotes e funções"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sklearn as sk\n",
    "import matplotlib as mpl\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "from imblearn import under_sampling, over_sampling\n",
    "from imblearn.over_sampling import SMOTE\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Base de dados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_data = pd.read_excel('credit.xls', skiprows = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0',\n",
      "       'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2',\n",
      "       'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1',\n",
      "       'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6',\n",
      "       'default payment next month'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Variáveis da base de dados\n",
    "print(credit_data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(30000, 25)\n"
     ]
    }
   ],
   "source": [
    "# Quantidades de observações e de variáveis\n",
    "print(credit_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Variável-target**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'default payment next month'\n",
    "y = np.asarray(credit_data[target])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Variáveis explicativas**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = credit_data.columns.drop(['ID', target])\n",
    "X = np.asarray(credit_data[features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bases para treino (70%) e para teste (30%)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelo 1 - Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestClassifier(max_depth=4, max_features=16, min_samples_leaf=2,\n",
      "                       min_samples_split=8, n_estimators=10)\n"
     ]
    }
   ],
   "source": [
    "# Parâmetros a serem testados:\n",
    "param_dist = {\"n_estimators\": [10, 20, 50],\n",
    "              \"max_depth\": [1, 4, 6, 8],\n",
    "              \"max_features\": [10, 16, 20],\n",
    "              \"min_samples_split\": [2, 8, 16, 18],\n",
    "              \"min_samples_leaf\": [1, 2, 3, 4, 6, 8],\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "# Treinamento do modelo:\n",
    "rs_search = RandomizedSearchCV(RandomForestClassifier(), param_distributions = param_dist, n_iter = 25, return_train_score = True)\n",
    "rs_search.fit(X_train, y_train)\n",
    "\n",
    "# Melhor estimador:\n",
    "bestrs = rs_search.best_estimator_\n",
    "print (bestrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comentário:** dentre os parâmetros apresentados ao novo modelo, o melhor foi construído com as configurações acima."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acurácia em treino =  0.8238095238095238\n",
      "Acurácia em teste =  0.8211111111111111\n",
      "[[6684  298]\n",
      " [1312  706]]\n"
     ]
    }
   ],
   "source": [
    "# Predição usando melhor o estimador:\n",
    "rs_pred = bestrs.predict(X_test)\n",
    "print(\"Acurácia em treino = \", rs_search.score(X_train, y_train))\n",
    "print(\"Acurácia em teste = \", accuracy_score(y_test, rs_pred))\n",
    "print(confusion_matrix(y_test, rs_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelo 2 - Extra Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExtraTreesClassifier(max_depth=6, max_features=20, min_samples_split=16,\n",
      "                     n_estimators=20)\n"
     ]
    }
   ],
   "source": [
    "# Parâmetros a serem testados:\n",
    "param_dist_et = {\"n_estimators\": [10, 20, 50],\n",
    "                 \"max_depth\": [1, 4, 6, 8],\n",
    "                 \"max_features\": [10, 16, 20],\n",
    "                 \"min_samples_split\": [2, 8, 16, 18],\n",
    "                 \"min_samples_leaf\": [1, 2, 3, 4, 6, 8],\n",
    "                 \"bootstrap\": [True, False],\n",
    "                 \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "# Combinando parâmetros:\n",
    "rs_extrees = RandomizedSearchCV(ExtraTreesClassifier(), param_distributions = param_dist_et, n_iter = 25, return_train_score = True)  \n",
    "rs_extrees.fit(X_train, y_train)\n",
    "\n",
    "# Melhor estimador\n",
    "bestrset = rs_extrees.best_estimator_\n",
    "print (bestrset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acurácia em treino =  0.8252380952380952\n",
      "Acurácia em teste =  0.8212222222222222\n",
      "[[6665  317]\n",
      " [1292  726]]\n"
     ]
    }
   ],
   "source": [
    "# Nova predição\n",
    "rs_et_pred = bestrset.predict(X_test)\n",
    "print(\"Acurácia em treino = \", rs_extrees.score(X_train, y_train))\n",
    "print(\"Acurácia em teste = \", accuracy_score(y_test, rs_et_pred))\n",
    "print(confusion_matrix(y_test, rs_et_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelo 3 - Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GradientBoostingClassifier(max_depth=4, max_features=20, min_samples_leaf=8,\n",
      "                           min_samples_split=8, n_estimators=20)\n"
     ]
    }
   ],
   "source": [
    "# Parâmetros a serem testados\n",
    "param_dist_gb = {\"n_estimators\": [10, 20, 50],\n",
    "                 \"max_depth\": [1, 4, 6, 8],\n",
    "                 \"max_features\": [10, 16, 20],\n",
    "                 \"min_samples_split\": [2, 8, 16, 18],\n",
    "                 \"min_samples_leaf\": [1, 2, 3, 4, 6, 8],\n",
    "                 \"learning_rate\": [1.0, 0.1, 0.01]}\n",
    "\n",
    "# Combinando parâmetros\n",
    "rs_gb = RandomizedSearchCV(GradientBoostingClassifier(), param_distributions = param_dist_gb, n_iter = 25, return_train_score = True)  \n",
    "rs_gb.fit(X_train, y_train)\n",
    "\n",
    "# Melhor estimador\n",
    "bestrsgb = rs_gb.best_estimator_\n",
    "print (bestrsgb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acurácia em treino =  0.8257619047619048\n",
      "Acurácia em teste =  0.8207777777777778\n",
      "[[6695  287]\n",
      " [1326  692]]\n"
     ]
    }
   ],
   "source": [
    "# Nova predição\n",
    "rs_gb_pred = bestrsgb.predict(X_test)\n",
    "print(\"Acurácia em treino = \", rs_gb.score(X_train, y_train))\n",
    "print(\"Acurácia em teste = \", accuracy_score(y_test, rs_gb_pred))\n",
    "print(confusion_matrix(y_test, rs_gb_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resultados:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As otimizações obtidas usando os parâmetros e hiperparâmetros selecionados através do Randomized Search foram positivas, mas bem discretas:  \n",
    "- Random Forest = a acurácia passou de 81,64% para 82,11% (diferença de 0,47%).\n",
    "- Extra Trees = a acurácia passou de 81,25% para 82,12% (diferença de 0,87%).\n",
    "- Gradient Boosting: a acurária passou de 82,03% para 82,07% (diferença de 0,04%).\n",
    "Nessa etapa, o melhor resultado foi obtido com o **Extra Trees**: 82,12% de acurária, com incremento de 0,87%."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}