2021/seminars/sem15_boostings/sem15_boostings_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Семинар 15: различные имплементации градиентного бустинга\n",
    "\n",
    "* XGboost + подбор гиперпараметров\n",
    "* CatBoost + SHAP\n",
    "* LightGBM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![\"История\"](bst.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. XGboost + подбор гиперпараметров\n",
    "\n",
    "Код по xgboost основан на статье https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Основные особенности xgboost:\n",
    "* базовый алгоритм приближает направление, посчитанное с учетом второй производной функции потерь\n",
    "\n",
    "* функционал регуляризуется – добавляются штрафы за количество листьев и за норму коэффициентов\n",
    "\n",
    "* при построении дерева используется критерий информативности, зависящий от оптимального вектора сдвига\n",
    "\n",
    "* критерий останова при обучении дерева также зависит от оптимального сдвига\n",
    "\n",
    "Почитать подробнее тут: https://github.com/esokolov/ml-course-hse/blob/master/2020-fall/lecture-notes/lecture10-ensembles.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from matplotlib import pylab as plot\n",
    "%pylab inline\n",
    "\n",
    "import xgboost as xgb\n",
    "from xgboost.sklearn import XGBClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Будем решать задачу классификации: по различным характеристикам клиента попытаемся определить, выплатит клиент долг или нет."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = pd.read_csv('train_small.csv')\n",
    "X_test = pd.read_csv('test_small.csv')\n",
    "len(X_train), len(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для того, чтобы бустинг показал хорошее качество и не переобучился, необходимо аккуратно подобрать его гиперпараметры. \n",
    "\n",
    "Алгоритм подбора гиперпараметров следующий:\n",
    "\n",
    "* Выбираем относительно большую learning_rate ($ \\eta \\in [0.05, 0.3]$), подбираем оптимальное число деревьев для выбранного $ \\eta $\n",
    "\n",
    "* Настраиваем параметры деревьев, начиная с самых значимых (max_depth, min_child_weight, gamma, subsample, colsample_bytree)\n",
    "\n",
    "* Настраиваем регуляризации ($ \\lambda, \\alpha $)\n",
    "\n",
    "* Уменьшаем learning_rate, пропорционально увеличиваем число деревьев"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    "        learning_rate=0.1,\n",
    "        n_estimators=150,\n",
    "        objective= 'binary:logistic',\n",
    "        nthread=4,\n",
    "        scale_pos_weight=1,\n",
    "        seed=27)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target='Disbursed'\n",
    "IDcol = 'ID'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train['Disbursed'].value_counts(), len(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим функцию для моделирования и кросс-валидации\n",
    "\n",
    "Эта функция делает следующее:\n",
    "1. обучает модель\n",
    "2. вычисляет accuracy на тренировочной выборке\n",
    "3. вычисляет AUC на тренировочной выборке\n",
    "4. вычисляет AUC на тестовой выборке\n",
    "5. обновляет n_estimators с помощью встроенной в пакет xgboost функции cv\n",
    "6. строит диаграмму Feature Importance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "def modelfit(alg, dtrain, dtest, predictors, useTrainCV=True, cv_folds=5, \\\n",
    "             early_stopping_rounds=50):\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)\n",
    "        xgtest = xgb.DMatrix(dtest[predictors].values)\n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n",
    "            metrics='auc', early_stopping_rounds=early_stopping_rounds)\n",
    "        alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(dtrain[predictors], dtrain['Disbursed'],eval_metric='auc')\n",
    "        \n",
    "    #Predict training set:\n",
    "    dtrain_predictions = alg.predict(dtrain[predictors])\n",
    "    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]\n",
    "        \n",
    "    #Print model report:\n",
    "    print(\"\\nModel Report\")\n",
    "    print(\"Accuracy : %.4g\" % metrics.accuracy_score(dtrain['Disbursed'].values, dtrain_predictions))\n",
    "    print(\"AUC Score (Train): %f\" % metrics.roc_auc_score(dtrain['Disbursed'], dtrain_predprob))\n",
    "    \n",
    "    #Predict on testing data:\n",
    "    dtest['predprob'] = alg.predict_proba(dtest[predictors])[:,1]\n",
    "    print('AUC Score (Test): %f' % metrics.roc_auc_score(dtest['Disbursed'], dtest['predprob']))\n",
    "                \n",
    "    feat_imp = pd.Series(alg.get_booster().get_fscore()).sort_values(ascending=False)\n",
    "    feat_imp.plot(kind='bar', title='Feature Importances')\n",
    "    plt.ylabel('Feature Importance Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Начинаем последовательный подбор параметров при большом learning_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors = [x for x in X_train.columns if x not in [target, IDcol]]\n",
    "xgb1 = XGBClassifier(\n",
    "        learning_rate=0.1,\n",
    "        n_estimators=150,\n",
    "        objective= 'binary:logistic',\n",
    "        nthread=4,\n",
    "        scale_pos_weight=1,\n",
    "        seed=27)\n",
    "modelfit(xgb1, X_train, X_test, predictors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Перебираем max_depth и min_child_weight по сетке."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_test1 = {\n",
    "    'max_depth':range(3,10,2),\n",
    "    'min_child_weight':range(1,6,2)\n",
    "}\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier(learning_rate=0.1, n_estimators=150,\n",
    "                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    "                       param_grid = param_test1, scoring='roc_auc',n_jobs=4, cv=5)\n",
    "gsearch1.fit(X_train[predictors],X_train[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Уточняем предсказание"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test2 = {\n",
    "    'max_depth':[2,3,4],\n",
    "    'min_child_weight':[2,3,4]\n",
    "}\n",
    "gsearch2 = GridSearchCV(estimator = XGBClassifier( learning_rate=0.1, n_estimators=150, \n",
    "                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    "                       param_grid = param_test2, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "gsearch2.fit(X_train[predictors],X_train[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gsearch2.best_params_, gsearch2.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Зафиксируем найденный max_depth и подберем min_child_weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test2b = {\n",
    "    'min_child_weight':[5,6,7]\n",
    "}\n",
    "gsearch2b = GridSearchCV(estimator = XGBClassifier( learning_rate=0.1, n_estimators=150, max_depth=4,\n",
    "                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    "                       param_grid = param_test2b, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "gsearch2b.fit(X_train[predictors],X_train[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gsearch2b.best_params_, gsearch2b.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы подобрали max_depth и min_child_weight - дальше будем использовать подобранные значения параметров. Подберите по сетке параметр gamma на отрезке от 0 до 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test3 = {\n",
    "    'gamma':#your code here\n",
    "}\n",
    "gsearch3b = #your code here\n",
    "gsearch3b.fit(X_train[predictors],X_train[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gsearch3b.best_params_, gsearch3b.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Установите в качестве параметров алгоритма подобранные max_depth, min_child_weight и gamma и посмотрите как изменилось качество."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors = [x for x in X_train.columns if x not in [target, IDcol]]\n",
    "xgb2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=150,\n",
    "        max_depth=4,\n",
    "        min_child_weight=...,\n",
    "        gamma = ...,\n",
    "        objective= 'binary:logistic',\n",
    "        nthread=4,\n",
    "        scale_pos_weight=1,\n",
    "        seed=27)\n",
    "modelfit(xgb2, X_train, X_test, predictors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подберите по сетке параметры subsample и colsample_bytree. Используйте значения этих параметров: [0.6,0.7,0.8,0.9,1.]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подберите параметр регуляризации reg_alpha. Попробуйте [1e-5, 1e-2, 0.1, 1, 100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Запустите обучение и предсказание на найденных параметрах и посмотрите на качество."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb3 = XGBClassifier(\n",
    "        learning_rate=0.1,     \n",
    "        n_estimators=150,   \n",
    "        max_depth=...,\n",
    "        min_child_weight=...,\n",
    "        gamma=...,\n",
    "        subsample=...,\n",
    "        colsample_bytree=...,\n",
    "        reg_alpha=...,\n",
    "        objective= 'binary:logistic',\n",
    "        nthread=4,\n",
    "        scale_pos_weight=1,\n",
    "        seed=27)\n",
    "modelfit(xgb3, X_train, X_test, predictors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Уменьшите learning_rate до 0.01 и пропорционально увеличьте число деревьев. Все подобранные параметры остаются такими же. Обучите модель и сделайте предсказание. Посмотрите на качество."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your code here\n",
    "xgb3 = XGBClassifier(\n",
    "        learning_rate=0.01,\n",
    "        n_estimators=1500,\n",
    "        max_depth=...,\n",
    "        min_child_weight=...,\n",
    "        gamma=...,\n",
    "        subsample=...,\n",
    "        colsample_bytree=...,\n",
    "        reg_alpha=...,\n",
    "        objective= 'binary:logistic',\n",
    "        nthread=4,\n",
    "        scale_pos_weight=1,\n",
    "        seed=27)\n",
    "modelfit(xgb3, X_train, X_test, predictors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. CatBoost\n",
    "\n",
    "Оригинальный ноутбук здесь: https://github.com/catboost/tutorials/blob/master/classification/classification_tutorial.ipynb\n",
    "\n",
    "Особенности катбуста:\n",
    "\n",
    "* используются симметричные деревья решений\n",
    "\n",
    "* для кодирования категориальных признаков используется большой набор методов\n",
    "\n",
    "* динамический бустинг \n",
    "\n",
    "Благодаря использованию симметричных деревьев и динамическому бустингу катбуст гораздо более устойчив к переобучению, чем другие имплементации градиентного бустинга. Поэтому он показывает хорошее качество даже при дефолтных параметрах.\n",
    "\n",
    "Посмотреть видео про катбуст: https://yandex.ru/video/preview/?filmId=10919886479285152550&from=tabbar&text=catboost+анна+вероника"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Устанавливаем библиотеки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install --user --upgrade catboost\n",
    "#!pip install --user --upgrade ipywidgets\n",
    "#!pip install shap\n",
    "#!pip install sklearn\n",
    "#!pip install --upgrade numpy\n",
    "#!jupyter nbextension enable --py widgetsnbextension\n",
    "\n",
    "import catboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pylab as plt\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=4)\n",
    "import catboost\n",
    "from catboost import *\n",
    "from catboost import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Считываем данные. Данные содержат только категориальные признаки, причем с большим количеством категорий. Признаки - закодированные характеристики сотрудника и некоторого ресурса, целевая переменная - есть ли у сотрудника доступ к ресурсу."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_df, test_df) = catboost.datasets.amazon()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Готовим данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = train_df.ACTION\n",
    "X = train_df.drop('ACTION', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создаем список категориальных фичей"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = list(range(0, X.shape[1]))\n",
    "print(cat_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Смотрим на соотношение классов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Labels: {}'.format(set(y)))\n",
    "print('Zero count = {}, One count = {}'.format(len(y) - sum(y), sum(y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Способы создать Pool - объект, содержащий матрицу объект-признак X, вектор ответов, список номеров колонок с категориальными признаками."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pool1 = Pool(data=X, label=y, cat_features=cat_features)\n",
    "\n",
    "pool2 = Pool(data=X, cat_features=cat_features)\n",
    "\n",
    "# Fastest way to create a Pool is to create it from numpy matrix.\n",
    "# This way should be used if you want fast predictions\n",
    "# or fastest way to load the data in python.\n",
    "\n",
    "X_prepared = X.values.astype(str).astype(object)\n",
    "# For FeaturesData class categorial features must have type str\n",
    "\n",
    "pool3 = Pool(\n",
    "    data=FeaturesData(\n",
    "        cat_feature_data=X_prepared,\n",
    "        cat_feature_names=list(X)\n",
    "    ),\n",
    "    label=y.values\n",
    ")\n",
    "\n",
    "print('Dataset shape')\n",
    "print('dataset 1:' + str(pool1.shape) +\n",
    "      '\\ndataset 2:' + str(pool2.shape) +\n",
    "      '\\ndataset 3: ' + str(pool3.shape))\n",
    "\n",
    "print('\\n')\n",
    "print('Column names')\n",
    "print('dataset 1:')\n",
    "print(pool1.get_feature_names()) \n",
    "print('\\ndataset 2:')\n",
    "print(pool2.get_feature_names())\n",
    "print('\\ndataset 3:')\n",
    "print(pool3.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разбиваем данные на train и validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.8, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выбираем функцию потерь"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для бинарной классификации можем использовать:\n",
    "\n",
    "* `Logloss`\n",
    "\n",
    "* `CrossEntropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "model = CatBoostClassifier(\n",
    "    iterations=5,\n",
    "    learning_rate=0.1,\n",
    "    # loss_function='CrossEntropy'\n",
    ")\n",
    "model.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    verbose=False\n",
    ")\n",
    "print('Model is fitted: ' + str(model.is_fitted()))\n",
    "print('Model params:')\n",
    "print(model.get_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Печать результатов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "model = CatBoostClassifier(\n",
    "    iterations=15,\n",
    "#     verbose=5,\n",
    ")\n",
    "model.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Визуализация метрик"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "model = CatBoostClassifier(\n",
    "    iterations=50,\n",
    "    random_seed=63,\n",
    "    learning_rate=0.5,\n",
    "    custom_loss=['AUC', 'Accuracy']\n",
    ")\n",
    "model.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    verbose=False,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сравнение моделей"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model1 = CatBoostClassifier(\n",
    "    learning_rate=0.7,\n",
    "    iterations=100,\n",
    "    random_seed=0,\n",
    "    train_dir='learing_rate_0.7'\n",
    ")\n",
    "\n",
    "model2 = CatBoostClassifier(\n",
    "    learning_rate=0.01,\n",
    "    iterations=100,\n",
    "    random_seed=0,\n",
    "    train_dir='learing_rate_0.01'\n",
    ")\n",
    "model1.fit(\n",
    "    X_train, y_train,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    cat_features=cat_features,\n",
    "    verbose=False\n",
    ")\n",
    "model2.fit(\n",
    "    X_train, y_train,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    cat_features=cat_features,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import MetricVisualizer\n",
    "MetricVisualizer(['learing_rate_0.01', 'learing_rate_0.7']).start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Оптимальное количество деревьев"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "model = CatBoostClassifier(\n",
    "    iterations=100,\n",
    "    random_seed=63,\n",
    "    learning_rate=0.5,\n",
    "#     use_best_model=False\n",
    ")\n",
    "model.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    verbose=False,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Tree count: ' + str(model.tree_count_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Кросс-валидация"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import cv\n",
    "\n",
    "params = {}\n",
    "params['loss_function'] = 'Logloss'\n",
    "params['iterations'] = 80\n",
    "params['custom_loss'] = 'AUC'\n",
    "params['random_seed'] = 63\n",
    "params['learning_rate'] = 0.5\n",
    "\n",
    "cv_data = cv(\n",
    "    params = params,\n",
    "    pool = Pool(X, label=y, cat_features=cat_features),\n",
    "    fold_count=5,\n",
    "    shuffle=True,\n",
    "    partition_random_seed=0,\n",
    "    plot=True,\n",
    "    stratified=False,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_value = np.min(cv_data['test-Logloss-mean'])\n",
    "best_iter = np.argmin(cv_data['test-Logloss-mean'])\n",
    "\n",
    "print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n",
    "    best_value,\n",
    "    cv_data['test-Logloss-std'][best_iter],\n",
    "    best_iter)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Детектор переобучения"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_with_early_stop = CatBoostClassifier(\n",
    "    iterations=200,\n",
    "    random_seed=63,\n",
    "    learning_rate=0.5,\n",
    "    early_stopping_rounds=20\n",
    ")\n",
    "model_with_early_stop.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    verbose=False,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model_with_early_stop.tree_count_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_with_early_stop = CatBoostClassifier(\n",
    "    eval_metric='AUC',\n",
    "    iterations=200,\n",
    "    random_seed=63,\n",
    "    learning_rate=0.5,\n",
    "    early_stopping_rounds=20\n",
    ")\n",
    "model_with_early_stop.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    verbose=False,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model_with_early_stop.tree_count_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вычисление метрик качества и выбор порога принятия решения"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CatBoostClassifier(\n",
    "    random_seed=63,\n",
    "    iterations=200,\n",
    "    learning_rate=0.03,\n",
    ")\n",
    "model.fit(\n",
    "    X_train, y_train,\n",
    "    cat_features=cat_features,\n",
    "    verbose=False,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://habrastorage.org/webt/y4/1q/yq/y41qyqfm9mcerp2ziys48phpjia.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost.utils import get_roc_curve\n",
    "import sklearn\n",
    "from sklearn import metrics\n",
    "\n",
    "eval_pool = Pool(X_validation, y_validation, cat_features=cat_features)\n",
    "curve = get_roc_curve(model, eval_pool)\n",
    "(fpr, tpr, thresholds) = curve\n",
    "roc_auc = sklearn.metrics.auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 8))\n",
    "lw = 2\n",
    "\n",
    "plt.plot(fpr, tpr, color='darkorange',\n",
    "         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc, alpha=0.5)\n",
    "\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--', alpha=0.5)\n",
    "\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xticks(fontsize=16)\n",
    "plt.yticks(fontsize=16)\n",
    "plt.grid(True)\n",
    "plt.xlabel('False Positive Rate', fontsize=16)\n",
    "plt.ylabel('True Positive Rate', fontsize=16)\n",
    "plt.title('Receiver operating characteristic', fontsize=20)\n",
    "plt.legend(loc=\"lower right\", fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost.utils import get_fpr_curve\n",
    "from catboost.utils import get_fnr_curve\n",
    "\n",
    "(thresholds, fpr) = get_fpr_curve(curve=curve)\n",
    "(thresholds, fnr) = get_fnr_curve(curve=curve)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 8))\n",
    "lw = 2\n",
    "\n",
    "plt.plot(thresholds, fpr, color='blue', lw=lw, label='FPR', alpha=0.5)\n",
    "plt.plot(thresholds, fnr, color='green', lw=lw, label='FNR', alpha=0.5)\n",
    "\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xticks(fontsize=16)\n",
    "plt.yticks(fontsize=16)\n",
    "plt.grid(True)\n",
    "plt.xlabel('Threshold', fontsize=16)\n",
    "plt.ylabel('Error Rate', fontsize=16)\n",
    "plt.title('FPR-FNR curves', fontsize=20)\n",
    "plt.legend(loc=\"lower left\", fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost.utils import select_threshold\n",
    "\n",
    "print(select_threshold(model=model, data=eval_pool, FNR=0.01))\n",
    "print(select_threshold(model=model, data=eval_pool, FPR=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Получение предсказаний"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.predict_proba(data=X_validation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.predict(data=X_validation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Последовательные предсказания"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_gen = model.staged_predict_proba(\n",
    "    data=X_validation,\n",
    "    ntree_start=0, \n",
    "    ntree_end=5, \n",
    "    eval_period=1\n",
    ")\n",
    "try:\n",
    "    for iteration, predictions in enumerate(predictions_gen):\n",
    "        print('Iteration ' + str(iteration) + ', predictions:')\n",
    "        print(predictions)\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Важность признаков"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_feature_importance(prettified=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shap values \n",
    "\n",
    "Классические способы измерения важности признаков обладают существенным недостатком: непонятно, как именно каждый признак влияет на предсказание модели. Shap values (значения Шэпли) - способ измерения важности признаков, исправляющий этот недостаток.\n",
    "\n",
    "Подробнее читайте тут: https://habr.com/ru/post/428213/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shap\n",
    "\n",
    "shap_values = model.get_feature_importance(pool1, type='ShapValues')\n",
    "\n",
    "expected_value = shap_values[0,-1]\n",
    "shap_values = shap_values[:,:-1]\n",
    "\n",
    "print(shap_values.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.summary_plot(shap_values, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shap-values для конкретного объекта."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.initjs()\n",
    "shap.force_plot(expected_value, shap_values[3,:], X.iloc[3,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сохранение модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_best_model = CatBoostClassifier(iterations=10)\n",
    "my_best_model.fit(\n",
    "    X_train, y_train,\n",
    "    eval_set=(X_validation, y_validation),\n",
    "    cat_features=cat_features,\n",
    "    verbose=False\n",
    ")\n",
    "my_best_model.save_model('catboost_model.bin')\n",
    "my_best_model.save_model('catboost_model.json', format='json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_best_model.load_model('catboost_model.bin')\n",
    "print(my_best_model.get_params())\n",
    "print(my_best_model.random_seed_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подбор гиперпараметров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cat = CatBoostClassifier(n_estimators=50)\n",
    "\n",
    "grid = {'learning_rate': [0.03, 0.1],\n",
    "        'depth': [4, 6, 10]}\n",
    "#        'l2_leaf_reg': [1, 3, 5, 7, 9]}\n",
    "\n",
    "grid_search_result = cat.grid_search(grid, X=X_train, y=y_train, verbose=False, plot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search_result['params']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Визуализация деревьев"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.plot_tree(tree_idx=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Хорошее описание гиперпараметров катбуста: https://habr.com/ru/company/otus/blog/527554/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM\n",
    "\n",
    "Особенности LightGBM:\n",
    "\n",
    "* полиственное (leaf-wise), а не поуровневое (depth-wise) построение деревьев\n",
    "\n",
    "* быстрое построение дерева, позволяющее не кодировать категориальные признаки, а использовать исходные категории для получения предикатов\n",
    "\n",
    "* ускорение построения деревьев за счёт бинаризации признаков (объединение значений признака в группы)\n",
    "\n",
    "Почитать подробнее можно тут: https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data = load_boston()\n",
    "X_full = data.data\n",
    "y_full = data.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_full, y_full, test_size=100, \n",
    "                                        random_state=241)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Starting training...')\n",
    "# train\n",
    "gbm = lgb.LGBMRegressor(num_leaves=31,\n",
    "                        learning_rate=0.05,\n",
    "                        n_estimators=20)\n",
    "gbm.fit(X_train, y_train,\n",
    "        eval_set=[(X_test, y_test)],\n",
    "        eval_metric='l1',\n",
    "        early_stopping_rounds=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предсказание и оценка качества"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Starting predicting...')\n",
    "\n",
    "y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration_)\n",
    "\n",
    "print('The rmse of prediction is:', mean_squared_error(y_test, y_pred) ** 0.5)\n",
    "\n",
    "# feature importances\n",
    "print('Feature importances:', list(gbm.feature_importances_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно задать любую (придумать свою, например) метрику качества"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# self-defined eval metric\n",
    "# f(y_true: array, y_pred: array) -> name: string, eval_result: float, is_higher_better: bool\n",
    "# Root Mean Squared Logarithmic Error (RMSLE)\n",
    "def rmsle(y_true, y_pred):\n",
    "    return 'RMSLE', np.sqrt(np.mean(np.power(np.log1p(y_pred) - np.log1p(y_true), 2))), False\n",
    "\n",
    "\n",
    "print('Starting training with custom eval function...')\n",
    "# train\n",
    "gbm.fit(X_train, y_train,\n",
    "        eval_set=[(X_test, y_test)],\n",
    "        eval_metric=rmsle,\n",
    "        early_stopping_rounds=5)\n",
    "\n",
    "print('Starting predicting...')\n",
    "# predict\n",
    "y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration_)\n",
    "# eval\n",
    "print('The rmsle of prediction is:', rmsle(y_test, y_pred)[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Подбор гиперпараметров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# other scikit-learn modules\n",
    "estimator = lgb.LGBMRegressor(num_leaves=31)\n",
    "\n",
    "param_grid = {\n",
    "    'learning_rate': [0.01, 0.1, 1],\n",
    "    'n_estimators': [20, 40]\n",
    "}\n",
    "\n",
    "gbm = GridSearchCV(estimator, param_grid, cv=3)\n",
    "gbm.fit(X_train, y_train)\n",
    "\n",
    "print('Best parameters found by grid search are:', gbm.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предсказание и оценка качества"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = gbm.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "mean_squared_error(y_test, pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "state": {
    "1057714ebc614324aa3ba2cf69408966": {
     "views": [
      {
       "cell_index": 17
      }
     ]
    },
    "8381e9eed05f4a03905ae8a56d7ab4ea": {
     "views": [
      {
       "cell_index": 48
      }
     ]
    },
    "f49684e8c5c44241bfe2c7f577f5cb41": {
     "views": [
      {
       "cell_index": 53
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}