Sklearn/supervised algorithm/XGBoost_in_Machine_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "XGBoost in Machine Learning.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIsI9dS-FNPO"
      },
      "source": [
        "# **Introduction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZdupLwSYFUHh"
      },
      "source": [
        "XGBoost or Gradient Boosting is a machine learning algorithm that goes through cycles to iteratively add models to a set. In this article, I will take you through the XGBoost algorithm in Machine Learning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0EpPpnxScrD4"
      },
      "source": [
        "The cycle of the XGBoost algorithm begins by initializing the whole with a unique model, the predictions of which can be quite naive."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "22KV7aATHIU5"
      },
      "source": [
        "# **The Process of XGBoost Algorithm:**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nQkZsGzvc1pi"
      },
      "source": [
        "- First, we use the current set to generate predictions for each observation in the dataset. To make a prediction, we add the predictions of all the models in the set.\n",
        "- These predictions are used to calculate a loss function.\n",
        "- Then we use the loss function to fit a new model which will be added to the set. Specifically, we determine the parameters of the model so that adding this new model to the set reduces the loss.\n",
        "- Finally, we add the new model to the set, and …\n",
        "then repeat!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpOJPL34rP5Z"
      },
      "source": [
        "# **XGBoost Algorithm in Action**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ePqKoUWTre6h"
      },
      "source": [
        "I’ll start by loading the training and validation data into X_train, X_valid, y_train and y_valid. The dataset, I am using here can be easily downloaded from here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HoOKm_KQsBWL",
        "outputId": "1d16352d-393f-46a2-88dc-22c1ac52da25"
      },
      "source": [
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fw0YMemUsOWj"
      },
      "source": [
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Read the data\n",
        "data = pd.read_csv('/content/drive/MyDrive/Datasets/melb_data.csv')\n",
        "\n",
        "# Select subset of predictors\n",
        "cols_to_use = ['Rooms', 'Distance', 'Landsize', 'BuildingArea', 'YearBuilt']\n",
        "X = data[cols_to_use]\n",
        "\n",
        "# Select target\n",
        "y = data.Price\n",
        "\n",
        "# Separate data into training and validation sets\n",
        "X_train, X_valid, y_train, y_valid = train_test_split(X,y) "
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otrycWU6sqTK"
      },
      "source": [
        "Now, here you will learn how to use the XGBoost algorithm. Here we need to import the scikit-learn API for XGBoost (xgboost.XGBRegressor). This allows us to create and adjust a model like we would in scikit-learn. As you will see in the output, the XGBRegressor class has many adjustable parameters:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IsTLbK6GssO6"
      },
      "source": [
        "from xgboost import XGBRegressor\n",
        "\n",
        "my_model = XGBRegressor()\n",
        "my_model.fit(X_train, y_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZAYMrKnXs3aj"
      },
      "source": [
        "Now, we need to make predictions and evaluate our model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V_A5LMrws-5i",
        "outputId": "2c451630-6381-4aec-a62c-590e92000d0e"
      },
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "predictions = my_model.predict(X_valid)\n",
        "print(\"Mean Absolute Error: \" + str(mean_absolute_error(predictions, y_valid)))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean Absolute Error: 279829.9009295499\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XD-d44CUH_8n"
      },
      "source": [
        "# **Parameter Tuning**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYkOlpS737BL"
      },
      "source": [
        "XGBoost has a few features that can drastically affect the accuracy and speed of training. The first feature you need to understand are:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eJwABNAd3_Jc"
      },
      "source": [
        "**n_estimators**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zaqY98h4Ct6"
      },
      "source": [
        "n_estimators specifies the number of times to skip the modelling cycle described above. It is equal to the number of models we include in the set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BECHRnND4Ih2"
      },
      "source": [
        "- Too low a value results in an underfitting, leading to inaccurate predictions on training data and test data.\n",
        "- Too high a value results in overfitting, resulting in accurate predictions on training data, but inaccurate predictions on test data (which is important to us)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m45TqHHI4THz"
      },
      "source": [
        "Typical the values ​​lie between 100 to 1000, although it all depends a lot on the learning_rate parameter described below. Here is the code to set the number of models in the set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uc_9nhDP4U19"
      },
      "source": [
        "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
        "             colsample_bynode=1, colsample_bytree=1, gamma=0,\n",
        "             importance_type='gain', learning_rate=0.1, max_delta_step=0,\n",
        "             max_depth=3, min_child_weight=1, missing=None, n_estimators=500,\n",
        "             n_jobs=1, nthread=None, objective='reg:linear', random_state=0,\n",
        "             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
        "             silent=None, subsample=1, verbosity=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OA6GZob_4cCd"
      },
      "source": [
        "**early_stopping_rounds**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U6wv3Tnv4eNL"
      },
      "source": [
        "early_stopping_rounds provides a way to automatically find the ideal value for n_estimators. Stopping early causes the iteration of the model to stop when the validation score stops improving, even though we are not stopping hard for n_estimators. It’s a good idea to set n_estimators high and then use early_stopping_rounds to find the optimal time to stop the iteration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbHxfNEq4jH1"
      },
      "source": [
        "Since random chance sometimes causes a single round where validation scores do not improve, you must specify a number for the number of direct deterioration turns to allow before stopping. Setting early_stopping_rounds = 5 is a reasonable choice. In this case, we stop after 5 consecutive rounds of deterioration of validation scores. Now let’s see how we can use early_stopping:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4nFErKq4r1T"
      },
      "source": [
        "my_model = XGBRegressor(n_estimators=500)\n",
        "my_model.fit(X_train, y_train, \n",
        "             early_stopping_rounds=5, \n",
        "             eval_set=[(X_valid, y_valid)],\n",
        "             verbose=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4FdGAi104xp0"
      },
      "source": [
        "**learning_rate**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTaHClVE43Z0"
      },
      "source": [
        "Instead of getting predictions by simply adding up the predictions of each component model, we can multiply the predictions of each model by a small number before adding them.\n",
        "\n",
        "This means that every tree we add to the set helps us less. So we can set a high value for the n_estimators without overfitting. If we use early shutdown, the appropriate number of trees will be determined automatically. Now, let’s see how we can use learning_rate in XGBoost algorithm:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C0TxFTgp45S6",
        "outputId": "cc0ed6cb-e257-431b-a1f9-28c3d9d9f431"
      },
      "source": [
        "my_model = XGBRegressor(n_estimators=1000, learning_rate=0.05)\n",
        "my_model.fit(X_train, y_train, \n",
        "             early_stopping_rounds=5, \n",
        "             eval_set=[(X_valid, y_valid)], \n",
        "             verbose=False)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[07:22:39] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
              "             colsample_bynode=1, colsample_bytree=1, gamma=0,\n",
              "             importance_type='gain', learning_rate=0.05, max_delta_step=0,\n",
              "             max_depth=3, min_child_weight=1, missing=None, n_estimators=1000,\n",
              "             n_jobs=1, nthread=None, objective='reg:linear', random_state=0,\n",
              "             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
              "             silent=None, subsample=1, verbosity=1)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fxR6L_c5E6F"
      },
      "source": [
        "**n_jobs**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxepfNxS5HMr"
      },
      "source": [
        "On larger datasets where execution is a consideration, you can use parallelism to build your models faster. It is common to set the n_jobs parameter equal to the number of cores on your machine. On smaller data sets, this won’t help.\n",
        "\n",
        "The resulting model will not be better, so micro-optimizing the timing of the fit is usually just a distraction. But it’s very useful in large datasets where you would spend a lot of time waiting for the fit command. Now, let’s see how to use this parameter in the XGBoost algorithm:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k4CmOX1O5Ov2"
      },
      "source": [
        "my_model = XGBRegressor(n_estimators=1000, learning_rate=0.05, n_jobs=4)\n",
        "my_model.fit(X_train, y_train, \n",
        "             early_stopping_rounds=5, \n",
        "             eval_set=[(X_valid, y_valid)], \n",
        "             verbose=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fby7CCu5E0-C"
      },
      "source": [
        "# **References**\n",
        "[XGBoost in Machine Learning](https://thecleverprogrammer.com/2020/09/04/xgboost-in-machine-learning/)"
      ]
    }
  ]
}