cpd4.0/notebooks/python_sdk/deployments/xgboost/Use XGBoost to classify tumors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use XGBoost to classify tumors with `ibm-watson-machine-learning`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains steps and code to get data from the IBM Watson Studio Community, create a predictive model, and start scoring new data. It introduces commands for getting data and for basic data cleaning and exploration, model training, model persistance to Watson Machine Learning repository, model deployment, and scoring.\n",
    "\n",
    "Some familiarity with Python is helpful. This notebook uses Python 3.9, XGBoost, and scikit-learn.\n",
    "\n",
    "You will use a publicly available data set, the Breast Cancer Wisconsin (Diagnostic) Data Set, to train an XGBoost Model to classify breast cancer tumors (as benign or malignant) from 569 diagnostic images based on measurements such as radius, texture, perimeter and area. XGBoost is short for “E**x**treme **G**radient **Boost**ing”.\n",
    "\n",
    "The XGBoost classifier makes its predictions based on the majority vote from collection of models which are a set of classification trees. It uses the combination of weak learners to create a single strong learner. It’s a sequential training process, whereby new learners focus on the misclassified examples of previous learners.\n",
    "\n",
    "\n",
    "## Learning goals\n",
    "\n",
    "You will learn how to:\n",
    "\n",
    "-  Load a CSV file into numpy array\n",
    "-  Explore data\n",
    "-  Prepare data for training and evaluation\n",
    "-  Create an XGBoost machine learning model\n",
    "-  Train and evaluate a model\n",
    "-  Use cross-validation to optimize model's hyperparameters\n",
    "-  Persist a model in Watson Machine Learning repository\n",
    "-  Deploy a model for online scoring\n",
    "-  Score sample data\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "This notebook contains the following parts:\n",
    "\n",
    "1.\t[Setup](#setup)\n",
    "2.\t[Load and explore the data](#load)\n",
    "3.\t[Create the XGBoost model](#model)\n",
    "4.\t[Persist model](#upload)\n",
    "5.\t[Deployment](#deploy)\n",
    "6.  [Score the model](#score)\n",
    "7.  [Clean up](#cleanup)\n",
    "8.\t[Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Set up the environment\n",
    "\n",
    "Before you use the sample code in this notebook, you must perform the following setup tasks:\n",
    "\n",
    "-  Contact with your Cloud Pack for Data administrator and ask him for your account credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connection to WML\n",
    "\n",
    "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = 'PASTE YOUR USERNAME HERE'\n",
    "api_key = 'PASTE YOUR API_KEY HERE'\n",
    "url = 'PASTE THE PLATFORM URL HERE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "wml_credentials = {\n",
    "    \"username\": username,\n",
    "    \"apikey\": api_key,\n",
    "    \"url\": url,\n",
    "    \"instance_id\": 'openshift',\n",
    "    \"version\": '4.0'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively you can use `username` and `password` to authenticate WML services.\n",
    "\n",
    "```\n",
    "wml_credentials = {\n",
    "    \"username\": ***,\n",
    "    \"password\": ***,\n",
    "    \"url\": ***,\n",
    "    \"instance_id\": 'openshift',\n",
    "    \"version\": '4.0'\n",
    "}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install and import the `ibm-watson-machine-learning` package\n",
    "**Note:** `ibm-watson-machine-learning` documentation can be found <a href=\"https://ibm.github.io/watson-machine-learning-sdk/\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U ibm-watson-machine-learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "client = APIClient(wml_credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with spaces\n",
    "\n",
    "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
    "\n",
    "- Click New Deployment Space\n",
    "- Create an empty space\n",
    "- Go to space `Settings` tab\n",
    "- Copy `space_id` and paste it below\n",
    "\n",
    "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
    "\n",
    "**Action**: Assign space ID below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_id = 'PASTE YOUR SPACE ID HERE'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `list` method to print all existing spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to interact with all resources available in Watson Machine Learning, you need to set **space** which you will be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.set.default_space(space_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"load\"></a>\n",
    "## 2. Load and explore the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you will load the data as a numpy array and perform a basic exploration.\n",
    "\n",
    "To load the data as a numpy array, user `wget` to download the data, then use the `genfromtxt` method to read the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: First, you need to install the required packages. You can do this by running the following code. Run it only one time.<BR><BR>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install wget --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BreastCancerWisconsinDataSet.csv\n"
     ]
    }
   ],
   "source": [
    "import wget, os\n",
    "\n",
    "WisconsinDataSet = 'BreastCancerWisconsinDataSet.csv' \n",
    "if not os.path.isfile(WisconsinDataSet):\n",
    "    link_to_data = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.0/data/cancer/' + WisconsinDataSet\n",
    "    print(link_to_data)\n",
    "    WisconsinDataSet = wget.download(link_to_data)\n",
    "\n",
    "print(WisconsinDataSet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The csv file **BreastCancerWisconsinDataSet.csv** is downloaded. Run the code in the next cells to load the file to the numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(842302, 'M', 17.99, 10.38, 122.8, 1001., 0.1184, 0.2776, 0.3001, 0.1471, 0.2419, 0.07871, 1.095, 0.9053, 8.589, 153.4, 0.006399, 0.04904, 0.05373, 0.01587, 0.03003, 0.006193, 25.38, 17.33, 184.6, 2019., 0.1622, 0.6656, 0.7119, 0.2654, 0.4601, 0.1189)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "np_data = np.genfromtxt(WisconsinDataSet, delimiter=',', names=True, dtype=None, encoding='utf-8')\n",
    "print(np_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code in the next cell to view the feature names and data storage types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('id', '<i8'), ('diagnosis', '<U1'), ('radius_mean', '<f8'), ('texture_mean', '<f8'), ('perimeter_mean', '<f8'), ('area_mean', '<f8'), ('smoothness_mean', '<f8'), ('compactness_mean', '<f8'), ('concavity_mean', '<f8'), ('concave_points_mean', '<f8'), ('symmetry_mean', '<f8'), ('fractal_dimension_mean', '<f8'), ('radius_se', '<f8'), ('texture_se', '<f8'), ('perimeter_se', '<f8'), ('area_se', '<f8'), ('smoothness_se', '<f8'), ('compactness_se', '<f8'), ('concavity_se', '<f8'), ('concave_points_se', '<f8'), ('symmetry_se', '<f8'), ('fractal_dimension_se', '<f8'), ('radius_worst', '<f8'), ('texture_worst', '<f8'), ('perimeter_worst', '<f8'), ('area_worst', '<f8'), ('smoothness_worst', '<f8'), ('compactness_worst', '<f8'), ('concavity_worst', '<f8'), ('concave_points_worst', '<f8'), ('symmetry_worst', '<f8'), ('fractal_dimension_worst', '<f8')]\n"
     ]
    }
   ],
   "source": [
    "# Display the feature names and data storage types.\n",
    "print(np_data.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows: 569\n",
      "Number of columns: 32\n"
     ]
    }
   ],
   "source": [
    "# Display the number of records and features.\n",
    "print('Number of rows: {}'.format(np_data.size))\n",
    "print('Number of columns: {}'.format(len(np_data[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the data set has 569 records and 32 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"model\"></a>\n",
    "## 3. Create an XGBoost model\n",
    "\n",
    "In this section you will learn how to train and test an XGBoost model.\n",
    "\n",
    "- [3.1. Prepare the data](#prepare)\n",
    "- [3.2. Create the XGBoost model](#create)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Update `xgboost` to ensure you have `1.5` version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U xgboost==1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Prepare data<a id=\"prepare\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can prepare your data for model building. You will use the `diagnosis` column as your target variable so you must remove it from the set of predictors. You must also remove the `id` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = 1 * (np_data['diagnosis'] == 'M')\n",
    "X = np.array([list(r)[2:] for r in np_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data set into: \n",
    "- Train data set\n",
    "- Test data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data set and create two data sets.\n",
    "# from sklearn.cross_validation import train_test_split \n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.35, random_state=143)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training records: 369\n",
      "Number of testing records : 200\n"
     ]
    }
   ],
   "source": [
    "# List the number of records in each data set.\n",
    "print(\"Number of training records: \" + str(X_train.shape[0]))\n",
    "print(\"Number of testing records : \" + str(X_test.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data has been successfully split into two data sets:\n",
    "- The train data set, which is the largest group, will be used for training\n",
    "- The test data set will be used for model evaluation and is used to test the assumptions of the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Create the XGBoost model<a id=\"create\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by importing the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the libraries you need to create the XGBoost model.\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.1. Create an XGBoost classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you create an XGBoost classifier with default hyperparameter values and you will call it *xgb_model*. \n",
    "\n",
    "**Note** The next sections show you how to improve this base model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Usage of default or `n_jobs=-1` parameter in XGBoost classifier is not recommended as underlying process often cannot correctly discover number of CPUs / threads allowed.  \n",
    "Other ways to controll the number of cores used is through environmental variables `OMP_NUM_THREADS` and `MKL_NUM_THREADS`, which should be set by default if this notebook is executed inside \n",
    "Watson Studio.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the XGB classifier, xgb_model.\n",
    "xgb_model = XGBClassifier(use_label_encoder=False, n_jobs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the default parameters for *xgb_model*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'objective': 'binary:logistic', 'base_score': None, 'booster': None, 'colsample_bylevel': None, 'colsample_bynode': None, 'colsample_bytree': None, 'gamma': None, 'gpu_id': None, 'interaction_constraints': None, 'learning_rate': None, 'max_delta_step': None, 'max_depth': None, 'min_child_weight': None, 'monotone_constraints': None, 'n_jobs': None, 'num_parallel_tree': None, 'random_state': None, 'reg_alpha': None, 'reg_lambda': None, 'scale_pos_weight': None, 'subsample': None, 'tree_method': None, 'validate_parameters': None, 'verbosity': None}\n"
     ]
    }
   ],
   "source": [
    "# List the default parameters.\n",
    "print(xgb_model.get_xgb_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your XGBoost classifier, *xgb_model*, is set up, you can train it by invoking the fit method. You will also evaluate *xgb_model* while the train and test data are being trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-error:0.01626\tvalidation_1-error:0.06000\n",
      "[1]\tvalidation_0-error:0.00813\tvalidation_1-error:0.06000\n",
      "[2]\tvalidation_0-error:0.00813\tvalidation_1-error:0.05500\n",
      "[3]\tvalidation_0-error:0.00542\tvalidation_1-error:0.05000\n",
      "[4]\tvalidation_0-error:0.00542\tvalidation_1-error:0.05000\n",
      "[5]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04500\n",
      "[6]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05500\n",
      "[7]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05000\n",
      "[8]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05000\n",
      "[9]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05000\n",
      "[10]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05500\n",
      "[11]\tvalidation_0-error:0.00271\tvalidation_1-error:0.05000\n",
      "[12]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04500\n",
      "[13]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04000\n",
      "[14]\tvalidation_0-error:0.00271\tvalidation_1-error:0.04000\n",
      "[15]\tvalidation_0-error:0.00271\tvalidation_1-error:0.03000\n",
      "[16]\tvalidation_0-error:0.00271\tvalidation_1-error:0.03500\n",
      "[17]\tvalidation_0-error:0.00271\tvalidation_1-error:0.03000\n",
      "[18]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04000\n",
      "[19]\tvalidation_0-error:0.00000\tvalidation_1-error:0.03500\n",
      "[20]\tvalidation_0-error:0.00000\tvalidation_1-error:0.03500\n",
      "[21]\tvalidation_0-error:0.00000\tvalidation_1-error:0.03500\n",
      "[22]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04000\n",
      "[23]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[24]\tvalidation_0-error:0.00000\tvalidation_1-error:0.03500\n",
      "[25]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[26]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04000\n",
      "[27]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[28]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[29]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[30]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[31]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[32]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[33]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[34]\tvalidation_0-error:0.00000\tvalidation_1-error:0.04500\n",
      "[35]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[36]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[37]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[38]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[39]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[40]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[41]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[42]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[43]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[44]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[45]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[46]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[47]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[48]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[49]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[50]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[51]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05000\n",
      "[52]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[53]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[54]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[55]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[56]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[57]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[58]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[59]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[60]\tvalidation_0-error:0.00000\tvalidation_1-error:0.05500\n",
      "[61]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[62]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[63]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[64]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[65]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[66]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[67]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[68]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[69]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[70]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[71]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[72]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[73]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[74]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[75]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[76]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[77]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[78]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[79]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[80]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[81]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[82]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[83]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[84]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[85]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[86]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[87]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[88]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[89]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[90]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[91]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[92]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[93]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[94]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[95]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[96]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[97]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[98]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n",
      "[99]\tvalidation_0-error:0.00000\tvalidation_1-error:0.06000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "              importance_type='gain', interaction_constraints='',\n",
       "              learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "              min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "              n_estimators=100, n_jobs=12, num_parallel_tree=1, random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
       "              tree_method='exact', use_label_encoder=False,\n",
       "              validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train and evaluate.\n",
    "xgb_model.fit(X_train, y_train, eval_metric=['error'], eval_set=[((X_train, y_train)),(X_test, y_test)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can also use a pandas dataFrame instead of the numpy array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the model performance evaluated during the training process to assess model overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the library\n",
    "from matplotlib import pyplot\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAGDCAYAAADkllOoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4nElEQVR4nO3de3xcdZ3/8denaZu06T2ZFmjoDUovam0hCyqyFBQFbyDeYGUFb/zYFXV3f6zCul6W/bHi6v5cXd0Lq8iy63r5oSgqys1WcFGhyLVNUsq1aWkzSS9p06Ztks/vj3NOcjKdNDPJJHNm5v18PPrIzLnM+WYY8p7v5Xy/5u6IiIhIMk0odgFERERkaApqERGRBFNQi4iIJJiCWkREJMEU1CIiIgmmoBYREUkwBbWIiEiCKahFSoSZrTez3WZWXeyyiMj4UVCLlAAzWwScBTjwtnG87sTxupaIZKegFikN7wN+C9wCXB5tNLMTzeyHZpY2sw4z+1ps34fNrMnM9pnZJjM7NdzuZnZy7LhbzOz/hI/XmlmrmX3SzHYA3zKz2Wb20/Aau8PHDbHz55jZt8xse7j/R+H2p8zsrbHjJplZu5mtGas3SaQcKahFSsP7gG+H/95oZvPMrAr4KfACsAiYD3wXwMzeBXwuPG8GQS28I8drHQfMARYCVxL8nfhW+HwBcBD4Wuz4/wSmAi8D5gJfDrffClwWO+5NwEvu/miO5RARwDTXt0iymdlrgXXA8e7ebmbNwL8R1LDvCLf3ZJxzF3Cnu38ly+s5sNTdt4TPbwFa3f2vzWwtcDcww927hyjPamCdu882s+OBbUCdu+/OOO4EoAWY7+6dZnYb8JC7//0I3wqRiqQatUjyXQ7c7e7t4fP/DredCLyQGdKhE4FnRni9dDykzWyqmf2bmb1gZp3A/cCssEZ/IrArM6QB3H078D/AO8xsFnABQYuAiORBA0VEEszMpgDvBqrCPmOAamAWsBNYYGYTs4T1VuCkIV72AEFTdeQ4oDX2PLOZ7X8Dy4Az3H1HWKN+FLDwOnPMbJa778lyrf8APkTwt+Y37r5tiDKJyBBUoxZJtouAXmAlsDr8twJ4INz3EnCjmdWaWY2ZnRme9w3gGjM7zQInm9nCcN9jwB+ZWZWZnQ+cPUwZphP0S+8xsznAZ6Md7v4S8HPgn8NBZ5PM7A9j5/4IOBX4OEGftYjkSUEtkmyXA99y9xfdfUf0j2Aw16XAW4GTgRcJasXvAXD3/wfcQNBMvo8gMOeEr/nx8Lw9wHvDfcfyj8AUoJ2gX/wXGfv/GDgCNANtwJ9FO9z9IPADYDHww9x/bRGJaDCZiIwpM/sMcIq7XzbswSJyFPVRi8iYCZvKP0hQ6xaREVDTt4iMCTP7MMFgs5+7+/3FLo9IqVLTt4iISIKpRi0iIpJgCmoREZEES9xgsvr6el+0aFGxiyEiIjJuHnnkkXZ3T2Xbl7igXrRoERs2bCh2MURERMaNmb0w1D41fYuIiCSYglpERCTBFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLSIikmAKahERkQRTUIuIiCTYsEFtZjebWZuZPTXEfjOzr5rZFjN7wsxOje273MyeDv9dXsiCi4iIVIJcatS3AOcfY/8FwNLw35XAv0D/gvGfBc4ATgc+a2azR1NYERGRSjPsXN/ufr+ZLTrGIRcCt3qwsPVvzWyWmR0PrAXucfddAGZ2D0Hgf2fUpZaCcncefn43XYd7il0UEZGS8PITZpKaXj0u1yrEohzzga2x563htqG2H8XMriSojbNgwYICFEny8eAzHbz3G78rdjFERErGTX98Gm942XHjcq1ErJ7l7jcBNwE0NjZ6kYtTce7auIOaSRP49ofOYIJZsYsjIpJ4S+qnjdu1ChHU24ATY88bwm3bCJq/49vXF+B6UkDuzr2bdnLW0hSnLZxT7OKIiEiGQtyedQfwvnD096uAve7+EnAX8AYzmx0OIntDuE0SZOP2Trbv7ea8FfOKXRQREcli2Bq1mX2HoGZcb2atBCO5JwG4+78CdwJvArYAB4D3h/t2mdnfAg+HL3V9NLBMkuPepp2Ywbkr5ha7KCIikkUuo74vHWa/Ax8ZYt/NwM0jK5qMh3ubdnLqgtnUTxuf0YsiIpIfzUxWwbbvOchT2zp5vZq9RUQSS0Fdwe5r2gnAeSsV1CIiSaWgrmD3NLWxuL6Wk1K1xS6KiIgMQUFdofZ1H+E3z7Rz3sp5mO6dFhFJLAV1hbp/cztHel390yIiCaegrlD3bNrB7KmTOHXBrGIXRUREjkFBXYGO9Pbxy+Y2zl0+j4lV+giIiCRZIub6LiU/fWI7f/ezJvpKeEbynj6ns7uH81ZqkhMRkaRTUOfpv377Aj19zjnLSjvkZkyZyDnLS/t3EBGpBArqPOw5cJiHn9/NVWcv4S/fuLzYxRERkQqgDso8rGtpo7fPOW/l+KxBKiIioqDOw72b2khNr2bV/JnFLoqIiFQIBXWODvX0sr6ljdevmMuECZogRERExoeCOke/fXYXXYd7NS+2iIiMKwV1ju7ZtIMpk6p4zUn1xS6KiIhUEAV1Dtydeze1cdbSemomVRW7OCIiUkEU1DnYuL2THZ3davYWEZFxp6DOwd2bdjLB4FxNECIiIuNMQZ2Dezft5LSFs6mbVl3sooiISIVRUA9j256DbHqpU8tBiohIUWgK0Qy/f3E3v2pJ9z/fvHMfAK9X/7SIiBSBgjrDdT94kpYwnCONC2dzUmpakUokIiKVTEEd82LHAVp27uOv37yCD521pNjFERERUR913D1NOwF0G5aIiCSGgjrmnk07OGXeNBbW1Ra7KCIiIoCCul+01rRq0yIikiQK6tD6ljS9fa7bsEREJFEU1KF7Nu0kNb2aVzbMKnZRRERE+imoCdaa/tXmtNaaFhGRxFFQE6w1vf9Qj5q9RUQkcRTUBHN5T5lUxZkna61pERFJlooPanfn3qadWmtaREQSqeKDeuP2Tl7a2625vEVEJJEqbgrR7iO97D5wuP/5Tx7fjmmtaRERSaiKC+q3fe3XbN65f9C2xoWzqdda0yIikkAVFdQ9vX083baf81bO43WxGvSrltQVsVQiIiJDq6igbt9/GHdYuyzFJacvKHZxREREhlVRg8l2dnYDcNyMmiKXREREJDcVGdTzFNQiIlIiKiuo9x0CYO4MDRwTEZHSkFNQm9n5ZtZiZlvM7Nos+xea2X1m9oSZrTezhti+L5jZU+G/9xSy8PnaubebqglGXa2CWkRESsOwQW1mVcDXgQuAlcClZrYy47AvAbe6+yrgeuDz4blvBk4FVgNnANeY2YyClT5POzu7SU2rpkoLb4iISInIpUZ9OrDF3Z9198PAd4ELM45ZCfwyfLwutn8lcL+797h7F/AEcP7oiz0yO/cdYp6avUVEpITkEtTzga2x563htrjHgYvDx28HpptZXbj9fDObamb1wDnAiaMr8si1dXYzVwPJRESkhBRqMNk1wNlm9ihwNrAN6HX3u4E7gQeB7wC/AXozTzazK81sg5ltSKfTBSrS0XZ2dqtGLSIiJSWXoN7G4FpwQ7itn7tvd/eL3X0N8Klw257w5w3uvtrdzwMM2Jx5AXe/yd0b3b0xlUqN7DcZxqGeXnYfOMK86apRi4hI6cglqB8GlprZYjObDFwC3BE/wMzqzSx6reuAm8PtVWETOGa2ClgF3F2owuejrTO4NWveTAW1iIiUjmGnEHX3HjO7GrgLqAJudveNZnY9sMHd7wDWAp83MwfuBz4Snj4JeMDMADqBy9y9p/C/xvDa9mmyExERKT05zfXt7ncS9DXHt30m9vg24LYs53UTjPwuuh17wxq1+qhFRKSEVMzMZP3Th6qPWkRESkjlBPW+biZXTWDW1EnFLoqIiEjOKiao2zoPMXdGNWF/uYiISEmomKAO7qFWs7eIiJSWCgtqDSQTEZHSUjFB3dZ5iLkaSCYiIiWmIoK661AP+w71cJwmOxERkRJTEUHdf2uWmr5FRKTEVEhQh5OdqOlbRERKTEUEdTR9qJa4FBGRUlMRQa2mbxERKVUVEtSHmDq5imnVOU1tLiIikhgVEtTBZCealUxEREpNRQR1cA+1mr1FRKT0VERQ79yn6UNFRKQ0lX1Quzs79nZrshMRESlJZR/UnQd7ONTTp6ZvEREpSWUf1Dv3RbdmqUYtIiKlp/yDulNBLSIipasCgjqcPlSTnYiISAmqgKAOpw/VPN8iIlKCyj6o2zq7mVEzkSmTq4pdFBERkbyVfVDv7Dyk/mkRESlZ5R/UmuxERERKWPkH9V4FtYiIlK6yDuq+Pqdt3yGN+BYRkZJV1kF9uLeP95+5iDOW1BW7KCIiIiNS1gs010yq4lNvXlnsYoiIiIxYWdeoRURESp2CWkREJMEU1CIiIgmmoBYREUkwBbWIiEiCKahFREQSTEEtIiKSYApqERGRBFNQi4iIJJiCWkREJMEU1CIiIgmmoBYREUkwBbWIiEiC5RTUZna+mbWY2RYzuzbL/oVmdp+ZPWFm682sIbbv781so5k1mdlXzcwK+QuIiIiUs2GD2syqgK8DFwArgUvNLHPtyC8Bt7r7KuB64PPhua8BzgRWAS8H/gA4u2ClFxERKXO51KhPB7a4+7Pufhj4LnBhxjErgV+Gj9fF9jtQA0wGqoFJwM7RFlpERKRS5BLU84Gtseet4ba4x4GLw8dvB6abWZ27/4YguF8K/93l7k2ZFzCzK81sg5ltSKfT+f4OIiIiZatQg8muAc42s0cJmra3Ab1mdjKwAmggCPdzzeyszJPd/SZ3b3T3xlQqVaAiiYiIlL6JORyzDTgx9rwh3NbP3bcT1qjNbBrwDnffY2YfBn7r7vvDfT8HXg08UICyi4iIlL1catQPA0vNbLGZTQYuAe6IH2Bm9WYWvdZ1wM3h4xcJatoTzWwSQW37qKZvERERyW7YoHb3HuBq4C6CkP2+u280s+vN7G3hYWuBFjPbDMwDbgi33wY8AzxJ0I/9uLv/pLC/goiISPkydy92GQZpbGz0DRs2FLsYIiIi48bMHnH3xmz7NDOZiIhIgimoRUREEkxBLSIikmAKahERkQRTUIuIiCSYglpERCTBFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLSIikmAKahERkQRTUIuIiCSYglpERCTBFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLSIikmAKahERkQRTUIuIiCSYglpERCTBFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLSIikmATi10AERGpbEeOHKG1tZXu7u5iF2XM1dTU0NDQwKRJk3I+R0EtIiJF1drayvTp01m0aBFmVuzijBl3p6Ojg9bWVhYvXpzzeWr6FhGRouru7qaurq6sQxrAzKirq8u75UBBLSIiRVfuIR0Zye+poBYREUkwBbWIiFS0jo4OVq9ezerVqznuuOOYP39+//PDhw8f89wNGzbwsY99bEzLp8FkIiJS0erq6njssccA+NznPse0adO45ppr+vf39PQwcWL2uGxsbKSxsXFMy6catYiISIYrrriCq666ijPOOINPfOITPPTQQ7z61a9mzZo1vOY1r6GlpQWA9evX85a3vAUIQv4DH/gAa9euZcmSJXz1q18tSFlUoxYRkcT4m59sZNP2zoK+5soTZvDZt74s7/NaW1t58MEHqaqqorOzkwceeICJEydy77338ld/9Vf84Ac/OOqc5uZm1q1bx759+1i2bBl/8id/ktc909koqEVERLJ417veRVVVFQB79+7l8ssv5+mnn8bMOHLkSNZz3vzmN1NdXU11dTVz585l586dNDQ0jKocCmoREUmMkdR8x0ptbW3/409/+tOcc8453H777Tz//POsXbs26znV1dX9j6uqqujp6Rl1OdRHLSIiMoy9e/cyf/58AG655ZZxvbaCWkREZBif+MQnuO6661izZk1Basn5MHcf/iCz84GvAFXAN9z9xoz9C4GbgRSwC7jM3VvN7Bzgy7FDlwOXuPuPhrpWY2Ojb9iwId/fQ0RESlRTUxMrVqwodjHGTbbf18wecfes93kNW6M2syrg68AFwErgUjNbmXHYl4Bb3X0VcD3weQB3X+fuq919NXAucAC4O6/fSEREpILl0vR9OrDF3Z9198PAd4ELM45ZCfwyfLwuy36AdwI/d/cDIy2siIhIpcklqOcDW2PPW8NtcY8DF4eP3w5MN7O6jGMuAb4zkkJKnn7/n/AvZ8JQ3RrbHoEvLoX9beNbLhERyVuhBpNdA5xtZo8CZwPbgN5op5kdD7wCuCvbyWZ2pZltMLMN6XS6QEWqYNsegZ1Pwd7W7Pufux+62qB98/iWS0RE8pZLUG8DTow9bwi39XP37e5+sbuvAT4VbtsTO+TdwO3unvUOcXe/yd0b3b0xlUrlU37J5kB78DPdnH1/W7i9q318yiMiIiOWS1A/DCw1s8VmNpmgCfuO+AFmVm9m0WtdRzACPO5S1Ow9fro6gp9DBXW0/YCCWkQk6Yadmczde8zsaoJm6yrgZnffaGbXAxvc/Q5gLfB5M3PgfuAj0flmtoigRv6rwhdfsuoKuw+yBXVf30CTdxToIiIVrKOjg9e97nUA7Nixg6qqKqLW3YceeojJkycf8/z169czefJkXvOa14xJ+XKaQtTd7wTuzNj2mdjj24Dbhjj3eY4efCZjKaopt2UJ6r0vwpFw4H2XxgOIiAy3zOVw1q9fz7Rp08YsqDUzWbnp7YGDu4PH6ZajR36nWwYeq+lbRCSrRx55hLPPPpvTTjuNN77xjbz00ksAfPWrX2XlypWsWrWKSy65hOeff55//dd/5ctf/jKrV6/mgQceKHhZtChHuTm4K/iZWgHpJujcBjNjK7e0NQ3s12AyEUman18LO54s7Gse9wq44Mbhjwu5Ox/96Ef58Y9/TCqV4nvf+x6f+tSnuPnmm7nxxht57rnnqK6uZs+ePcyaNYurrroq71p4PhTU5SYK30WvDYI63Tw4qNMtMO04qDsJdj1bnDKKiCTYoUOHeOqppzjvvPMA6O3t5fjjjwdg1apVvPe97+Wiiy7ioosuGpfyKKjLTdTvvOhMePjfg37qk18/sD/dBHOXw9Q62Pq74pRRRGQoedR8x4q787KXvYzf/OY3R+372c9+xv33389PfvITbrjhBp58ssC1/yzUR11uon7n1HKYWj945HdfH6Q3B/tqU3BgV7BNRET6VVdXk06n+4P6yJEjbNy4kb6+PrZu3co555zDF77wBfbu3cv+/fuZPn06+/btG7PyKKjLTXTL1dR6mLticFDv3QpHusKgrgfvhe49RSmmiEhSTZgwgdtuu41PfvKTvPKVr2T16tU8+OCD9Pb2ctlll/GKV7yCNWvW8LGPfYxZs2bx1re+ldtvv12DySRHB9oBg6lzILUMnvh+MPLbbGDE99wVsCecvr2rPThWRET43Oc+1//4/vvvP2r/r3/966O2nXLKKTzxxBNjVibVqMtNVxqmzIYJVUHN+VAndG4P9qWjEd/LoDZcM0W3aImIJJqCutx0tQf9zxAENQw0f0cjvqfMHjhGk56IiCSagrrcHOgI+p8haOKGgaBuawpq0xD0YYPupRaRRPChluUtMyP5PRXU5aarPbj1CoLAnloXBLV7UKOOwjs65oDm+xaR4qqpqaGjo6Psw9rd6ejooKamJq/zNJis3HSlg3uoI6nlwb3U/SO+wxr1xMlQPVM1ahEpuoaGBlpbW0mny78rrqamhoaGhuEPjFFQl5O+3mCe79rYmt6p5fDkbQMLdKRWDOyrrVMftYgU3aRJk1i8eHGxi5FYavouJwd2AT7Q/wzhyO+98Oz68PmygX21KY36FhFJOAV1OYlCN7r1CoLpQgE2/RimzRt8z/TUeq1JLSKScArqchI1Y2fWqAE6WwfXpiEIdNWoRUQSTUFdTqKBYfE+6toUTAlr0fH+aQgC/UCH5vsWEUkwBXU5iW61qo3VqM0GatVH1ahT0Nej+b5FRBJMo77LSVSjnpIxd/fc5fDigwP3UEeiQD/QcfR838//z7HvsTaDhWfmPk/4gV3BdKazF+V2vIiIAArq8hLN812V8Z+14XR4/HtHB3U06UlXO9QvHdi+6zm45U3DX+8PPgxv/lJuZbv3c/Dib+Dqh3M7XkREAAV1eTnQPrh/OrLqPXDKG4MQj+uvUWcMKNv5VPDznTdDfUZzeeSOqweOy8XerQMrdomISM4U1OWkq2PwiO/IhAnZm6j75/vOmPQkmht86Ruhelr2ax3/Stj4o4ElNIctWzv0HITDXTC5dvjjRUQE0GCy8nKgffA91MOJatSZ91K3NcPMBUOHNAQjyLv3wP62HMsWXkNTloqI5EVBXU660tlr1EOZWA3VM45u+k63HD1CPFO0P6p9H4v7QK1dQS0ikhcFdbno6w1GVmfroz6WqXWDw7OvF9o3D8xoNpTMJTSP5dA+6D0cPNYEKyIieVFQl4uDuwEffA91LmrrB4fn7ueh99DAvddDmTYPambmFtTx11eNWkQkLwrqchEF4NQ8+qghnO87Fp5tTcHPzFnMMpkFx6RbcihbrA9cNWoRkbwoqMtF/4IcI6hRx4M6qiGnThn+3NSyINiHW+w9Pqpcy2qKiORFQV0uogDMt4+6NpzvOwrbdDPMPBGqpw9/7twVcHDX8M3Z0ZeICRO1WpeISJ4U1OWiv+k7zxr11HroOwLde4Pn6ebhR3xH+kd+N+VWtjknqelbRCRPCupyEd2nnOvc25H4fN99vdD+9PADySJRP/Zw/dQHOmBSLcw6UYPJRETypJnJykVXO9TMgqpJ+Z1XmzE7WU937kE9/TionjkwAG3IsqWDiVim1kN6c37lExGpcArqctGVzr9/GmLTiLYP1MozF+8Yillwv/VwNequ9uA6mbeCiYjIsNT0XS4OdOQ/4hsGL8wRjfiuz2HEdyS1bPg+6gPtwXWm1sGRA3D4QP7lFBGpUArqctHVnv891DC4Rt3WDDMaoGZG7uenVgRfEo7V99zVEdT2oxq/atUiIjlTUJeLqNaar0k1MHlaELT5jPiORMcP1U8dzfM9te7o/nARERmWgroc9PWFTd8j6KOGcNKTtnCO7xz7pyPDzfl9eH8wJWltfaz2rnupRURypaAuBwd3g/flfw91ZGo9bHskHPGdZ416+vHBClxDBXX8/u5oCU41fYuI5ExBXQ5GOn1opLY+WIwDhp/jO5NZcDvXUCO/o5Hkg2rUCmoRkVwpqMvBSBfkiMRr4rnM8Z0pmvM7m/6pTeuDaUmrqtVHLSKSBwV1ORjpPN+RqCY+/YRg6cp8pZYHtfpsNeV407fZwNziIiKSk5yC2szON7MWM9tiZtdm2b/QzO4zsyfMbL2ZNcT2LTCzu82sycw2mdmiApZfoDBN3xBMXjIS0XnZ+qkzyza1Tk3fIiJ5GDaozawK+DpwAbASuNTMVmYc9iXgVndfBVwPfD6271bgi+6+AjgdaCtEwSUmGkU92qbvXKcOzZQ6RlB3tcPEKTC5Nniu2clERPKSyxSipwNb3P1ZADP7LnAhsCl2zErgL8LH64AfhceuBCa6+z0A7r6/MMWucJvvgqY7Bp63PhI0Wec7z3ekdpRBPWM+TJ4eTJiSqat9cJN8bQo6thx93N5W2Hg7vPrqoIlcRESA3Jq+5wNbY89bw21xjwMXh4/fDkw3szrgFGCPmf3QzB41sy+GNfRBzOxKM9tgZhvSaQ00Gtb6z8OTP4Bn1gX/DnXCsjeP/PWOXw0nngFL1o7sfLNwKtEhmr5rYzX9qfXZ76P+/a1w91/Dvh0jK4OISJkq1KIc1wBfM7MrgPuBbUBv+PpnAWuAF4HvAVcA34yf7O43ATcBNDY2eoHKVJ76+oJboU67Ai64sTCvOS0FH7x7dK8xd3lQ0890VI26Do50wZGDMGnKwPZo1PiBdphx/OjKIiJSRnKpUW8DTow9bwi39XP37e5+sbuvAT4VbttDUPt+zN2fdfcegibxUwtQ7sq1d2uwsMVIB36NldTyYPR5Zm05c7GQoe6lju7D1kAzEZFBcgnqh4GlZrbYzCYDlwB3xA8ws3ozi17rOuDm2LmzzCyqUp3L4L5tyVfUvDzS/uSxksoylWg0z3c8qKPadfxe6p7DsOuZcLuCWkQkbtigDmvCVwN3AU3A9919o5ldb2ZvCw9bC7SY2WZgHnBDeG4vQbP4fWb2JGDAvxf8t6gk/UGd51SfYy0qTzyoD3cF05LGJ1TpX1YzVvPe9Qz09YTbFdQiInE59VG7+53AnRnbPhN7fBtw2xDn3gOsGkUZJa6tGaYdB1NmF7skg81sCFbhigd1tvu7o1vI4jXn+KxmqlGLiAxSqMFkMl5GshTleMg28rv//u5sNepYIKdbAAsW91CNWkRkEE0hWkrcg1DLdynK8ZJaMXhxjmw16uoZMGHS4JpzuhnmLIYZJ6hGLSKSQUFdSvZuDW5tSmKNGoJy7d8JB3YFz+MLckTMggFlmUGdWh6ui62gFhGJU1CXkqi2mu9SlOMlquln3mqVuU52bd1Abbv3SDBTWRTUavoWERlEQV1KokFXSa5RA6Rjk5dMrBmY5zsyNVZz7ghHfKeWD94uIiKAgrq0pFtg2jyYOqfYJcluRgNMqo3VqDsGlreMi9ec47eb1dZD956gli0iIoCCurSkm5JbmwaYMCEoX1Tzz5zsJBLvo043Awb1p2S/x1pEpMIpqEtFNOI7qf3TkdTygRr1gfbsQT21Dg7vhyPdQVDPXgiTpw49vaiISAVTUJeKva1BuCW5Rg3BHOT7d8DB3QNN35ni91K3NQ98+ch2j7WISIVTUJeK/hHfCZvjO1NUvnTLMWrU4bZ9O8IR38sGb1eNWkSkn4K6VEQjqZM62UkkCuptvw9W+Rqqjxpg60PQd2Tgd+pfsENBLSISUVCXinRzEGRJHfEdmXkiTJoKL/xP8PxYTd/RMVGNespssAlq+hYRiVFQl4q25uQ3e8PAyO/nfx08H2owGYTHGNQvGzh3yhzVqEVEYhTUpaB/xHcJBDUE5ezeEzzOVqOumRnM9929B2YtCEZ8RzQ7mYjIIArqUtC5HQ7vC0ZUl4L4F4rauqP3mw3UqjP73DPnARcRqXAK6lIQDSQrpRp1JBoglinannm72dQ6BbWISIyCuhQkfTGOTFH4VlXD5GnZj4lq2plfPtT0LSIyiIK6FLQ1BX292ZqRk2jWQpg4JQjdzHm+I1HfdWZQT60PJkvp7RnbMoqIlIiJxS5AyWn6KTz2bbj0O+N3zVIaSAbhyO9Tjn1M1PRdn3FcfL7v6fMGtnduh+9dBu/8VjDlaC7++z2w/dHcjo0sOQcu/rf8zhERGUMK6nw13QEtd8LhrqOXbxwL7sE91KvePfbXKqTzrj92rbjx/TDvZVCd0TQen0Y0HtTP/xq2PQLP/Qpmv2/46x/cA5t/ASe+KvdBeNsfg423w0X/DBOqcjtHRGSMKajzFS3L2NU+PkG97yU41FlaNWqAJWuPvT+1LPu85UNNIxq971F//XDaNwc/X/vnsOz83M559Nvw4z+F3c9D3Um5nSMiMsbUR52Pvj5IhwEwXiOT20psxPdoDbUwR1sY1NH7MZz+9y2PRUyi9zjXa4iIjAMFdT72vAA9B4PH4zUyuVQW4yiUQtWo0y3BgLZZOfZnw0C/enQtEZEEUFDnI/4HfLxq1Omm4N7iaUPcj1xups4BbPD7e6Qbdj8X3OrV2QrdncO/Tro5CN4JeXzEq6cHc5UrqEUkQRTU+Yj/AR/PGnWl1KYhGMQ1dc7g97fjafA+WHZB8Dzqfz6W9AjnRk8tV1CLSKIoqPPR1gzTjw8m8uhKj/31ohHflRTUEDR/x2vUUXP3youCn8P1IXd3Que2EQb1Mmh/Gvp68z9XRGQMKKjzEYVmbQq6Osb+evt2QPfeygvq2oygbmsCq4KTXwcTa4av8Y6mXz+1HHq6g5HfIiIJoKDOVV9f0OQ6d0UwQ9h4NH1HgVQqi3EUSuY0oulmmLMEJk2B+qU5BPUo3rdokRA1f4tIQiioc7X3RThyIGgazWyaHStRWFRajfqopu/mgdBNrRh+5He6Oah55zPiO1Kvkd8ikiwK6ly1xUJzvBaOSDfDlDlDr0BVrmpj8333HIJdzw58WUktg71b4dC+oc9PNweBO5LZxWpmwIyGgf/eIiJFpqDOVX/tdtn4rZncFvaJD7WwRbmaWg84HNwVDOzyvoGg7m+aPsbI7+h9G6nUMtWoRSQxFNS5SjfDtONgyuzgvuYjB+DwgbG7XjTiu9L6p2FgdrKu9qOb/6Of6SFGfnd3Bvda5zMjWaa5K4LxCBr5LSIJoKDOVTw0h5rmspD274TuPZXXPw2D3990C9gEqDs52DZ7UXB73FA13vang59zR7F2d2pZMPJ7zwsjfw0RkQJRUOcimuM7Cs2hprkspEodSAaD3990UzjiuybYNqEq6H8eakBZugBzo6ei5vUcpysVERlDCupc7N0KR7oG/vhHg7vGNKgrbI7vuEFN31lmZkstG3qwV7o5qHHPXjTy60dzfmtxDhFJAAV1LjJDs7Yu+DmWTd9tTUF/+LS5Y3eNpJoSzve9bzt0PHN0UM9dHtwud2j/0ee2jWLEd6RmJsyYrxq1iCSCgjoX6YwlE8el6bulMkd8A1RNDL6kvPg78N4sNerweXuWIE23FGYAXmrZ0APWRETGkYI6F+kWmDYvXNmJYJWlqsljV6N2D0KiEpu9I7X1sG1D8DgzeIfqQz60P6hpj2bEd/wa6c3B+AQRkSJSUOeirWnwH3+zsZ2drCsdTPhRyUE9tR56D4cjvpcO3jd7UfBFKbMPub2A/fqpZcHa4xr5LSJFpqAejnvYDJ1xu0/mwhGFFAVQJd5DHYkGlM1ePDDiO1I1MfvI7/6xBKO4NSsyVyO/RSQZcgpqMzvfzFrMbIuZXZtl/0Izu8/MnjCz9WbWENvXa2aPhf/uKGThx0X/iO+M5tSxnEa0kkd8R6KgHuo9yNaH3NYU1LRHM+I70j/nt/qpRaS4hg1qM6sCvg5cAKwELjWzlRmHfQm41d1XAdcDn4/tO+juq8N/bytQucdPFJqZE2iMZdN3uglqZgX94pUqGrA3VKtCagXseREOdw1sS7cEAVs1cfTXnzILpp+gGrWIFF0uf9FOB7a4+7MAZvZd4EJgU+yYlcBfhI/XAT8qYBnH16H90LFl4Pmz64OfmTW7oZq+o6bynu6hr1E1OXi9CUN8T6rkEd+RXGrUAC0/H5i1bOdGWHBG4cqQWgYvPQ7bHyvca5Yis+CL0cTJxS6JSEXKJajnA1tjz1uBzL+GjwMXA18B3g5MN7M6d+8AasxsA9AD3OjuPxp1qcfS7f8Lmn86eNuM+QMjviO19UGT+JGDwTrJkS33wbffMfx13vFNeMU7j97uHjThrnhr/mUvJ7MWBD+PW5V9/3EvD37+4IMZ2z9UuDIc9wp4cB3cdHbhXrNUnXUNvO7TxS6FSEUqQBshANcAXzOzK4D7gW1AtKLBQnffZmZLgF+a2ZPu/kz8ZDO7ErgSYMGCBQUq0ghtfwwWnw1nXDWwrX7p0cfF76WedWLs/EeDn+/+T5iQ7e11uO0DwXHZgrqrPVg1ajRzVZeDpW+Eq349dNP3nCXwgbvhQMfAtglVsOiswpXhD/8SFp4ZrN5Vye75NLz0WLFLIVKxcgnqbUAsiWgIt/Vz9+0ENWrMbBrwDnffE+7bFv581szWA2uAZzLOvwm4CaCxsdFH8HsURrTyUuP7Yfmbjn1sfOGIeFCnm4Pa4MpjdMfXLx16UYlKnuM7bsKEoEZ7LIVs5s6mZgYsO39sr1EKNt4OLzxY7FKIVKxcRn0/DCw1s8VmNhm4BBg0etvM6s0seq3rgJvD7bPNrDo6BjiTwX3bydIernGcS212qNnJ0s3D3x6UWnHsuapBQS3JkVoWfIHt7ix2SUQq0rBB7e49wNXAXUAT8H1332hm15tZVG1cC7SY2WZgHnBDuH0FsMHMHicYZHajuyc3qPMJydosQd3bEyyzONzMWMf6w5duhuqZMP243MosMtaiL67REqIiMq5y6qN29zuBOzO2fSb2+DbgtiznPQgM036ZIG1Nua+8lG1N6t3PQ++h4YO+f67qzdDQmFGGcN3rSh7xLckSfV7TTdBwWnHLIlKBNDNZXHQfbi4rL1XPgAmTBteooxr5cDOK9c96laX5O91cmLmqRQpl9qLgC+xQ4ypEZEwpqOPyCUmzo++ljmaxqh/mNaI/fJlzVXe1BzX0QkyBKVIoE6qCL7BDjasQkTGloI4c2hdMF5rP/NqZ04imW2DmAqieduzzoj98R81VHfWRq0YtCTN3uWZpEykSBXUkHY74zme0deY0om151MhTy45uSuxfjEM1akmY1LJgCdFD+4tdEpGKo6CO9Ndm8wjJeI26rzcYHJZrjXzu8qAGf2hfrAwtQd/39ONzL4PIeIj+v2hXrVpkvCmoI+kRrLwUr1HnOuI70j+SdnOsDM2a41uSKfq8qp9aZNwpqCMjWXmpth4O74cj3fnXyFNZRn5rxLck1exFwRdZjfwWGXcK6shIQjJ+L3V/UJ+S27n9f/jCfumuDuhKq39akqlqYjgAUkEtMt4U1BAMkNnzYv63RcWnEW1rhpknQvX03M7t/8MX9vlpxLckXbYBkCIy5hTUMDDH90hr1F3tI6uRp5YN9PlFNWvdQy1JlVoRfKHVyG+RcaWghtiMYnmGZG0q+NnVFoR9vgtppFYM3PKSboHJ02HGCfm9hsh4ib6Itm8+9nEiUlAKagiCumoyzF6c33lT64Kf2x6Bnu4RBHX0h68luIc6tUwjviW5jjX1rYiMGQU1BM3PdUvzG/ENUDMzmO/7+V8Hz/Otkff/4WsJ/uUzK5rIeJu9OPi8K6hFxpWCGkZ+W5RZUKuO/nDV5zjiOzJ7cVCTf+HBoPlca1BLklVNhPqlupdaZJwpqA93wZ4XRn5bVNRPPaMBambkd27VxKAm3/yz4LkGkknSpZarRi0yzhTUIx3xHamtG935qWVwcNfoXkNkvKSWB19sD3cVuyQiFUNBHd3HPNLabHQv9Uhr5NF5k6fBzIaRvYbIeInGUWjkt8i4UVC3NQUDZObkOeI7Et1LPZoadfRTI74l6TTnt8i4y3OYcxlKtwQDZKomjez8/qAeYY06Ok/901IK5iwJvtg+/t+wt7V45ZhQBWv+GKalilcGkXGioE43wQlrRn7+CacGo7dH2vQ9Z3EQ0iedM/IyiIyXqkmw+Cx45pfw3P3FLUtfL5z9l8Utg8g4qOygPnwAdr8Ar/yjkb/Gya+Djz828vOrJsFHfjvy80XG22U/DEKymP5pzcC0uyJlrrKDun0z4BptLZIPs/wnByq01Ar1k0vFqOzBZNGIby0tKVJa5i6Hjqeht6fYJREZcxUe1NGI7yXFLomI5CO1HHoPw+7nil0SkTFX4UHdAnUnj3zEt4gUR3SbmGZJkwpQ2UEdrVglIqUlmldf/dRSASo3qI8chN3Pq39apBRVT4NZC1SjlopQuUGtEd8ipU0LhEiFqNyg7p/jW0tLipSk1HJo18hvKX+VG9RtTTBhIsw5qdglEZGRSC2H3kNBF5ZIGavcoE63BCE9cXKxSyIiIzFXI7+lMlRwUDcN/I8uIqWnPhxfoqlEpcxVZlBHI77VPy1SuqqnwcwFA+NNRMpUZQZ1+9PgfQpqkVKXWqZ7qaXsVWZQa8S3SHmYuzy41bLYq3mJjKEKDeomsKpg+lARKV0a+S0VoEKDugXqNOJbpOSlwpkFNfJbylhlBnVbk5q9RcpBKprzWyO/pXxVXlAf6Q6WxlNQi5S+6ukwo0Ejv6WsVV5Qd4QjvnUPtUh5mLtc91JLWau8oNaIb5HyEs35rZHfUqZyCmozO9/MWsxsi5ldm2X/QjO7z8yeMLP1ZtaQsX+GmbWa2dcKVfARa9OIb5GykloOPd0a+S1la9igNrMq4OvABcBK4FIzW5lx2JeAW919FXA98PmM/X8L3D/64hZAuhnmLIGJ1cUuiYgUQtQ6pn5qKVMTczjmdGCLuz8LYGbfBS4ENsWOWQn8Rfh4HfCjaIeZnQbMA34BNI6+yHno64Ob3zB4285NcPK541oMERlD0Zryv/gk/Pr/FrcsUjle/zew6MxxuVQuQT0f2Bp73gqckXHM48DFwFeAtwPTzawO2A38A3AZ8PqhLmBmVwJXAixYsCDXsuemevrg5wteBadeUdhriEjx1MyA13wUdm4sdkmkkkzIJT4Lo1BXugb4mpldQdDEvQ3oBf4UuNPdW81syJPd/SbgJoDGxkYvUJlgwgT449sL9nIiklBv+D/FLoHImMklqLcBJ8aeN4Tb+rn7doIaNWY2DXiHu+8xs1cDZ5nZnwLTgMlmtt/djxqQJiIiIkfLJagfBpaa2WKCgL4E+KP4AWZWD+xy9z7gOuBmAHd/b+yYK4BGhbSIiEjuhh317e49wNXAXUAT8H1332hm15vZ28LD1gItZraZYODYDWNUXhERkYpi7oXrEi6ExsZG37BhQ7GLISIiMm7M7BF3z3pnVOXNTCYiIlJCFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLSIikmAKahERkQRTUIuIiCSYglpERCTBEjczmZmlgRcK/LL1QHuBX7MS6X0sDL2PhaH3sTD0PhbGaN/Hhe6eyrYjcUE9Fsxsw1BTs0nu9D4Wht7HwtD7WBh6HwtjLN9HNX2LiIgkmIJaREQkwSolqG8qdgHKhN7HwtD7WBh6HwtD72NhjNn7WBF91CIiIqWqUmrUIiIiJamsg9rMzjezFjPbYmbXFrs8pcLMTjSzdWa2ycw2mtnHw+1zzOweM3s6/Dm72GUtBWZWZWaPmtlPw+eLzex34efye2Y2udhlTDozm2Vmt5lZs5k1mdmr9XnMn5n9efj/9FNm9h0zq9HncXhmdrOZtZnZU7FtWT9/Fvhq+H4+YWanjvb6ZRvUZlYFfB24AFgJXGpmK4tbqpLRA/xvd18JvAr4SPjeXQvc5+5LgfvC5zK8jwNNsedfAL7s7icDu4EPFqVUpeUrwC/cfTnwSoL3U5/HPJjZfOBjQKO7vxyoAi5Bn8dc3AKcn7FtqM/fBcDS8N+VwL+M9uJlG9TA6cAWd3/W3Q8D3wUuLHKZSoK7v+Tuvw8f7yP4ozif4P37j/Cw/wAuKkoBS4iZNQBvBr4RPjfgXOC28BC9j8Mws5nAHwLfBHD3w+6+B30eR2IiMMXMJgJTgZfQ53FY7n4/sCtj81CfvwuBWz3wW2CWmR0/muuXc1DPB7bGnreG2yQPZrYIWAP8Dpjn7i+Fu3YA84pVrhLyj8AngL7weR2wx917wuf6XA5vMZAGvhV2IXzDzGrR5zEv7r4N+BLwIkFA7wUeQZ/HkRrq81fw7CnnoJZRMrNpwA+AP3P3zvg+D24X0C0Dx2BmbwHa3P2RYpelxE0ETgX+xd3XAF1kNHPr8zi8sA/1QoIvPicAtRzdnCsjMNafv3IO6m3AibHnDeE2yYGZTSII6W+7+w/DzTujJpzwZ1uxylcizgTeZmbPE3S9nEvQ1zorbHoEfS5z0Qq0uvvvwue3EQS3Po/5eT3wnLun3f0I8EOCz6g+jyMz1Oev4NlTzkH9MLA0HNE4mWDQxB1FLlNJCPtRvwk0ufv/je26A7g8fHw58OPxLlspcffr3L3B3RcRfP5+6e7vBdYB7wwP0/s4DHffAWw1s2XhptcBm9DnMV8vAq8ys6nh/+PR+6jP48gM9fm7A3hfOPr7VcDeWBP5iJT1hCdm9iaCPsIq4GZ3v6G4JSoNZvZa4AHgSQb6Vv+KoJ/6+8ACghXO3u3umQMsJAszWwtc4+5vMbMlBDXsOcCjwGXufqiIxUs8M1tNMCBvMvAs8H6CioY+j3kws78B3kNwZ8ejwIcI+k/1eTwGM/sOsJZghaydwGeBH5Hl8xd+CfoaQbfCAeD97r5hVNcv56AWEREpdeXc9C0iIlLyFNQiIiIJpqAWERFJMAW1iIhIgimoRUREEkxBLTJKZuZm9g+x59eY2ecK9Nq3mNk7hz9y1Nd5V7gq1bqM7SeY2W3h49XhLY+FuuYsM/vTbNcSkQEKapHROwRcbGb1xS5IXGy2qVx8EPiwu58T3+ju2909+qKwGsgrqIcpwyygP6gzriUiIQW1yOj1ADcBf565I7NGbGb7w59rzexXZvZjM3vWzG40s/ea2UNm9qSZnRR7mdeb2QYz2xzOHx6tcf1FM3s4XPP2f8Ve9wEzu4Ng1qnM8lwavv5TZvaFcNtngNcC3zSzL2Ycvyg8djJwPfAeM3vMzN5jZrXhOr0PhYtlXBiec4WZ3WFmvwTuM7NpZnafmf0+vHa0it2NwEnh630xulb4GjVm9q3w+EfN7JzYa//QzH5hwTrAfx97P24Jy/qkmR3130KkVOXzjVtEhvZ14IkoOHL0SmAFwfJ5zwLfcPfTzezjwEeBPwuPW0SwbOtJwDozOxl4H8HUhH9gZtXA/5jZ3eHxpwIvd/fn4hczsxMI1h4+jWDd4bvN7CJ3v97MziWYOS3rDErufjgM9EZ3vzp8vb8jmBb1A2Y2C3jIzO6NlWFVOFPTRODt7t4Ztjr8NvwicW1YztXh6y2KXfIjwWX9FWa2PCzrKeG+1QQruh0CWszsn4C5wPxwnWXC8oiUBdWoRQogXF3sVuBjeZz2cLj29yHgGSAK2icJwjnyfXfvc/enCQJ9OfAGgvmEHyOY2rWOYKF6gIcyQzr0B8D6cFGGHuDbBOs8j9QbgGvDMqwHagimUwS4JzadpwF/Z2ZPAPcSTFk53JKUrwX+C8DdmwmmaIyC+j533+vu3QStBgsJ3pclZvZPZnY+0JnlNUVKkmrUIoXzj8DvgW/FtvUQfiE2swkEc1VH4vMp98We9zH4/83MeX6dIPw+6u53xXeEc4p3jaTwI2DAO9y9JaMMZ2SU4b1ACjjN3Y9YsJpYzSiuG3/feoGJ7r7bzF4JvBG4Cng38IFRXEMkMVSjFimQsAb5fYKBWZHnCZqaAd4GTBrBS7/LzCaE/dZLgBbgLuBPLFiOFDM7xcxqh3mdh4CzzazezKqAS4Ff5VGOfcD02PO7gI+GixBgZmuGOG8mwbrcR8K+5oVDvF7cAwQBT9jkvYDg984qbFKf4O4/AP6aoOldpCwoqEUK6x8IVtiJ/DtBOD4OvJqR1XZfJAjZnwNXhU2+3yBo9v19OADr3ximhSxcau9agmUNHwcecfd8ljRcB6yMBpMBf0vwxeMJM9sYPs/m20CjmT1J0LfeHJang6Bv/anMQWzAPwMTwnO+B1wxzIpO84H1YTP8fwHX5fF7iSSaVs8SERFJMNWoRUREEkxBLSIikmAKahERkQRTUIuIiCSYglpERCTBFNQiIiIJpqAWERFJMAW1iIhIgv1/edoDBgq64KoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot and display the performance evaluation\n",
    "xgb_eval = xgb_model.evals_result()\n",
    "eval_steps = range(len(xgb_eval['validation_0']['error']))\n",
    "\n",
    "fig, ax = pyplot.subplots(1, 1, sharex=True, figsize=(8, 6))\n",
    "\n",
    "ax.plot(eval_steps, [1-x for x in xgb_eval['validation_0']['error']], label='Train')\n",
    "ax.plot(eval_steps, [1-x for x in xgb_eval['validation_1']['error']], label='Test')\n",
    "ax.legend()\n",
    "ax.set_title('Accuracy')\n",
    "ax.set_xlabel('Number of iterations');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that there is model overfitting, and there is a decrease in model accuracy after about 60 iterations \n",
    "\n",
    "Select the trained model obtained after 30 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select trained model.\n",
    "n_trees = 30\n",
    "y_pred = xgb_model.predict(X_test, ntree_limit= n_trees)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 95.0%\n"
     ]
    }
   ],
   "source": [
    "# Check the accuracy of the trained model.\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You will use the accuracy value obtained on the test data to compare the accuracy of the model with default parameters to the accuracy of the model with tuned parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2. Use grid search and cross-validation to tune the model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use grid search and cross-validation to tune your model to achieve better accuracy.\n",
    "\n",
    "XGBoost has an extensive catalog of hyperparameters which provides great flexibility to shape an algorithm’s desired behavior. Here you will the optimize the model tuning which adds an L1 penalty (`reg_alpha`).\n",
    "\n",
    "Use a 5-fold cross-validation because your training data set is small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, create the XGBoost pipeline and set up the parameter grid for the search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create XGBoost pipeline, set up parameter grid.\n",
    "xgb_model_gs = XGBClassifier(eval_metric=['error'], use_label_encoder=False, n_jobs=1)\n",
    "parameters = {'reg_alpha': [0.0, 1.0], 'reg_lambda': [0.0, 1.0], 'n_estimators': [n_trees], 'seed': [1337]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use ``GridSearchCV`` to search for the best parameters over the parameters values that were specified in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  16 out of  20 | elapsed:    3.3s remaining:    0.8s\n",
      "[Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    3.4s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=XGBClassifier(base_score=None, booster=None,\n",
       "                                     colsample_bylevel=None,\n",
       "                                     colsample_bynode=None,\n",
       "                                     colsample_bytree=None,\n",
       "                                     eval_metric=['error'], gamma=None,\n",
       "                                     gpu_id=None, importance_type='gain',\n",
       "                                     interaction_constraints=None,\n",
       "                                     learning_rate=None, max_delta_step=None,\n",
       "                                     max_depth=None, min_child_weight=None,\n",
       "                                     missing=nan, monotone_constraints=None,\n",
       "                                     n_estimators=100, n_jobs=None,\n",
       "                                     num_parallel_tree=None, random_state=None,\n",
       "                                     reg_alpha=None, reg_lambda=None,\n",
       "                                     scale_pos_weight=None, subsample=None,\n",
       "                                     tree_method=None, use_label_encoder=False,\n",
       "                                     validate_parameters=None, verbosity=None),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'n_estimators': [30], 'reg_alpha': [0.0, 1.0],\n",
       "                         'reg_lambda': [0.0, 1.0], 'seed': [1337]},\n",
       "             scoring='accuracy', verbose=-1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Search for the best parameters.\n",
    "clf = GridSearchCV(xgb_model_gs, parameters, scoring='accuracy', cv=5, verbose=-1, n_jobs=1, refit=True)\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the grid scores, you can see the performance result of all parameter combinations including the best parameter combination based on model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the accuracy estimated using cross-validation and the hyperparameter values for the best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 95.9%\n",
      "Best parameter set: {'n_estimators': 30, 'reg_alpha': 1.0, 'reg_lambda': 1.0, 'seed': 1337}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best score: %.1f%%\" % (clf.best_score_*100))\n",
    "print(\"Best parameter set: %s\" % (clf.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the accuracy of best parameter combination on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 96.5%\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf.best_estimator_.predict(X_test, ntree_limit= n_trees)\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: %.1f%%\" % (accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy on test set is about the same for tuned model as it is for the trained model that has default hyperparameters values, even though the selected hyperparameters are different to the default parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.3. Model with pipeline data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you learn how to use the XGBoost model within the scikit-learn pipeline. \n",
    "\n",
    "Let's start by importing the required objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=10)\n",
    "xgb_model_pca = XGBClassifier(n_estimators=n_trees, verbosity=1, eval_metric=['error'], use_label_encoder=False, n_jobs=1)\n",
    "pipeline = Pipeline(steps=[('pca', pca), ('xgb', xgb_model_pca)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('pca', PCA(n_components=10)),\n",
       "                ('xgb',\n",
       "                 XGBClassifier(base_score=0.5, booster='gbtree',\n",
       "                               colsample_bylevel=1, colsample_bynode=1,\n",
       "                               colsample_bytree=1, eval_metric=['error'],\n",
       "                               gamma=0, gpu_id=-1, importance_type='gain',\n",
       "                               interaction_constraints='',\n",
       "                               learning_rate=0.300000012, max_delta_step=0,\n",
       "                               max_depth=6, min_child_weight=1, missing=nan,\n",
       "                               monotone_constraints='()', n_estimators=30,\n",
       "                               n_jobs=12, num_parallel_tree=1, random_state=0,\n",
       "                               reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "                               subsample=1, tree_method='exact',\n",
       "                               use_label_encoder=False, validate_parameters=1,\n",
       "                               verbosity=1))])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you are ready to evaluate accuracy of the model trained on the reduced set of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 94.5%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('pca', PCA(n_components=10)),\n",
       "                ('xgb',\n",
       "                 XGBClassifier(base_score=0.5, booster='gbtree',\n",
       "                               colsample_bylevel=1, colsample_bynode=1,\n",
       "                               colsample_bytree=1, eval_metric=['error'],\n",
       "                               gamma=0, gpu_id=-1, importance_type='gain',\n",
       "                               interaction_constraints='',\n",
       "                               learning_rate=0.300000012, max_delta_step=0,\n",
       "                               max_depth=6, min_child_weight=1, missing=nan,\n",
       "                               monotone_constraints='()', n_estimators=30,\n",
       "                               n_jobs=12, num_parallel_tree=1, random_state=0,\n",
       "                               reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "                               subsample=1, tree_method='exact',\n",
       "                               use_label_encoder=False, validate_parameters=1,\n",
       "                               verbosity=1))])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = pipeline.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: %.1f%%\" % (accuracy * 100.0))\n",
    "pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that this model has a similar accuracy to the model trained using default hyperparameter values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how you can save your XGBoost pipeline using the WML service instance and deploy it for online scoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"upload\"></a>\n",
    "## 4. Persist model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you learn how to use the Python client libraries to store your XGBoost model in the WML repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the XGBoost model to the WML Repository\n",
    "\n",
    "Save the model artifact as *XGBoost model for breast cancer* to your WML instance.\n",
    "\n",
    "Get software specification for XGBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ab9e1b80-f2ce-592c-a7d2-4f2344f77194'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "software_spec_uid = client.software_specifications.get_uid_by_name('runtime-22.1-py3.9')\n",
    "software_spec_uid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "    client.repository.ModelMetaNames.NAME: \"XGBoost model for breast cancer\",\n",
    "    client.repository.ModelMetaNames.TYPE: \"scikit-learn_1.0\",\n",
    "    client.repository.ModelMetaNames.SOFTWARE_SPEC_UID : software_spec_uid,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_details = client.repository.store_model(pipeline, metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the saved model metadata from WML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"deploy\"></a>\n",
    "## 5. Deployment\n",
    "In this section you will learn how to create batch deployment to create job using the Watson Machine Learning Client.\n",
    "\n",
    "You can use commands bellow to create batch deployment for stored model (web service).\n",
    "\n",
    "### 5.1: Create model deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need the model uid to create the deployment. You can extract the model uid from the saved model details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "572df224-e590-4c21-a474-bbb0c4cd3b77\n"
     ]
    }
   ],
   "source": [
    "# Extract the uid.\n",
    "model_uid = client.repository.get_model_uid(model_details)\n",
    "print(model_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use this modul_uid in the next section to create the deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can create a deployment, *Predict breast cancer*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: '572df224-e590-4c21-a474-bbb0c4cd3b77' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
      "\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='4e8e9303-b9f3-4ff6-b183-ec273c16b032'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create the deployment.\n",
    "meta_props = {\n",
    "    client.deployments.ConfigurationMetaNames.NAME: \"Predict breast cancer'\",\n",
    "    client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "}\n",
    "\n",
    "deployment_details = client.deployments.create(model_uid,meta_props)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get a list of all deployments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List the deployments.\n",
    "client.deployments.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *Predict breast cancer model* has been successfully deployed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Get deployment details\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show deployments details, you need get deployment_uid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_uid = client.deployments.get_uid(deployment_details)\n",
    "client.deployments.get_details(deployment_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id=\"score\"></a>\n",
    "## 6. Score the model\n",
    "Let's see if our deployment works.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, extract the url endpoint, *scoring_url*, which will be used to send scoring requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_id = client.deployments.get_id(deployment_details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the scoring payload with the values to score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_data': [{'values': [[12.23, 19.56, 78.54, 461.0, 0.09586, 0.08087, 0.04187, 0.04107, 0.1979, 0.06013, 0.3534, 1.326, 2.308, 27.24, 0.007514, 0.01779, 0.01401, 0.0114, 0.01503, 0.003338, 14.44, 28.36, 92.15, 638.4, 0.1429, 0.2042, 0.1377, 0.108, 0.2668, 0.08174]]}]}\n"
     ]
    }
   ],
   "source": [
    "# Prepare scoring payload.\n",
    "payload_scoring = {client.deployments.ScoringMetaNames.INPUT_DATA:\n",
    "    [\n",
    "        {\n",
    "        'values': [X_test[0].tolist()]\n",
    "        }\n",
    "   ]\n",
    "}\n",
    "print(payload_scoring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'predictions': [{'fields': ['prediction', 'probability'], 'values': [[0, [0.9749674201011658, 0.02503260038793087]]]}]}\n"
     ]
    }
   ],
   "source": [
    "# Perform prediction and display the result.\n",
    "response_scoring = client.deployments.score(deployment_id, payload_scoring)\n",
    "print(response_scoring)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Result**: The patient record is classified as a benign tumor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "## 7. Clean up "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to clean up all created assets:\n",
    "- experiments\n",
    "- trainings\n",
    "- pipelines\n",
    "- model definitions\n",
    "- models\n",
    "- functions\n",
    "- deployments\n",
    "\n",
    "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"summary\"></a>\n",
    "## 8. Summary and next steps\n",
    "\n",
    "You successfully completed this notebook! You learned how to use Keras machine learning library as well as Watson Machine Learning for model creation and deployment. \n",
    "\n",
    "Check out our [Online Documentation](https://dataplatform.cloud.ibm.com/docs/content/analyze-data/wml-setup.html) for more samples, tutorials, documentation, how-tos, and blog posts. \n",
    "\n",
    "### Authors\n",
    "\n",
    "**Wojciech Jargielo**, Software Engineer\n",
    "\n",
    "Copyright © 2020, 2021, 2022 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}