notebooks/tutorial3-learn-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Learn machine learning models\n",
    "\n",
    "This tutorial demonstrates how to learn machine learning models to value on-the-ball actions of football players with the open-source [VAEP framework](https://github.com/ML-KULeuven/socceraction) using the publicly available [Wyscout match event dataset](https://figshare.com/collections/Soccer_match_event_dataset/4415000). The Wyscout dataset includes data for the 2017/2018 English Premier League, the 2017/2018 Spanish Primera División, the 2017/2018 German 1. Bundesliga, the 2017/2018 Italian Serie A, the 2017/2018 French Ligue 1, the 2018 FIFA World Cup, and the UEFA Euro 2016. Covering 1,941 matches, 3,251,294 events and 4,299 players, the dataset is large enough to train machine-learning models and obtain robust ratings for the players.\n",
    "\n",
    "This tutorial demonstrates the following four steps:\n",
    "1. Split the dataset into a training set and a test set.\n",
    "2. Construct the baseline classifiers by using conservative hyperparameters for the learning algorithm.\n",
    "3. Optimize the classifiers by tuning the hyperparameters for the learning algorithm.\n",
    "4. Construct the final classifiers using the optimal hyperparameters for the learning algorithm.\n",
    "\n",
    "This notebook is compatible with `socceraction` version `0.2.0`.\n",
    "\n",
    "**Conventions:**\n",
    "* Variables that refer a `DataFrame` object are prefixed with `df_`.\n",
    "* Variables that refer a collection of `DataFrame` objects (e.g., a list, a set or a dict) are prefixed with `dfs_`.\n",
    "\n",
    "**References:**\n",
    "* Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. \"[Actions Speak Louder than Goals: Valuing Player Actions in Soccer.](https://arxiv.org/abs/1802.07127)\" In *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp. 1851-1861. 2019.\n",
    "* Luca Pappalardo, Paolo Cintia, Alessio Rossi, Emanuele Massucco, Paolo Ferragina, Dino Pedreschi, and Fosca Giannotti. \"[A Public Data Set of Spatio-Temporal Match Events in Soccer Competitions.](https://www.nature.com/articles/s41597-019-0247-7)\" *Scientific Data 6*, no. 1 (2019): 1-15."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If you run this notebook on Google Colab, then uncomment the code in the following cell and execute the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install tables==3.6.1\n",
    "# !pip install scikit-plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If you run this notebook on Google Colab and wish to store all data in a Google Drive folder, then uncomment the code in the following cell and execute the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from google.colab import drive\n",
    "# drive.mount('/content/gdrive')\n",
    "# %mkdir -p '/content/gdrive/My Drive/Friends of Tracking/'\n",
    "# %cd '/content/gdrive/My Drive/Friends of Tracking/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scikitplot.metrics import plot_calibration_curve\n",
    "from sklearn.calibration import CalibratedClassifierCV\n",
    "from sklearn.metrics import brier_score_loss, roc_auc_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tqdm.notebook import tqdm\n",
    "from xgboost import XGBClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore', category=pd.io.pytables.PerformanceWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This third tutorial assumes that the `spadl.h5` HDF5 file as well as the `features.h5` and `labels.h5` files have been created for a set of games in the first or second tutorial.\n",
    "\n",
    "This third tutorial only uses features that have been generated in the first tutorial. However, you are strongly encouraged to toy around with the additional features from the second tutorial and to try out your own features to improve the accuracy of the predictive machine learning models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load games"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games = pd.read_hdf('spadl.h5', key='games')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If you plan to produce results for a particular season in the Wyscout match event dataset (e.g., 2017/2018 English Premier League), you should consider leaving that season out of the dataset to avoid the same game states from appearing in both the training set and the test set. If you would like to only include a particular subset of games, then uncomment the code in the following cell, adapt the selector and execute the cell. The example selector will select all games that were played in the 2017/2018 English Premier League."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_games = df_games[\n",
    "#     df_games['competition_id'] == 364\n",
    "# ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the *features* for the selected games and combine them into the `df_features` `DataFrame` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_features = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_features = pd.read_hdf('features.h5', key=f'game_{game_id}')\n",
    "    df_features['game_id'] = game_id\n",
    "    dfs_features.append(df_features)\n",
    "df_features = pd.concat(dfs_features).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the *labels* for the selected games and combine them into the `df_labels` `DataFrame` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_labels = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']  \n",
    "    df_labels = pd.read_hdf('labels.h5', key=f'game_{game_id}')\n",
    "    df_labels['game_id'] = game_id\n",
    "    dfs_labels.append(df_labels)\n",
    "df_labels = pd.concat(dfs_labels).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To accurately predict the labels for game states, the challenge is to train a robust machine learning model that generalizes well from *observed* game states (e.g., game states in games that have been played already) to *unobserved* game states (e.g., game states in games that are yet to be played). Hence, the machine learning model needs to capture interactions between features of game states that are not too specific to identify *similar* game states yet specific enough to distinguish between *dissimilar* game states.\n",
    "\n",
    "Typically, a machine learning model is said to be *overfitted* on the data if the interactions are too specific (i.e., apply to just a few game states) and is said to be *underfitted* on the data if the interactions are too general (i.e., apply to about all game states). To avoid underfitting, we define an expressive set of features and use a machine learning algorithm that learns a model that can capture complex relationships between these features. To avoid overfitting, we apply [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html) to assess how well a candidate model would perform in practice and apply regularization techniques to keep the learned model as simple as possible.\n",
    "\n",
    "In this tutorial, we will apply the following methodology:\n",
    "1. Split the available data into a training set and a test set. The training set will be used to learn the machine learning model, whereas the test set will be kept aside until the very end to assess the real-world performance of the learned model.\n",
    "2. Learn models with different values for the hyperparameters of the learning algorithm on the training set by adopting a $k$-fold cross-validation setup:\n",
    "  * Split the training set in $k$ random folds or subsets of equal size.\n",
    "  * Use $k$-1 training folds to train the model and compute the evaluation metric on the remaining validation fold.\n",
    "  * Repeat $k$ times until each fold has served as the validation fold once.\n",
    "3. Select the values for the hyperparameters that yield the best performance across the $k$ validation folds according to the evaluation metric.\n",
    "4. Train the final model on the full training set using the optimal values for the hyperparameters.\n",
    "5. Apply the final model on the test set and compute the evaluation metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell splits the available data into a training set and test set using the `train_test_split` function.\n",
    "\n",
    "* **Test set size:** The `test_size` parameter controls the number of examples in the test set. The challenge is to find an appropriate balance between the number of training examples and the number of test examples. Typically, more training examples yield better models, whereas more test examples yield more reliable evaluation metrics.\n",
    "* **Random state:** The `random_state` parameter sets the *seed* for the random number generator. By setting a *seed*, the train-test split will be the same for each execution of the `train_test_split` function, which is important for reproducing the results and comparing different models.\n",
    "* **Stratification:** The `stratify` parameter enforces a *stratified* train-test split according to the provided class label. By doing so, the distribution of the provided class label will be the same in the training set and the test set, which can be helpful to obtain a well-calibrated model. Since we will be using two different class labels (i.e., `scores` and `concedes`), we use a concatenation of both class labels for the stratification procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_X_train, df_X_test, df_y_train, df_y_test = train_test_split(\n",
    "    df_features,\n",
    "    df_labels,\n",
    "    test_size=0.10,\n",
    "    random_state=42,\n",
    "    stratify=df_labels['scores'].astype(str) + '_' + df_labels['concedes'].astype(str)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells inspect whether the training set and test set have the same proportion of positive and negative examples for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_y_train['scores'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_y_test['scores'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_y_train['concedes'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_y_test['concedes'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** In a real-world scenario where more data is available, you should consider respecting the chronological order of the games to construct the training set, validation set and test set. For instance, use the data for the 2016/2017 and 2017/2018 seasons to train the models, use the data for the 2018/2019 season to tune the models, and use the data for the 2019/2020 season to obtain the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Construct baseline classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell provides a list of features that the machine learning algorithm will consider to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = [\n",
    "    'start_distance_to_goal-0',\n",
    "    'end_distance_to_goal-0',\n",
    "    'start_distance_to_goal-1',\n",
    "    'end_distance_to_goal-1',\n",
    "    'start_distance_to_goal-2',\n",
    "    'end_distance_to_goal-2',\n",
    "    'start_angle_to_goal-0',\n",
    "    'end_angle_to_goal-0',\n",
    "    'start_angle_to_goal-1',\n",
    "    'end_angle_to_goal-1',\n",
    "    'start_angle_to_goal-2',\n",
    "    'end_angle_to_goal-2',\n",
    "    'team-1',\n",
    "    'team-2'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell provides a list of class labels for which the machine learning algorithm will train a model.\n",
    "\n",
    "**Note:** The `concedes` class label has been commented to speed up the execution of the entire notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [\n",
    "    'scores',\n",
    "#     'concedes'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell trains an XGBoost classifier for each label using *conservative* hyperparamters for the learning algorithm, which will serve as *baseline* models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "models = {}\n",
    "for label in tqdm(labels):\n",
    "    model = XGBClassifier(\n",
    "        eval_metric='logloss',\n",
    "        use_label_encoder=False,\n",
    "        n_estimators=10,\n",
    "        max_depth=3\n",
    "    )\n",
    "    model.fit(\n",
    "        X=df_X_train[features],\n",
    "        y=df_y_train[label]\n",
    "    )\n",
    "    models[label] = model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell estimates the probabilities for each label using the trained *baseline* models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_predictions = {}\n",
    "for label in tqdm(labels):\n",
    "    model = models[label]\n",
    "    probabilities = model.predict_proba(\n",
    "        df_X_test[features]\n",
    "    )\n",
    "    predictions = probabilities[:, 1]\n",
    "    dfs_predictions[label] = pd.Series(predictions, index=df_X_test.index)\n",
    "df_predictions = pd.concat(dfs_predictions, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute base rate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the *base rate* or *prior probability* of each class label in the training set. We use the *base rate* as a naive estimate for each example in the test set being true to establish a baseline for the evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_base_rates = pd.DataFrame({\n",
    "    label:np.full(len(df_y_test[label]), df_y_train[label].mean()) for label in labels\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_base_rates.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute Brier score loss for goal scored model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the base rate predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brier_score_loss(\n",
    "    y_true=df_y_test['scores'],\n",
    "    y_prob=df_base_rates['scores']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the predictions by the learned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brier_score_loss(\n",
    "    y_true=df_y_test['scores'],\n",
    "    y_prob=df_predictions['scores']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute Brier score loss for goal conceded model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the base rate predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# brier_score_loss(\n",
    "#     y_true=df_y_test['concedes'],\n",
    "#     y_prob=df_base_rates['concedes']\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the predictions by the learned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# brier_score_loss(\n",
    "#     y_true=df_y_test['concedes'],\n",
    "#     y_prob=df_predictions['concedes']\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot calibration curve and probability histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell creates a plot to show both a calibration curve and a probability histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(\n",
    "    nrows=2,\n",
    "    ncols=1,\n",
    "    figsize=(8, 8),\n",
    "    gridspec_kw={\n",
    "        'height_ratios': [3, 1]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell plots the *calibration curve* for the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_calibration_curve(\n",
    "    y_true=df_y_test['scores'],\n",
    "    probas_list=[df_predictions['scores'].tolist()],\n",
    "    clf_names=['Goal scored model'],\n",
    "    n_bins=10,\n",
    "    ax=ax1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell plots the histogram of the predictions produced by the trained model. Clearly, the majority of the examples is *negative*, which means that the start of the calibration curve is extremely important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions['scores'].plot.hist(\n",
    "    range=(0, 1),\n",
    "    bins=10,\n",
    "    ax=ax2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimize classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train *accurate* models, we perform a *grid search* over different combinations of parameter values for the most important hyperparameters for the learning algorithm. We will focus on the number of estimators and the maximum depth of the decision trees although more hyperparameters influence the performance of the trained models. The more combinations of parameter values need to be explored, the longer the *grid search* will take.\n",
    "\n",
    "Furthermore, restricting the number of estimators (i.e., the number of decision trees) and the maximum depth of the decision trees is an important mechanism to reduce the complexity of the trained models and thus also to avoid overfitting on the training data. The more decision trees that the learning algorithm can use and the deeper these decision trees can become, the more opportunities the learning algorithm has to *memorize* the training data rather than to discover patterns that generalize to the unseen test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell trains an XGBoost classifier for each label by trying different combinations of hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "models_cv = {}\n",
    "for label in tqdm(labels):\n",
    "    model = GridSearchCV(\n",
    "        estimator=XGBClassifier(\n",
    "            eval_metric='logloss',\n",
    "            use_label_encoder=False,\n",
    "        ),\n",
    "        param_grid={\n",
    "            'n_estimators': [50, 100],\n",
    "            'max_depth': [3, 4]\n",
    "        },\n",
    "        scoring='neg_brier_score',\n",
    "        refit=True,  # train final model on full training set using best hyperparameters\n",
    "        verbose=10,\n",
    "        n_jobs=1\n",
    "    )\n",
    "    model.fit(\n",
    "        X=df_X_train[features],\n",
    "        y=df_y_train[label]\n",
    "    )\n",
    "    models_cv[label] = model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** We have considered a manually selected set of features to represent the game states. In addition to optimizing the hyperparameters for the learning algorithm, we could also optimize the set of features to be considered by the learning algorithm. However, the XGBoost algorithm should be able to figure out by itself which features are most important to include in the model by the nature of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell estimates the probabilities for each label using the trained *baseline* models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_predictions_cv = {}\n",
    "for label in tqdm(labels):\n",
    "    model = models_cv[label]\n",
    "    probabilities = model.predict_proba(\n",
    "        df_X_test[features]\n",
    "    )\n",
    "    predictions = probabilities[:, 1]\n",
    "    dfs_predictions_cv[label] = pd.Series(predictions, index=df_X_test.index)\n",
    "df_predictions_cv = pd.concat(dfs_predictions, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute Brier score loss for goal scored model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the base rate predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brier_score_loss(\n",
    "    y_true=df_y_test['scores'],\n",
    "    y_prob=df_base_rates['scores']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the predictions by the learned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brier_score_loss(\n",
    "    y_true=df_y_test['scores'],\n",
    "    y_prob=df_predictions_cv['scores']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute Brier score loss for goal conceded model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the base rate predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# brier_score_loss(\n",
    "#     y_true=df_y_test['concedes'],\n",
    "#     y_prob=df_base_rates['concedes']\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell computes the [Brier loss score](https://en.wikipedia.org/wiki/Brier_score) for the predictions by the learned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# brier_score_loss(\n",
    "#     y_true=df_y_test['concedes'],\n",
    "#     y_prob=df_predictions_cv['concedes']\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot calibration curve and probability histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell creates a plot to show both a calibration curve and a probability histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig_cv, (ax1_cv, ax2_cv) = plt.subplots(\n",
    "    nrows=2,\n",
    "    ncols=1,\n",
    "    figsize=(8, 8),\n",
    "    gridspec_kw={\n",
    "        'height_ratios': [3, 1]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell plots the *calibration curve* for the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_calibration_curve(\n",
    "    y_true=df_y_test['scores'],\n",
    "    probas_list=[df_predictions_cv['scores'].tolist()],\n",
    "    clf_names=['Goal scored model'],\n",
    "    n_bins=10,\n",
    "    ax=ax1_cv\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell plots the histogram of the predictions produced by the trained model. Clearly, the majority of the examples is *negative*, which means that the start of the calibration curve is extremely important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions_cv['scores'].plot.hist(\n",
    "    range=(0, 1),\n",
    "    bins=10,\n",
    "    ax=ax2_cv\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Calibrate probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the trained model produces poorly calibrated probability estimates, the probability estimates can be re-calibrated using [CalibratedClassifierCV](https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html), which performs probability calibration with isotonic regression or logistic regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Construct final classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have found the *best* feature set and *best* hyperparameters for the learning algorithm, we can learn the final model.\n",
    "\n",
    "1. If we use `GridSearchCV` and the `refit` parameter was set to `True`, we can retrieve the *final* model, which has been re-trained on the entire training set, by accessing the `best_estimator_` attribute of the object.\n",
    "2. We can manually train the *final* model by creating a `XGBClassifier` object using the *best* hyperparameters and calling the `fit` method with the entire training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 1: Retrieve classifier from grid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_scores = models_cv['scores']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell shows the *best* hyperparameter combination that was found using the *grid search*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(\n",
    "    model_scores.best_params_\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell shows the full results of the *grid search*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(\n",
    "    model_scores.cv_results_\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell shows the hyperparameters that were used to re-train the model on the entire training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(\n",
    "    model_scores.best_estimator_.get_params()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell retrieves the final `XGBClassifier` object from the `GridSearchCV` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_scores_final = model_scores.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_scores_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 2: Train classifier using optimal hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell constructs a new `XGBoostClassifier` object using the *best* hyperparameters that were found by the *grid search*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_scores_final = XGBClassifier(\n",
    "    eval_metric='logloss',\n",
    "    use_label_encoder=False,\n",
    "    n_estimators=100,\n",
    "    max_depth=4\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell fits the `XGBoostClassifier` object on the entire training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_scores_final.fit(\n",
    "    X=df_X_train[features],\n",
    "    y=df_y_train[label]\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "498.594px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}