notebooks/tutorial2-generate-features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Generate features\n",
    "This tutorial demonstrates how to generate features to represent game states to value on-the-ball actions of football players with the open-source [VAEP framework](https://github.com/ML-KULeuven/socceraction) using the publicly available [Wyscout match event dataset](https://figshare.com/collections/Soccer_match_event_dataset/4415000). The Wyscout dataset includes data for the 2017/2018 English Premier League, the 2017/2018 Spanish Primera División, the 2017/2018 German 1. Bundesliga, the 2017/2018 Italian Serie A, the 2017/2018 French Ligue 1, the 2018 FIFA World Cup, and the UEFA Euro 2016. Covering 1,941 matches, 3,251,294 events and 4,299 players, the dataset is large enough to train machine-learning models and obtain robust ratings for the players.\n",
    "\n",
    "This tutorial demonstrates the following three steps:\n",
    "   1. Investigate the data in SPADL representation.\n",
    "   2. Construct features to represent actions.\n",
    "   3. Construct features to represent game states.\n",
    "\n",
    "This notebook is compatible with `socceraction` version `0.2.0`.\n",
    "\n",
    "**Conventions:**\n",
    "   * Variables that refer a `DataFrame` object are prefixed with `df_`.\n",
    "   * Variables that refer a collection of `DataFrame` objects (e.g., a list, a set or a dict) are prefixed with `dfs_`.\n",
    "   \n",
    "**References:**\n",
    "   * Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. \\\"[Actions Speak Louder than Goals: Valuing Player Actions in Soccer.](https://arxiv.org/abs/1802.07127)\\\" In *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp. 1851-1861. 2019.\n",
    "   * Luca Pappalardo, Paolo Cintia, Alessio Rossi, Emanuele Massucco, Paolo Ferragina, Dino Pedreschi, and Fosca Giannotti. \\\"[A Public Data Set of Spatio-Temporal Match Events in Soccer Competitions.](https://www.nature.com/articles/s41597-019-0247-7)\\\" *Scientific Data 6*, no. 1 (2019): 1-15.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If you run this notebook on Google Colab, then uncomment the code in the following cell and execute the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install tables==3.6.1\n",
    "# !pip install matplotsoccer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional:** If you run this notebook on Google Colab and wish to store all data in a Google Drive folder, then uncomment the code in the following cell and execute the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from google.colab import drive\n",
    "# drive.mount('/content/gdrive')\n",
    "# %mkdir -p '/content/gdrive/My Drive/Friends of Tracking/'\n",
    "# %cd '/content/gdrive/My Drive/Friends of Tracking/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "import matplotsoccer\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from ipywidgets import fixed, interact_manual, widgets\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore', category=pd.io.pytables.PerformanceWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we load the SPADL dataset that was created in the tutorial1-run-pipeline.ipynb notebook. We first load the teams, players and games dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_teams = pd.read_hdf('spadl.h5', key='teams')\n",
    "df_players = pd.read_hdf('spadl.h5', key='players')\n",
    "df_games = pd.read_hdf('spadl.h5', key='games')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add the team names to the df_games dataframe such that we know which two teams played in which match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "team_name_mapping = df_teams.set_index('team_id')['team_name'].to_dict()\n",
    "df_games['home_team_name'] = df_games['home_team_id'].map(team_name_mapping)\n",
    "df_games['away_team_name'] = df_games['away_team_id'].map(team_name_mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore SPADL actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this secion we will explore the SPADL actions for a selected match. In this notebook we chose the World Cup 2018 match between Belgium and Japan. The notebook can also be run for a match of your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games[(df_games['home_team_name'] == 'Belgium') & \n",
    "         (df_games['away_team_name'] == 'Japan')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** This notebook expects the data for the World Cup 2018 to be in the SPADL dataset. Make sure that the `game_id` selected in the cell below belongs to a game in your dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a game of choice\n",
    "game_id = 2058007"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load SPADL actions for selected game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each action, the dataset contains the following information for the action:\n",
    "\n",
    "* game_id: a unique identifier of the game;\n",
    "* team_id: a unique identifier of the team who performed the action;\n",
    "* player_id: a unique identifier of the player who performed the action;\n",
    "* period: 1 for the first half and 2 for the second half;\n",
    "* seconds: the time elapsed in seconds since the start of the half;\n",
    "* type_id: the identifier for the type of action;\n",
    "* type_name: the name for the type of action;\n",
    "* body_part_id: 0 for foot, 1 for head, 2 for other body part;\n",
    "* result: the result of the action: 0 for failure, 1 for success;\n",
    "* start_x: the x coordinate for the location where the action started, ranges from 0 to 105;\n",
    "* start_y: the y coordinate for the location where the action started, ranges from 0 to 68;\n",
    "* end_x: the x coordinate for the location where the action ended, ranges from 0 to 105;\n",
    "* end_y: the y coordinate for the location where the action ended, ranges from 0 to 68.\n",
    "\n",
    "The action types are:\n",
    "\n",
    "    0: pass\n",
    "    1: cross\n",
    "    2: throw in\n",
    "    3: freekick crossed\n",
    "    4: freekick short\n",
    "    5: corner crossed\n",
    "    6: corner short\n",
    "    7: take on\n",
    "    8: foul\n",
    "    9: tackle\n",
    "    10: interception\n",
    "    11: shot\n",
    "    12: shot penalty\n",
    "    13: shot freekick\n",
    "    14: keeper save\n",
    "    18: clearance\n",
    "    21: dribble\n",
    "    22: goalkick"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first load the actions for the selected game_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore('spadl.h5') as spadlstore:\n",
    "    df_actions = spadlstore[f'actions/game_{game_id}']\n",
    "    df_actions = (\n",
    "        df_actions.merge(spadlstore['actiontypes'], how='left')\n",
    "        .merge(spadlstore['results'], how='left')\n",
    "        .merge(spadlstore['bodyparts'], how='left')\n",
    "        .merge(spadlstore['players'], how=\"left\")\n",
    "        .merge(spadlstore['teams'], how='left')\n",
    "        .reset_index()\n",
    "        .rename(columns={'index': 'action_id'})\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at first 5 actions of the match\n",
    "df_actions[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells define some functions to make it more convenient to explore the actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nice_time(row):\n",
    "    minute = int((row['period_id']>=2) * 45 + (row['period_id']>=3) * 15 + \n",
    "                 (row['period_id']==4) * 15 + row['time_seconds'] // 60)\n",
    "    second = int(row['time_seconds'] % 60)\n",
    "    return f'{minute}m{second}s'\n",
    "\n",
    "df_actions['nice_time'] = df_actions.apply(nice_time,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def action_name(row):\n",
    "    return f\"{row['action_id']}: {row['nice_time']} - {row['short_name']} {row['type_name']}\"\n",
    "\n",
    "df_actions['action_name'] = df_actions.apply(action_name, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_actions(df_actions_to_plot):\n",
    "    matplotsoccer.actions(\n",
    "        location=df_actions_to_plot[['start_x', 'start_y', 'end_x', 'end_y']],\n",
    "        action_type=df_actions_to_plot['type_name'],\n",
    "        team=df_actions_to_plot['team_name'],\n",
    "        result=df_actions_to_plot['result_name'] == 'success',\n",
    "        label=df_actions_to_plot[['nice_time', 'type_name', 'short_name', 'short_team_name']],\n",
    "        labeltitle=['time', 'actiontype', 'player', 'team'],\n",
    "        zoom=False,\n",
    "        figsize=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_actions_from_action_name(df_actions, action_name):\n",
    "    action_id = int(action_name.split(':')[0])\n",
    "    df_actions_to_plot = df_actions[action_id-3: action_id+3]\n",
    "    plot_actions(df_actions_to_plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell shows a widget which we can use to easily explore the actions in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interact_manual(plot_actions_from_action_name, \n",
    "                df_actions = fixed(df_actions),\n",
    "                action_name=widgets.Select(\n",
    "                    options = df_actions['action_name'][3:],\n",
    "                    disabled=False)\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will create features that describe the action. We will take one action as an example. Every action has several characteristics, these include: \n",
    "* start location\n",
    "* end location\n",
    "* action type\n",
    "* bodypart used\n",
    "* result (successful or not)\n",
    "* time in the match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "action_id = 145\n",
    "plot_actions(df_actions[action_id:action_id+1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1].T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing locations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first take a look at how we can represent locations. We first define the pitch sizes as used in the SPADL representation. A very interesting blog post by the KU Leuven on the interplay between features and models in football can be found here: https://dtai.cs.kuleuven.be/sports/blog/illustrating-the-interplay-between-features-and-models-in-xg."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PITCH_LENGTH = 105\n",
    "PITCH_WIDTH = 68"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['start_x', 'start_y', 'end_x', 'end_y']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell adds columns to the actions dataframe with the normalized start and end coordinates of the actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for side in ['start', 'end']:\n",
    "    # Normalize the X location\n",
    "    key_x = f'{side}_x'\n",
    "    df_actions[f'{key_x}_norm'] = df_actions[key_x] / PITCH_LENGTH\n",
    "\n",
    "    # Normalize the Y location\n",
    "    key_y = f'{side}_y'\n",
    "    df_actions[f'{key_y}_norm'] = df_actions[key_y] / PITCH_WIDTH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['start_x_norm', 'start_y_norm', 'end_x_norm', 'end_y_norm']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells compute the polar coordinates of the start and end location of the actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GOAL_X = PITCH_LENGTH\n",
    "GOAL_Y = PITCH_WIDTH / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for side in ['start', 'end']:\n",
    "    diff_x = GOAL_X - df_actions[f'{side}_x']\n",
    "    diff_y = abs(GOAL_Y - df_actions[f'{side}_y'])\n",
    "    df_actions[f'{side}_distance_to_goal'] = np.sqrt(diff_x ** 2 + diff_y ** 2)\n",
    "    df_actions[f'{side}_angle_to_goal'] = np.divide(diff_x, diff_y, \n",
    "                                                    out=np.zeros_like(diff_x), \n",
    "                                                    where=(diff_y != 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['start_distance_to_goal', 'start_angle_to_goal', 'end_distance_to_goal', 'end_angle_to_goal']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell adds a column indicating whether the action starts and/or ends in the opponent penalty box. \n",
    "\n",
    "*Exercise*: think of more features describing the start and end location of the action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for side in ['start', 'end']:\n",
    "    df_actions[f'{side}_is_penalty_box'] = ((df_actions[f'{side}_x'] > (PITCH_LENGTH-16.5)) & \n",
    "                                            (df_actions[f'{side}_y'] > (13.85)) & \n",
    "                                            (df_actions[f'{side}_y'] > (PITCH_WIDTH-13.85)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['start_is_penalty_box', 'end_is_penalty_box']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing body part, action type and result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells inspect the type_name, bodypart_name and result_name columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(df_actions['type_name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_action_type_dummies(df_actions):\n",
    "    return df_actions.merge(pd.get_dummies(df_actions['type_name']), how='left',\n",
    "                             left_index=True, right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions = add_action_type_dummies(df_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions['bodypart_name'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions['result_name'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['bodypart_name', 'result_name']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine distance covered in x, y and total time played"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells define functions that add distance features and the actual time played in the match at the moment of the action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_distance_features(df_actions):\n",
    "    df_actions['diff_x'] = df_actions['end_x'] - df_actions['start_x']\n",
    "    df_actions['diff_y'] = df_actions['end_y'] - df_actions['start_y']\n",
    "    df_actions['distance_covered'] = np.sqrt((df_actions['end_x'] - df_actions['start_x']) ** 2 +\n",
    "                                             (df_actions['end_y'] - df_actions['start_y']) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_time_played(df_actions):\n",
    "    df_actions['time_played'] = (df_actions['time_seconds'] + \n",
    "                             (df_actions['period_id'] >= 2) * (45 * 60) + \n",
    "                             (df_actions['period_id'] >= 3) * (15 * 60) + \n",
    "                             (df_actions['period_id'] == 4) * (15 * 60)\n",
    "                             )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_distance_features(df_actions)\n",
    "add_time_played(df_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_actions[action_id:action_id+1][['diff_x', 'diff_y', 'distance_covered', 'time_played']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing game states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a game state is more than just the previous action, we want to add more information on the sequence leading up to the action. This includes information on the previous actions, but also more contextual features like the score and time in the match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing actions leading to game state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the \\\"[Actions Speak Louder than Goals: Valuing Player Actions in Soccer.](https://arxiv.org/abs/1802.07127)\\\" paper we take the three actions leading to the game state into account. We investigated how many actions we should include and got the best results for three actions back. \n",
    "\n",
    "*Exercise:* you could try out different numbers of actions to include in the game state and analyze the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first take a look at the action again and what happened before the action was executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    df_actions_to_plot = df_actions[145-i:146]\n",
    "    plot_actions(df_actions_to_plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells add extra columns for each action on the x actions before the action itself. You can define how far back you want to go and what features to take into account for all actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delays = 3  # you can play around with this number\n",
    "features_to_delay = ['game_id', 'period_id', 'time_seconds', 'team_id',\n",
    "                     'player_id', 'start_x', 'start_y', 'end_x', 'end_y', 'bodypart_id',\n",
    "                     'type_id', 'result_id', 'type_name', 'result_name', 'bodypart_name',\n",
    "                     'time_played']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_delayed_features(df_actions, features_to_delay, delays):\n",
    "    df_delays = [df_actions[features_to_delay].shift(step).add_suffix(f'-{step}') for step in range(0, delays)]\n",
    "    return pd.concat(df_delays, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features = create_delayed_features(df_actions, features_to_delay, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "location_cols = [f'{side}_{xy}-{delay}' for delay in reversed(range(0, delays))\n",
    "                 for xy in ['x', 'y'] for side in ['start', 'end']]\n",
    "df_features[action_id:action_id+1][location_cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = [f'{col}-{delay}' for delay in reversed(range(0, delays))\n",
    "        for col in ['period_id', 'time_seconds', 'type_name', 'bodypart_name', 'result_name']]\n",
    "df_features[action_id:action_id+1][cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to invert the coordinates for the actions made by the other team to make sure all features represent the same thing. Therefore we add an indicator for the delayed actions to indicate whether the action was executed by the same team or not. If not then we need to invert the coordinates as the coordinates normally always go from left to right for the team in possession."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_same_team(df_features, delays):\n",
    "    for step in range(1, delays):\n",
    "        df_features[f'team-{step}'] = df_features['team_id-0'] == df_features[f'team_id-{step}']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_same_team(df_features, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invert_coordinates(df_features, delays):\n",
    "    for step in range(1, delays):\n",
    "        for side in ['start', 'end']:\n",
    "            df_features.loc[~(df_features[f'team-{step}']), f'{side}_x-{step}'] = PITCH_LENGTH - df_features[f'{side}_x-{step}']\n",
    "            df_features.loc[~(df_features[f'team-{step}']), f'{side}_y-{step}'] = PITCH_WIDTH - df_features[f'{side}_y-{step}']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features[location_cols+['team-1', 'team-2']].tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "invert_coordinates(df_features, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features[location_cols+['team-1', 'team-2']].tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells add the location featues for all actions and their delayed actions using the right coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_location_features(df_features, delays):\n",
    "    for step in range(0, delays):\n",
    "        for side in ['start', 'end']:\n",
    "            # Normalize the X location\n",
    "            key_x = f'{side}_x'\n",
    "            df_features[f'{key_x}_norm-{step}'] = df_features[f'{key_x}-{step}'] / PITCH_LENGTH\n",
    "\n",
    "            # Normalize the Y location\n",
    "            key_y = f'{side}_y'\n",
    "            df_features[f'{key_y}_norm-{step}'] = df_features[f'{key_y}-{step}'] / PITCH_WIDTH\n",
    "\n",
    "            diff_x = GOAL_X - df_features[f'{side}_x-{step}']\n",
    "            diff_y = abs(GOAL_Y - df_features[f'{side}_y-{step}'])\n",
    "            df_features[f'{side}_distance_to_goal-{step}'] = np.sqrt(diff_x ** 2 + diff_y ** 2)\n",
    "            df_features[f'{side}_angle_to_goal-{step}'] = np.divide(diff_x, diff_y, out=np.zeros_like(diff_x), where=(diff_y != 0))\n",
    "\n",
    "            df_features[f'diff_x-{step}'] = df_features[f'end_x-{step}'] - df_features[f'start_x-{step}']\n",
    "            df_features[f'diff_y-{step}'] = df_features[f'end_y-{step}'] - df_features[f'start_y-{step}']\n",
    "            df_features[f'distance_covered-{step}'] = np.sqrt((df_features[f'end_x-{step}'] - df_features[f'start_x-{step}']) ** 2 +\n",
    "                                             (df_features[f'end_y-{step}'] - df_features[f'start_y-{step}']) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_location_features(df_features, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding contextual features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to add some additional context to the game state. For example the speed of play, the time in the match and the current score in the match might influence the value of the game state.\n",
    "\n",
    "*Exercise:* think of extra features that might influence the value of a game state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells add extra context on the sequence. The distance covered in x and y direction as well as the time passed in the sequence are added to the dataframe. These features might tell the model something about the speed of play, whether the game quickly progressed from one side to another (probably indicating a counter attack) or whether it is a slow build up play. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_sequence_pre_features(df_features, delays):\n",
    "    delay = delays - 1\n",
    "    df_features['xdiff_sequence_pre'] = df_features['start_x-0'] - df_features[f'start_x-{delay}']\n",
    "    df_features['ydiff_sequence_pre'] = df_features['start_y-0'] - df_features[f'start_y-{delay}']\n",
    "    df_features['time_sequence_pre'] = df_features['time_played-0'] - df_features[f'time_played-{delay}']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_sequence_pre_features(df_features, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_sequence_post_features(df_features, delays):\n",
    "    delay = delays - 1\n",
    "    df_features['xdiff_sequence_post'] = df_features['end_x-0'] - df_features[f'start_x-{delay}']\n",
    "    df_features['ydiff_sequence_post'] = df_features['end_y-0'] - df_features[f'start_y-{delay}']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_sequence_post_features(df_features, delays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_features[action_id:action_id+1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate features for a match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell defines a function that creates all features described above for a dataframe of actions for a given match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_features_match(df_actions, features_to_delay, delays):\n",
    "    df_action_features = add_action_type_dummies(df_actions)\n",
    "    add_time_played(df_action_features)\n",
    "    df_gamestate_features = create_delayed_features(df_action_features, features_to_delay, delays)\n",
    "    add_same_team(df_gamestate_features, delays)\n",
    "    invert_coordinates(df_gamestate_features, delays)\n",
    "    add_location_features(df_gamestate_features, delays)\n",
    "    add_sequence_pre_features(df_gamestate_features, delays)\n",
    "    add_sequence_post_features(df_gamestate_features, delays)\n",
    "    return df_gamestate_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate labels for a match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells define two functions that compute the labels of our two machine learning tasks. The first one tells us whether a goal was scored within x actions, whereas the second function tells us whether a goal was conceded within the next x actions.\n",
    "\n",
    "*Exercise*: play around with the number of actions to look ahead. In our paper we look 10 actions ahead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_scores(df_actions, nr_actions):\n",
    "    \"\"\"\n",
    "    This functiondf_actions determines whether a goal was scored by the team possessing \n",
    "    the ball within the next x actions\n",
    "    \"\"\"\n",
    "    # merging goals, owngoals and team_ids\n",
    "\n",
    "    goals = df_actions['type_name'].str.contains('shot') & (\n",
    "        df_actions['result_id'] == 1\n",
    "    )\n",
    "    owngoals = df_actions['type_name'].str.contains('shot') & (\n",
    "        df_actions['result_id'] == 2\n",
    "    )\n",
    "    y = pd.concat([goals, owngoals, df_actions['team_id']], axis=1)\n",
    "    y.columns = ['goal', 'owngoal', 'team_id']\n",
    "\n",
    "    # adding future results\n",
    "    for i in range(1, nr_actions):\n",
    "        for col in ['team_id', 'goal', 'owngoal']:\n",
    "            shifted = y[col].shift(-i)\n",
    "            shifted[-i:] = y[col][len(y) - 1]\n",
    "            y[f'{col}+{i}'] = shifted\n",
    "\n",
    "    scores = y['goal']\n",
    "    for i in range(1, nr_actions):\n",
    "        goal_scored = y[f'goal+{i}'] & (y[f'team_id+{i}'] == y['team_id'])\n",
    "        own_goal_opponent = y[f'owngoal+{i}'] & (y[f'team_id+{i}'] != y['team_id'])\n",
    "        scores = scores | goal_scored | own_goal_opponent\n",
    "\n",
    "    return pd.DataFrame(scores, columns=['scores'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_concedes(df_actions, nr_actions):\n",
    "    \"\"\"\n",
    "    This function determines whether a goal was scored by the team not \n",
    "    possessing the ball within the next x actions\n",
    "    \"\"\"\n",
    "    # merging goals,owngoals and team_ids\n",
    "    goals = df_actions['type_name'].str.contains('shot') & (\n",
    "        df_actions['result_id'] == 1\n",
    "    )\n",
    "    owngoals = df_actions['type_name'].str.contains('shot') & (\n",
    "        df_actions['result_id'] == 2\n",
    "    )\n",
    "    y = pd.concat([goals, owngoals, df_actions['team_id']], axis=1)\n",
    "    y.columns = ['goal', 'owngoal', 'team_id']\n",
    "    \n",
    "    # adding future results\n",
    "    for i in range(1, nr_actions):\n",
    "        for col in ['team_id', 'goal', 'owngoal']:\n",
    "            shifted = y[col].shift(-i)\n",
    "            shifted[-i:] = y[col][len(y) - 1]\n",
    "            y[f'{col}+{i}'] = shifted\n",
    "\n",
    "    concedes = y['owngoal']\n",
    "    for i in range(1, nr_actions):\n",
    "        goal_opponent = y[f'goal+{i}'] & (y[f'team_id+{i}'] != y['team_id'])\n",
    "        own_goal_team = y[f'owngoal+{i}'] & (y[f'team_id+{i}'] == y['team_id'])\n",
    "        concedes = concedes | goal_opponent | own_goal_team\n",
    "\n",
    "    return pd.DataFrame(concedes, columns=['concedes'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compute the labels for the match and investigate the labels for the last ten actions in the match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels = pd.concat([label_scores(df_actions, nr_actions=10), \n",
    "                           label_concedes(df_actions, nr_actions=10)], \n",
    "                          axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels.tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.merge(df_actions[['team_name', 'short_name', 'type_name']], \n",
    "         df_labels, \n",
    "         left_index=True, \n",
    "         right_index=True).tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate features and labels for all matches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells generate and store the features and labels for all matches in your spadl.h5 file. This might take some minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games = pd.read_hdf('spadl.h5', key='games')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    with pd.HDFStore('spadl.h5') as spadlstore:\n",
    "        df_actions = spadlstore[f'actions/game_{game_id}']\n",
    "        df_actions = (\n",
    "            df_actions.merge(spadlstore['actiontypes'], how='left')\n",
    "            .merge(spadlstore['results'], how='left')\n",
    "            .merge(spadlstore['bodyparts'], how='left')\n",
    "            .reset_index(drop=True)\n",
    "        )\n",
    "    \n",
    "    df_features = create_features_match(df_actions, features_to_delay, delays)\n",
    "    df_features.to_hdf('features.h5', f'game_{game_id}')\n",
    "    \n",
    "    df_labels = pd.concat([label_scores(df_actions, nr_actions=10), \n",
    "                           label_concedes(df_actions, nr_actions=10)], \n",
    "                          axis=1)\n",
    "    df_labels.to_hdf('labels.h5', f'game_{game_id}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}