Dense_Fully_Connected_NN/PyTorch/NBA_Example_Feedforward.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# ENGG 192 \n",
        "\n",
        "-----\n",
        "\n",
        "### Dartmouth College - January 20, 2019\n",
        "### Spencer Bertsch\n",
        "This program uses PyTorch to create a dense, feedforward neural networks designed to predict the results of NBA games betoween 2014 and 2018.  \n",
        "\n",
        "The next Assignment (1b) will entail a notebook which studies several regularization techniques such as hyperparameter optimization techniques, techniques to limit the effects of overfitting such as early stopping and dropout layers, and will discuss other methods and theiries currently used to improve the performance of funtional, applied deep learning models. \n",
        "\n",
        "The data source for this project can be found [here](https://www.kaggle.com/ionaskel/nba-games-stats-from-2014-to-2018#nba.games.stats.csv). \n",
        "\n"
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Deep learning \n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.autograd import Variable #<-- Variable can be thought of as a Torch matrix\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import pandas as pd\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.colors import ListedColormap\n",
        "\n",
        "#Data cleaning and preprocessing\n",
        "from sklearn import preprocessing\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "#Model evaluation\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn import metrics\n",
        "from sklearn.metrics import f1_score\n"
      ],
      "outputs": [],
      "execution_count": 27,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load the Dataset \n",
        "\nOur dataset exists as a .csv file downloaded form Kaggle. In a more developed project we could develop the way that data is loaded into this notebook, but for now we can simply use a handy feature of Pandas called 'read_csv' to import a .csv to a pandas dataframe. "
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import the dataset - load .csv file\n",
        "path = '/Users/spencerbertsch/Desktop/ENGG 192/Dense_NNs/Data/nba_raw_data.csv'\n",
        "#create a dataframe from the NBA data stored in the .csv file\n",
        "NBA_data_raw = pd.read_csv(path)\n",
        "\n",
        "print(\"Shapw of raw dataset:\", (NBA_data_raw.shape))\n",
        "print(\"So we can see that we have 40 features and one label vector\")\n",
        "print(\"We can also see that we have 9,840 rows\")\n",
        "print(\" \")\n",
        "print(\"We can also preview our dataset by observing the top five rows\")\n",
        "NBA_data_raw.head(5)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shapw of raw dataset: (9840, 41)\n",
            "So we can see that we have 40 features and one label vector\n",
            "We can also see that we have 9,840 rows\n",
            " \n",
            "We can also preview our dataset by observing the top five rows\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 28,
          "data": {
            "text/plain": [
              "   Unnamed: 0 Team  Game      Date  Home Opponent WINorLOSS  TeamPoints  \\\n",
              "0           1  ATL     1  10/29/14  Away      TOR         L         102   \n",
              "1           2  ATL     2   11/1/14  Home      IND         W         102   \n",
              "2           3  ATL     3   11/5/14  Away      SAS         L          92   \n",
              "3           4  ATL     4   11/7/14  Away      CHO         L         119   \n",
              "4           5  ATL     5   11/8/14  Home      NYK         W         103   \n",
              "\n",
              "   OpponentPoints  FieldGoals       ...        Opp.FreeThrows  \\\n",
              "0             109          40       ...                    27   \n",
              "1              92          35       ...                    18   \n",
              "2              94          38       ...                    27   \n",
              "3             122          43       ...                    20   \n",
              "4              96          33       ...                     8   \n",
              "\n",
              "   Opp.FreeThrowsAttempted  Opp.FreeThrows.  Opp.OffRebounds  \\\n",
              "0                       33            0.818               16   \n",
              "1                       21            0.857               11   \n",
              "2                       38            0.711               11   \n",
              "3                       27            0.741               11   \n",
              "4                       11            0.727               13   \n",
              "\n",
              "   Opp.TotalRebounds  Opp.Assists  Opp.Steals  Opp.Blocks  Opp.Turnovers  \\\n",
              "0                 48           26          13           9              9   \n",
              "1                 44           25           5           5             18   \n",
              "2                 50           25           7           9             19   \n",
              "3                 51           31           6           7             19   \n",
              "4                 44           26           2           6             15   \n",
              "\n",
              "   Opp.TotalFouls  \n",
              "0              22  \n",
              "1              26  \n",
              "2              15  \n",
              "3              30  \n",
              "4              29  \n",
              "\n[5 rows x 41 columns]"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>Team</th>\n",
              "      <th>Game</th>\n",
              "      <th>Date</th>\n",
              "      <th>Home</th>\n",
              "      <th>Opponent</th>\n",
              "      <th>WINorLOSS</th>\n",
              "      <th>TeamPoints</th>\n",
              "      <th>OpponentPoints</th>\n",
              "      <th>FieldGoals</th>\n",
              "      <th>...</th>\n",
              "      <th>Opp.FreeThrows</th>\n",
              "      <th>Opp.FreeThrowsAttempted</th>\n",
              "      <th>Opp.FreeThrows.</th>\n",
              "      <th>Opp.OffRebounds</th>\n",
              "      <th>Opp.TotalRebounds</th>\n",
              "      <th>Opp.Assists</th>\n",
              "      <th>Opp.Steals</th>\n",
              "      <th>Opp.Blocks</th>\n",
              "      <th>Opp.Turnovers</th>\n",
              "      <th>Opp.TotalFouls</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>ATL</td>\n",
              "      <td>1</td>\n",
              "      <td>10/29/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>TOR</td>\n",
              "      <td>L</td>\n",
              "      <td>102</td>\n",
              "      <td>109</td>\n",
              "      <td>40</td>\n",
              "      <td>...</td>\n",
              "      <td>27</td>\n",
              "      <td>33</td>\n",
              "      <td>0.818</td>\n",
              "      <td>16</td>\n",
              "      <td>48</td>\n",
              "      <td>26</td>\n",
              "      <td>13</td>\n",
              "      <td>9</td>\n",
              "      <td>9</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>ATL</td>\n",
              "      <td>2</td>\n",
              "      <td>11/1/14</td>\n",
              "      <td>Home</td>\n",
              "      <td>IND</td>\n",
              "      <td>W</td>\n",
              "      <td>102</td>\n",
              "      <td>92</td>\n",
              "      <td>35</td>\n",
              "      <td>...</td>\n",
              "      <td>18</td>\n",
              "      <td>21</td>\n",
              "      <td>0.857</td>\n",
              "      <td>11</td>\n",
              "      <td>44</td>\n",
              "      <td>25</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>18</td>\n",
              "      <td>26</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>ATL</td>\n",
              "      <td>3</td>\n",
              "      <td>11/5/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>SAS</td>\n",
              "      <td>L</td>\n",
              "      <td>92</td>\n",
              "      <td>94</td>\n",
              "      <td>38</td>\n",
              "      <td>...</td>\n",
              "      <td>27</td>\n",
              "      <td>38</td>\n",
              "      <td>0.711</td>\n",
              "      <td>11</td>\n",
              "      <td>50</td>\n",
              "      <td>25</td>\n",
              "      <td>7</td>\n",
              "      <td>9</td>\n",
              "      <td>19</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>ATL</td>\n",
              "      <td>4</td>\n",
              "      <td>11/7/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>CHO</td>\n",
              "      <td>L</td>\n",
              "      <td>119</td>\n",
              "      <td>122</td>\n",
              "      <td>43</td>\n",
              "      <td>...</td>\n",
              "      <td>20</td>\n",
              "      <td>27</td>\n",
              "      <td>0.741</td>\n",
              "      <td>11</td>\n",
              "      <td>51</td>\n",
              "      <td>31</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>19</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>ATL</td>\n",
              "      <td>5</td>\n",
              "      <td>11/8/14</td>\n",
              "      <td>Home</td>\n",
              "      <td>NYK</td>\n",
              "      <td>W</td>\n",
              "      <td>103</td>\n",
              "      <td>96</td>\n",
              "      <td>33</td>\n",
              "      <td>...</td>\n",
              "      <td>8</td>\n",
              "      <td>11</td>\n",
              "      <td>0.727</td>\n",
              "      <td>13</td>\n",
              "      <td>44</td>\n",
              "      <td>26</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "      <td>15</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 41 columns</p>\n",
              "</div>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 28,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\" \")\n",
        "print(\"We can then make a list from the column headers and observe all of the information that our dataset has aggregated for each game\")\n",
        "cols = list(NBA_data_raw.columns.values) #Make a list of all of the columns in the df\n",
        "cols"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " \n",
            "We can then make a list from the column headers and observe all of the information that our dataset has aggregated for each game\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 29,
          "data": {
            "text/plain": [
              "['Unnamed: 0',\n",
              " 'Team',\n",
              " 'Game',\n",
              " 'Date',\n",
              " 'Home',\n",
              " 'Opponent',\n",
              " 'WINorLOSS',\n",
              " 'TeamPoints',\n",
              " 'OpponentPoints',\n",
              " 'FieldGoals',\n",
              " 'FieldGoalsAttempted',\n",
              " 'FieldGoals.',\n",
              " 'X3PointShots',\n",
              " 'X3PointShotsAttempted',\n",
              " 'X3PointShots.',\n",
              " 'FreeThrows',\n",
              " 'FreeThrowsAttempted',\n",
              " 'FreeThrows.',\n",
              " 'OffRebounds',\n",
              " 'TotalRebounds',\n",
              " 'Assists',\n",
              " 'Steals',\n",
              " 'Blocks',\n",
              " 'Turnovers',\n",
              " 'TotalFouls',\n",
              " 'Opp.FieldGoals',\n",
              " 'Opp.FieldGoalsAttempted',\n",
              " 'Opp.FieldGoals.',\n",
              " 'Opp.3PointShots',\n",
              " 'Opp.3PointShotsAttempted',\n",
              " 'Opp.3PointShots.',\n",
              " 'Opp.FreeThrows',\n",
              " 'Opp.FreeThrowsAttempted',\n",
              " 'Opp.FreeThrows.',\n",
              " 'Opp.OffRebounds',\n",
              " 'Opp.TotalRebounds',\n",
              " 'Opp.Assists',\n",
              " 'Opp.Steals',\n",
              " 'Opp.Blocks',\n",
              " 'Opp.Turnovers',\n",
              " 'Opp.TotalFouls']"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 29,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Cleaning\n",
        "\n",
        "We can see that our target vector (Win or Loss) is the seventh column from the left, so the first thing we need to do is move that vector to the end (right side) of the dataframe. \n",
        "\n",
        "The next thing we can see is that several of these feature vectors in addition to our label vector contain categorical data in the form of strings. The general way to deal with categorical data is to encode this data into numeric data - this will allow our dense neural netowrk to learn from this data. \n",
        "\nAfter our features are seperated from the label and we have purely numeric data, we can scale our dataset and we will be ready for training! "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "#move label vector to the end of the dataframe \n",
        "\n",
        "cols = list(NBA_data_raw.columns.values) #Make a list of all of the columns in the df\n",
        "cols.pop(cols.index('WINorLOSS')) #Remove WINorLOSS from list\n",
        "df = NBA_data_raw[cols+['WINorLOSS']] #Create new dataframe with columns in the order you want\n",
        "df.head(5) "
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 30,
          "data": {
            "text/plain": [
              "   Unnamed: 0 Team  Game      Date  Home Opponent  TeamPoints  OpponentPoints  \\\n",
              "0           1  ATL     1  10/29/14  Away      TOR         102             109   \n",
              "1           2  ATL     2   11/1/14  Home      IND         102              92   \n",
              "2           3  ATL     3   11/5/14  Away      SAS          92              94   \n",
              "3           4  ATL     4   11/7/14  Away      CHO         119             122   \n",
              "4           5  ATL     5   11/8/14  Home      NYK         103              96   \n",
              "\n",
              "   FieldGoals  FieldGoalsAttempted    ...      Opp.FreeThrowsAttempted  \\\n",
              "0          40                   80    ...                           33   \n",
              "1          35                   69    ...                           21   \n",
              "2          38                   92    ...                           38   \n",
              "3          43                   93    ...                           27   \n",
              "4          33                   81    ...                           11   \n",
              "\n",
              "   Opp.FreeThrows.  Opp.OffRebounds  Opp.TotalRebounds  Opp.Assists  \\\n",
              "0            0.818               16                 48           26   \n",
              "1            0.857               11                 44           25   \n",
              "2            0.711               11                 50           25   \n",
              "3            0.741               11                 51           31   \n",
              "4            0.727               13                 44           26   \n",
              "\n",
              "   Opp.Steals  Opp.Blocks  Opp.Turnovers  Opp.TotalFouls  WINorLOSS  \n",
              "0          13           9              9              22          L  \n",
              "1           5           5             18              26          W  \n",
              "2           7           9             19              15          L  \n",
              "3           6           7             19              30          L  \n",
              "4           2           6             15              29          W  \n",
              "\n[5 rows x 41 columns]"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>Team</th>\n",
              "      <th>Game</th>\n",
              "      <th>Date</th>\n",
              "      <th>Home</th>\n",
              "      <th>Opponent</th>\n",
              "      <th>TeamPoints</th>\n",
              "      <th>OpponentPoints</th>\n",
              "      <th>FieldGoals</th>\n",
              "      <th>FieldGoalsAttempted</th>\n",
              "      <th>...</th>\n",
              "      <th>Opp.FreeThrowsAttempted</th>\n",
              "      <th>Opp.FreeThrows.</th>\n",
              "      <th>Opp.OffRebounds</th>\n",
              "      <th>Opp.TotalRebounds</th>\n",
              "      <th>Opp.Assists</th>\n",
              "      <th>Opp.Steals</th>\n",
              "      <th>Opp.Blocks</th>\n",
              "      <th>Opp.Turnovers</th>\n",
              "      <th>Opp.TotalFouls</th>\n",
              "      <th>WINorLOSS</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>ATL</td>\n",
              "      <td>1</td>\n",
              "      <td>10/29/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>TOR</td>\n",
              "      <td>102</td>\n",
              "      <td>109</td>\n",
              "      <td>40</td>\n",
              "      <td>80</td>\n",
              "      <td>...</td>\n",
              "      <td>33</td>\n",
              "      <td>0.818</td>\n",
              "      <td>16</td>\n",
              "      <td>48</td>\n",
              "      <td>26</td>\n",
              "      <td>13</td>\n",
              "      <td>9</td>\n",
              "      <td>9</td>\n",
              "      <td>22</td>\n",
              "      <td>L</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>ATL</td>\n",
              "      <td>2</td>\n",
              "      <td>11/1/14</td>\n",
              "      <td>Home</td>\n",
              "      <td>IND</td>\n",
              "      <td>102</td>\n",
              "      <td>92</td>\n",
              "      <td>35</td>\n",
              "      <td>69</td>\n",
              "      <td>...</td>\n",
              "      <td>21</td>\n",
              "      <td>0.857</td>\n",
              "      <td>11</td>\n",
              "      <td>44</td>\n",
              "      <td>25</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>18</td>\n",
              "      <td>26</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>ATL</td>\n",
              "      <td>3</td>\n",
              "      <td>11/5/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>SAS</td>\n",
              "      <td>92</td>\n",
              "      <td>94</td>\n",
              "      <td>38</td>\n",
              "      <td>92</td>\n",
              "      <td>...</td>\n",
              "      <td>38</td>\n",
              "      <td>0.711</td>\n",
              "      <td>11</td>\n",
              "      <td>50</td>\n",
              "      <td>25</td>\n",
              "      <td>7</td>\n",
              "      <td>9</td>\n",
              "      <td>19</td>\n",
              "      <td>15</td>\n",
              "      <td>L</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>ATL</td>\n",
              "      <td>4</td>\n",
              "      <td>11/7/14</td>\n",
              "      <td>Away</td>\n",
              "      <td>CHO</td>\n",
              "      <td>119</td>\n",
              "      <td>122</td>\n",
              "      <td>43</td>\n",
              "      <td>93</td>\n",
              "      <td>...</td>\n",
              "      <td>27</td>\n",
              "      <td>0.741</td>\n",
              "      <td>11</td>\n",
              "      <td>51</td>\n",
              "      <td>31</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>19</td>\n",
              "      <td>30</td>\n",
              "      <td>L</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>ATL</td>\n",
              "      <td>5</td>\n",
              "      <td>11/8/14</td>\n",
              "      <td>Home</td>\n",
              "      <td>NYK</td>\n",
              "      <td>103</td>\n",
              "      <td>96</td>\n",
              "      <td>33</td>\n",
              "      <td>81</td>\n",
              "      <td>...</td>\n",
              "      <td>11</td>\n",
              "      <td>0.727</td>\n",
              "      <td>13</td>\n",
              "      <td>44</td>\n",
              "      <td>26</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "      <td>15</td>\n",
              "      <td>29</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 41 columns</p>\n",
              "</div>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 30,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Continued data cleaning\n",
        "\nWe can see that our label vector has been moved to the end of the dataframe. We can now encode our categorical data so that we have a purely numeric dataset. "
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#define a function to label encode a 2D (x,y) matrix and return an encoded matrix\n",
        "def int_encode(df):\n",
        "    cols = df.shape[1]\n",
        "    rows = df.shape[0]\n",
        "    original_size = df.shape[1]\n",
        "    \n",
        "    for i in range(0, cols, 1):\n",
        "        values = df.iloc[:,i]\n",
        "        #integer encode each column\n",
        "        label_encoder = LabelEncoder()\n",
        "        integer_encoded = label_encoder.fit_transform(values) #integer encode the i'th column \n",
        "        encoded_vec = pd.DataFrame(integer_encoded) #convert to pd.dataframe\n",
        "        df = pd.concat([df, encoded_vec], axis=1) #concatenate the new, encoded vector onto the original dataframe\n",
        "    \n",
        "    final_df = df.iloc[:,-cols:] #Only keep the last (leftmost) half of the columns representing the encoded data\n",
        "    print(\"Integer encoding successful!\")\n",
        "    print(\"Dataframe shape: \", final_df.shape)\n",
        "    return(final_df)\n"
      ],
      "outputs": [],
      "execution_count": 31,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can now use our function to convert our categorical data into integer encoded, numeric data. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "df2 = int_encode(df)\n",
        "df2.head(5) #preview our new dataset"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Integer encoding successful!\n",
            "Dataframe shape:  (9840, 41)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 32,
          "data": {
            "text/plain": [
              "   0  0  0    0  0   0   0   0   0   0 ...   0    0   0   0   0   0  0   0  \\\n",
              "0  0  0  0  143  0  27  37  44  21  19 ...  32  210  16  27  20  13  9   7   \n",
              "1  1  0  1  155  1  11  37  27  16   8 ...  20  231  11  23  19   5  5  16   \n",
              "2  2  0  2  251  0  26  27  29  19  31 ...  37  145  11  29  19   7  9  17   \n",
              "3  3  0  3  259  0   4  54  57  24  32 ...  26  163  11  30  25   6  7  17   \n",
              "4  4  0  4  263  1  19  38  31  14  20 ...  10  154  13  23  20   2  6  13   \n",
              "\n",
              "    0  0  \n",
              "0  15  0  \n",
              "1  19  1  \n",
              "2   8  0  \n",
              "3  23  0  \n",
              "4  22  1  \n",
              "\n[5 rows x 41 columns]"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>...</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "      <th>0</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>143</td>\n",
              "      <td>0</td>\n",
              "      <td>27</td>\n",
              "      <td>37</td>\n",
              "      <td>44</td>\n",
              "      <td>21</td>\n",
              "      <td>19</td>\n",
              "      <td>...</td>\n",
              "      <td>32</td>\n",
              "      <td>210</td>\n",
              "      <td>16</td>\n",
              "      <td>27</td>\n",
              "      <td>20</td>\n",
              "      <td>13</td>\n",
              "      <td>9</td>\n",
              "      <td>7</td>\n",
              "      <td>15</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>155</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "      <td>37</td>\n",
              "      <td>27</td>\n",
              "      <td>16</td>\n",
              "      <td>8</td>\n",
              "      <td>...</td>\n",
              "      <td>20</td>\n",
              "      <td>231</td>\n",
              "      <td>11</td>\n",
              "      <td>23</td>\n",
              "      <td>19</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>16</td>\n",
              "      <td>19</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>251</td>\n",
              "      <td>0</td>\n",
              "      <td>26</td>\n",
              "      <td>27</td>\n",
              "      <td>29</td>\n",
              "      <td>19</td>\n",
              "      <td>31</td>\n",
              "      <td>...</td>\n",
              "      <td>37</td>\n",
              "      <td>145</td>\n",
              "      <td>11</td>\n",
              "      <td>29</td>\n",
              "      <td>19</td>\n",
              "      <td>7</td>\n",
              "      <td>9</td>\n",
              "      <td>17</td>\n",
              "      <td>8</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>259</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>54</td>\n",
              "      <td>57</td>\n",
              "      <td>24</td>\n",
              "      <td>32</td>\n",
              "      <td>...</td>\n",
              "      <td>26</td>\n",
              "      <td>163</td>\n",
              "      <td>11</td>\n",
              "      <td>30</td>\n",
              "      <td>25</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>17</td>\n",
              "      <td>23</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>4</td>\n",
              "      <td>263</td>\n",
              "      <td>1</td>\n",
              "      <td>19</td>\n",
              "      <td>38</td>\n",
              "      <td>31</td>\n",
              "      <td>14</td>\n",
              "      <td>20</td>\n",
              "      <td>...</td>\n",
              "      <td>10</td>\n",
              "      <td>154</td>\n",
              "      <td>13</td>\n",
              "      <td>23</td>\n",
              "      <td>20</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "      <td>13</td>\n",
              "      <td>22</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 41 columns</p>\n",
              "</div>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 32,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The last step in preprocessing is to scale our dataset. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "#Scaling\n",
        "df_scaled = preprocessing.scale(df2)\n",
        "print(df_scaled[:,5:]) #print first 5 rows of the new numpy matrix"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 1.44417822 -0.13558717  0.43892343 ... -1.19881172  0.4498342\n",
            "  -1.        ]\n",
            " [-0.4043699  -0.13558717 -0.9563166  ...  1.12716247  1.37653127\n",
            "   1.        ]\n",
            " [ 1.32864396 -0.9563166  -0.79217071 ...  1.38560405 -1.17188567\n",
            "  -1.        ]\n",
            " ...\n",
            " [-1.67524673 -0.54595188 -0.05351423 ...  1.12716247  0.4498342\n",
            "  -1.        ]\n",
            " [-1.55971247  0.7672152  -0.21766011 ...  0.61027932 -0.47686287\n",
            "   1.        ]\n",
            " [ 0.75097267 -0.9563166  -0.21766011 ...  0.61027932  1.60820554\n",
            "  -1.        ]]\n"
          ]
        }
      ],
      "execution_count": 33,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Seperate training and testing data\n",
        "y = df2.iloc[:, -1] #labels are the last column in the matrix\n",
        "X = df2.iloc[:, :40] #assign feature set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)\n",
        "print(\"X train shape: \", X_train.shape)\n",
        "print(\"y train shape: \", y_train.shape)\n",
        "print(\"X test shape: \", X_test.shape)\n",
        "print(\"y test shape: \", y_test.shape) \n",
        "\n",
        "X_train = X_train.values\n",
        "X_test = X_test.values\n",
        "y_train = y_train.values \n",
        "y_test = y_test.values"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "X train shape:  (6888, 40)\n",
            "y train shape:  (6888,)\n",
            "X test shape:  (2952, 40)\n",
            "y test shape:  (2952,)\n"
          ]
        }
      ],
      "execution_count": 34,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "And with that, we have fully prepared our dataset! Remember that the 41st column - the label vector - represents the result of the game. This is what we're trying to predict. We're now ready to create a neural network in TensorFlow and begin training."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Deep Learning - PyTorch\n",
        "\n",
        "------\n",
        "\n",
        "### SOURCES\n",
        "\n",
        "When learning how to construct feedforward networks (or any type of netowrk) in PyTorch, I referenced [Sung Kim's Github](https://github.com/hunkim). There, Sung Kim has layed out a repository called [PyTorch Zero To All](https://github.com/hunkim/PyTorchZeroToAll) in which he includes slide decks and code explaining how to build neural netowrks using pytorch from first principles. This was my main resource - in addition to Sung Kim's [Youtube Series](https://www.youtube.com/watch?v=SKq-pmkekTk) - which I used for the next section of this notebook. \n",
        "\n",
        "Other important sources include: \n",
        "\n",
        "\"Deep Learning\" by Ian Goodfellow and Yoshua Bengio and Aaron Courville. \n",
        "\n",
        "@book{Goodfellow-et-al-2016, <br>\n",
        "    title={Deep Learning}, <br>\n",
        "    author={Ian Goodfellow and Yoshua Bengio and Aaron Courville}, <br>\n",
        "    publisher={MIT Press}, <br>\n",
        "    note={\\url{http://www.deeplearningbook.org}}, <br>\n",
        "    year={2016} <br>\n",
        "}\n",
        "\n\n"
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class NBA_Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        \"\"\"\n",
        "        \"super can be used to refer to parent classes without naming them explicitly\"\n",
        "        \n",
        "        We can now construct the network using torch.nn.linear layers \n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "        self.l1 = torch.nn.Linear(40, 20)\n",
        "        self.dropout = nn.Dropout(0.5)\n",
        "        self.l2 = torch.nn.Linear(20, 10)\n",
        "        self.l3 = torch.nn.Linear(10, 1)\n",
        "\n",
        "        self.sigmoid = torch.nn.Sigmoid()\n",
        "        \n",
        "        \n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Our forward function takes in the input data and uses the sigmoid activation function to generate the output\n",
        "        \"\"\"\n",
        "        \n",
        "        out1 = self.sigmoid(self.l1(x))\n",
        "        out2 = self.sigmoid(self.l2(out1))\n",
        "        y_pred = self.sigmoid(self.l3(out2))\n",
        "        return y_pred\n",
        "        "
      ],
      "outputs": [],
      "execution_count": 35,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## A quick note on regularization techniques\n",
        "\n",
        "Note that one dropout layer with p=0.5 was used between the first and secodn layer of the network. There is no generally accepted way to introduce specific regularization techniques to boost model performance or limit the negative effects of overfitting other than simple experementation and testing. James Bergstra and Yoshua Bengio outline this well in their 2012 paper, [Random Search for Hyper-Parameter Optimization](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf) in which they discuess optimization of several hyperpapameters used in deep learning such as network topology (number of layers and neurons per layer) and learning rate. They conclude it's best to use a random search method as opposed to grid search when finding the correct hyperparameters which will yeild the best results from a network. \n",
        "\nThis topic will be further discussed in a later notebook when I focus exclusively on regularization techniques. "
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define Learning Parameters\n",
        "\n",
        "model = NBA_Net() #<-- Use the model created above \n",
        "#We use mean squared error as our loss function becasue the problem at hand is very simple \n",
        "loss_fn = nn.BCELoss()\n",
        "#Adam is a very strong genric optimizer, so we can use it here\n",
        "optimizer = optim.Adam(model.parameters(), lr=1e-3)\n",
        "\n",
        "training_epochs = 500\n",
        "minibatch_size = 10"
      ],
      "outputs": [],
      "execution_count": 36,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are now ready to train our network. Simple guess and check was used to arrive at the parameter values above. In the next notebook, I will explore the ways that Regularization texhniques can be used to improve model performance. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "t0 = time.clock()#define t0 so we can measure training time for different network variations\n",
        "\n",
        "for i in range(training_epochs):\n",
        "    if ((i%25) == 0):\n",
        "        print(\"Epoch\", i, \"/\", training_epochs) #Print our progress\n",
        "    \n",
        "    for batch_ind in range(4):\n",
        "        # wrap the data in variables\n",
        "        minibatch_state_var = Variable(torch.Tensor(X_train))\n",
        "        minibatch_label_var = Variable(torch.Tensor(y_train))\n",
        "        \n",
        "        # forward pass\n",
        "        y_pred = model(minibatch_state_var)\n",
        "        \n",
        "        # loss is MSE - defined above \n",
        "        # compute loss as the difference between the prediction and the true label \n",
        "        loss = loss_fn(y_pred, minibatch_label_var)\n",
        "\n",
        "        # now that the forward pass is done, we need to reset all gradients\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        # backwards pass\n",
        "        loss.backward()\n",
        "        \n",
        "        # step the optimizer - update the weights\n",
        "        optimizer.step()\n",
        "        \n",
        "print(\"- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \")     \n",
        "print(\"TRAINING COMPLETE\")\n",
        "print(\"Number of Epochs Trained: \", training_epochs)\n",
        "print(\"With minibatch size: \", minibatch_size)\n",
        "print(\"Training time: \", time.clock()) \n",
        "print(\"- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0 / 500\n",
            "Epoch 25 / 500\n",
            "Epoch 50 / 500\n",
            "Epoch 75 / 500\n",
            "Epoch 100 / 500\n",
            "Epoch 125 / 500\n",
            "Epoch 150 / 500\n",
            "Epoch 175 / 500\n",
            "Epoch 200 / 500\n",
            "Epoch 225 / 500\n",
            "Epoch 250 / 500\n",
            "Epoch 275 / 500\n",
            "Epoch 300 / 500\n",
            "Epoch 325 / 500\n",
            "Epoch 350 / 500\n",
            "Epoch 375 / 500\n",
            "Epoch 400 / 500\n",
            "Epoch 425 / 500\n",
            "Epoch 450 / 500\n",
            "Epoch 475 / 500\n",
            "- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \n",
            "TRAINING COMPLETE\n",
            "Number of Epochs Trained:  500\n",
            "With minibatch size:  10\n",
            "Training time:  127.476201\n",
            "- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2016: UserWarning: Using a target size (torch.Size([6888])) that is different to the input size (torch.Size([6888, 1])) is deprecated. Please ensure they have the same size.\n",
            "  \"Please ensure they have the same size.\".format(target.size(), input.size()))\n"
          ]
        }
      ],
      "execution_count": 37,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Test the trained model with our holdout data \"X_test\"\n",
        "input_1 = Variable(torch.Tensor(X_test)) # \"variable\" simply creates a new tenosr\n",
        "input_1\n",
        "\n",
        "#Make predictions on the inputs \n",
        "pred_1 = model(input_1)\n",
        "print(\"Prediction for the 1st inputs:\") \n",
        "pred_1"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction for the 1st inputs:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 38,
          "data": {
            "text/plain": [
              "tensor([[0.0030],\n",
              "        [0.0032],\n",
              "        [0.0025],\n",
              "        ...,\n",
              "        [0.9960],\n",
              "        [0.9956],\n",
              "        [0.9960]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 38,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Evaluation\n",
        "\n",
        "------\n",
        "\n",
        "Now that our network has been trained and we have predicted the results using our testing data, we can determine the model's performance. We know that the label veoctor is fairly well balanced, so using Accuracy as a performance metric makes sense here. I will also use both F1 Score and the ROC Score to make sure the results of testing hold up under different XXX. Lastly, and arguable most importantly, we will examine the confusion matrix to determine the rates of Type I and Type II error. \n",
        "\n",
        "We know that $Accuracy = [(TP + TN)/(TP + TN + FP + FN)]$, and becasue our label vector is well balanced with a ratio of (XX/YY), we know accuracy will yeild a strong performance metric. However, it's important to make sure the error is not shifted too far to one side, so we use the F1 Score as well. \n",
        "\n",
        "Remember that $F1 Score = 2 [(Precision*Recall)/(Precision + Recall)]$ \n",
        "\n",
        "where $Precision = [True Positive/(True Positive + False Positive)]$, in other words, $Precision = [True Positive/Total Predicted Positive]$\n",
        "\n",
        "and $Recall = [True Positive/(True Positive + False Negative)]$, in other words, $Precision = [True Positive/Total Actual Positive]$\n",
        "\n",
        "Source: [Towards Data Science](https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9)\n",
        "\nPerhaps most telling will be the confusion matrix, which will gave us the number of true and false predictions on both the \"prediction\" and \"actual\" side. "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "#Lets see how we did! \n",
        "pred_1_np = pred_1.detach().numpy()\n",
        "y_pred = np.where(pred_1_np>0.5,1,0) #Convert to binary \n",
        "y_true = y_test\n",
        "A = accuracy_score(y_true, y_pred)\n",
        "F1 = f1_score(y_true, y_pred)\n",
        "print(\"Accuracy:\", A)\n",
        "print(\"F1 Score:\", F1)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 0.9928861788617886\n",
            "F1 Score: 0.9928106812735363\n"
          ]
        }
      ],
      "execution_count": 39,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ROC SCORE - PLOT \n",
        "fpr1, tpr1, thresholds1 = metrics.roc_curve(y_true, y_pred)\n",
        "area1 = metrics.auc(fpr1, tpr1)\n",
        "area1 = str(round(area1, 3))\n",
        "print(\"ROC Score: \", area1)\n",
        "\n",
        "x = range(1)\n",
        "y = range(1)\n",
        "fig = plt.figure(figsize=(8, 6))\n",
        "ax1 = fig.add_subplot(111)\n",
        "\n",
        "plt.plot(fpr1, tpr1, color='darkorange', label=('ROC Curve (area = ' + area1 + ')')) \n",
        "plt.plot([0, 1], [0, 1], color='navy', linestyle='--', label=('Random Guess'))\n",
        "plt.legend(loc='lower right');\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.title('Validation Set ROC Curve - Three Layer Feedforward Neural Network')\n",
        "plt.grid()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ROC Score:  0.993\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 576x432 with 1 Axes>"
            ],
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3WeYFGX29/HvGTKCZAEJggQBUaIg4COIiqAIIqjAmkAFRQVz+BtWXd01i1lQWVdFEAUFEUVXGaJIEpCwAiJhABHJQ56Z+3lR1dAOExpmemqm+/e5rrmmq6u66nR1VZ0+d91Vbc45REREpOBLCDoAERERyR1K6iIiIjFCSV1ERCRGKKmLiIjECCV1ERGRGKGkLiIiEiOU1I+RmdUyM2dmhf3hr8zsukimPY5l/Z+ZvZOTeCU2+NtR3aDjiDdm9piZfRg23MPM1ptZspk1CzK27BSkbcbMrjezGUHHkRuCXu9xl9TNbLKZPZHB893N7PdjTcDOuS7Ouf/kQlwdzCwp3bz/6Zy7MafzzmBZRc3sBTNL8g9Ov5nZSxG+9i8HuUymWWNm+/x5/25m75lZqXTTtDWz781st5ntNLMvzKxRumlONLOhZrbOn9cqf7hiJss1MxtsZkvMbI///j4xszMieW9B8b+8Jft/+80sNWx4aT6IL9HMcn07zAl/f0kLW0/JZvZFHiz6eeA251wp59xPebC8qPA/0/1mViPsuQvMbE2AYWUorDj6Mt3zH5rZYxHOY42ZXRCVAPOZuEvqwHvANWZm6Z6/BhjpnEvJ+5Dy3INAS6AVUBo4D8jtA9SlzrlSQFOgmb9MAMysDfANMB44GagNLAJmmtmp/jRFge+A04HOwIlAW2CrH3dGXgaGAIOB8kB94HPgkmMN/nhbV46H/+WtlL++bgZ+CA07504/1vnlZex5IYv3szFsPZVyzl2aB+GcAhzXFy0zK5TLsYTme7yf9x7gkdyMJSO5uD2ebWbtcmleuS7f7HfOubj6A0oAO4Fzw54rB+wHmvjDl+AluV3AeuCxsGlrAQ4o7A8nAjf6jwvhfZP/E1gN3Jpu2n7AcmC3P36g//wJwD4gDUj2/04GHgM+DFt2N7wDyg5/uQ3Dxq0B7gEW++/vY6B4JutgInBHFuvoZGAssAX4DRjsP98ZOAgc8mNclMnr1wAXhA0/C3wZNjwdeCOD130FvO8/vhHYDJSK8HOtB6QCrbKY5vBn5Q9fD8wIG3b+Z7bSf99vAc+nm8d44K6s1lMOt8+/xJQutpv92LYDrwMW9pqZwEvANuBJ//n+/va2HZgMnBI2vwbAt/70vwBXRrre0o37BPjd3+amAaf7z5/lf36Fw6btCSz0HycADwC/4n1RGwOUT7eP3QCsA6ZlsNwOQFImMWU6b3/82cAsvP1oEdAhbFxtYCrePvot8BrwIVAMb5t3eMnwV3/6hv762YG3b3YLm9d7wJvAJP81N/nTJfjj3wH+CJv+Q/z9kkyOFeHvHbjfX/cf+M/fC2wCNvqfvQPqZvGZ/t2ff13/uQuANdkdB8Le25OZfR54x4D78Y5HB4DCYZ/JbmAZ0CO77T7d9nA/MCXd+go/NncFFvrreBZwpv/8B3jH1n3+Z3gf8B/gbn98NX/+g/zhunj7RWj/uglY5T83ATg5s2NG2HOhdXoOXg45L6fHhoiPIXm1oPz0B7wNvBM2PBD/YBO2gZ6Bd3A4E+/gdFm6DSyjpH4z8D+gBl6lOCXdtJcAdQAD2gN7geYZ7RT+c4/hJ3W8qnMPcCFQxN8wVwFFw3aiOXg7Ynm8A8LNmbz/h/EOloP892lh4xKA+cCjQFHgVLyDykXpY8pi/a7BT+pAdeBn4GV/uCRe8j1qI8c7kG3yH48G/nMMn+nNwNpspjn8WfnD13N0Uv/WX38lgHP9HTK0c5fDOzCcnN16ysG2+ZeY0sU2ESgL1MQ70HYOe00KcDvewbMEcJm/fTT0n3sYmOVPf4L/vvr545rjfRE9PZL1lm5cf7zWnmLAUP66Hy0DuoQNf8aRA+kdwGx/+ygGDANGpdvH3vdjLZHBcjuQeVLPat7V8BL9xf5neKE/XMkf/wPwov+6c/ES0IfpPofQAbuIv47/z98GOvrTn+aPfw/vy047f1nF8fa7Fv74X/xtpqE/vA5oFuGxIgV4xo+zBN4X7s1AY3+dfUT2Sf1G/72GjjGHkzrZHwfeI/ukvhDvWFjCf+4Kjuw7V+Edz6pmtd2n2x5KARs4cmw5nNTxtuE/gNZ4xdV1fgzF0h+TwrbbL/zHffG+bHwcNm68/7gj3r7R3F/XrxL2JZN0x4zwbQS4CG8/y7TQiMZfni0oP/3hfXvaGfYhzATuzGL6ocBL6TawjJL694QlUqBT+LQZzPdzYEhGO4X/3GNhO9wjwJiwcQn+Bt4hbKO9Omz8s8BbmSy3EN63y5l436I3Atf541oD69JN/yDw7/QxZbG+1uB9I97tv//vgLL+uOr+cw0yeF1n4JD/+Fvg6WP4TB8CZmczzeHPyh++nqOTesewYcM70J7rD98EfB/JesrBtvmXmNLFdk7Y8BjggbDXpI/lK+CGdNvLXrzm46uA6emmHwb8PZL1lkXsZf04y/jD9+Od0gLvoLeXIwfx5cD5Ya+titcCVJgj+9ipWSyrA171tSPs78oI5n0/fmUbNn4yXhKoiZcsTwgb9xGZJ/X/h1cpJ4SNH8WRRPMefstT2PgPgLuAKnhJ/Vm8L6S1CaviM3i/6Y8VBwlriQNGELa/4BUBkST1SnjHwtP5a1LP7jjwHtkn9f7ZbC8Lge5Zbff+uND2UBivEJntPx+e1N8E/pHudb8A7cPiCU/qdULrG69FbmAofrwqPtQa9y7wbNjrSvnbUq2w7aFjuuU6f12tBc7IyfHgeP7i8Zw6zrkZeJVOd/8c7ll4Oy8AZtbazKaY2RYz24m302XYOSudk/G+mYWsDR9pZl3MbLaZbTOzHXjVQiTzDc378Pycc2n+sqqFTfN72OO9eBvgUZxzqc65151z7fAOxE8BI8ysId5B/2Qz2xH6w6tEKkcYZ8hlzrnSeDt7A468z+14B+OqGbymKt63YvCqp4ymycyxTp+Zw5+f8/bQ0UAf/6m+wEj/ccTrycxqhnfoykFsWX2+69NNewrwclhs2/C+pFTzx7VOF/vf8BJNxMyskJk9bWa/mtkuvAMnHPmsPwQu9TtJXon3RWJTWHyfhS1/OV4LTvj6S/+e0tvonCsb9jcmgnmfAlyR7r2fg7ftnAxsd87tCVvGX/bhdE4G1vv7Yvj04ftk+vcwFW+fOBfvdEUiXiXeHm/9pEFEx4otzrn96WOJMO7DnHNb8E4xpO88nBvHgb+8dzO71swWhs2vMZEf/0LeBiqbWfr+E6cAd6eLtwbeejmKc+5XvMKjKd6Xs4nARjM7De+zmOpPmv64m4x3rMnqMwavtWiMc+7nY3x/ORaXSd33PnAtXge5b5xzm8PGfYR37qSGc64M3je59B3rMrIJb0MKqRl6YGbF8M5PPQ9Uds6VxTvXFpqvy2beG/E23ND8zF/WhgjiypRzbp9z7nW8ZNsIbwP9Ld3BsrRz7uII40w//6l43+qf94f34DVxXpHB5FfiVfUA/wUuMrMTIlzUd0B1M2uZxTR78Jr/QzJKYunf3yigl5mdgle9jPWfz249HZmhc+tcWIeuCN/PsUof93q887Dh8ZVwzs3yx01NN66Uc+6WY1xmX6A7XoVXBq+iAn+bds5twPuse+DtZx+ki69LuhiK+6/J7D1FKqt5r8er1MPHneCcexpv/y2XbpurmdECfBuBGmYWfhytyV/3yfTvYSpeEungP56B1zx/OJFEcKzIaL6ZHnsi8BxeZ9kWYc9lt30f077k7z9vA7cBFfz3tITIjqtHZujcIeBx4B/pXrseeCpdvCWdc6PSxxJmKtAL7xTmBn/4WrzTbAv9adIfd08AKpD1Zwze8e0yM7vjWN5fboj3pH4BXpNq+kvSSgPbnHP7zawV3sErEmOAwWZW3czK4XUMCSmKd05mC5BiZl3wmudDNgMVzKxMFvO+xMzON7MiwN14TeezIoztMDO7w78kqISZFTbvOvvSeJ0D5wC7zOx+f3whM2tsZmeFxVkr3YEsO0OBC82sqT/8AHCdeZeflTazcmb2JNAGb4cFLwGsB8aaWQMzSzCzCuZd/pVR4lwJvAGM8t9bUTMrbma9zSz0OSwELjezkuZdR3pDdoE777KlLXidmiY753b4o7JbT0F7C3jQzE4HMLMyZhb6IjURqG9m15hZEf/vLL+lJjOF/fUZ+iuCt80cwKtcSgL/zOB17+P1/zgD75x6eHxP+Qd7zKySmXXPwfsNl9W8Q60HF/mfWXF/e6nunFsLzAMe97efc4CsetT/iJfc7vPXYQd/+tGZvcDfTvcBV+Odm92Ft0/15Eh1mN2xIiNjgOvNrJGZlcTrBBcRf5t+Ae9zCslu+14IXGxm5c2sCl5lmpUT8JLfFgAz64dXqR+PD/DWT+ew594GbjavldXM7AQzu8TMSvvjN+P1Cwg3Fe9LxjR/OBGvX8oM51yq/9xHQD8za+p/2fon8KNzbk02MW4EzsfLB4OO+R3mQNwmdf9DmYW3sU1IN3oQ8ISZ7cbrKDKGyLyNd35uEbAAGBe2vN14l1qNwauK+4Yv1zn3P7yqcLXffPSXZiPn3C94B4JX8ZqoL8W7bOxghLGF24e3E//uz+tWoKdzbrW/MV+K1yz1mz/+HbxKDLzezgBbzWxBJAvzm/jex798xj/9cRFwOV6FsRbvsrdz/IMezrkDeF+6/od3fn0X3oGmIt7BNCOD8ZoSX8c7X/YrXpUYun75JbxzkZvxvsiNzGAeGRnlx3L4FE0E6ylQzrnP8DpSjTavaXwJ0MUftxsvSfTGO/j8zpFOV5l5E2+7Cf39G+8zXYtXtSzD65yW3mf4zeHpmrVfxtv+v/H3s9l4LSG5IdN5O+fW47Uu/B9eglmP12s8dCzs60+7DS8xvp/ZQvx9rxveev0T70vltf6+nJWpwFbn3LqwYcO/rDS7Y0UmsXyF9+X5e7zOe99nE0N6L+OdogjNL7vt+wO849wavMtTP84mvmV4x5wf8Pa/M/D69BwzP7a/4/XTCD03D69Aew1vna3CO08f8i/gYf/Yeo//3FS8L6ahpD4D78tpaBjn3Hd4x62xeMeqOnj7TSRxrsNL7PdbHt7nIdSrV0QkKszsV7xTAf8NOhaRWBe3lbqIRJ+Z9cRrdj3WylFEjkP+uAOOiMQcM0vE63x5Tboe4iISJWp+FxERiRFqfhcREYkRSuoiIiIxosCdU69YsaKrVatWrs1vz549nHBCpPc3kcxoPeac1mHOaR3mnNZhzuX2Opw/f/6fzrlKkUxb4JJ6rVq1mDdvXq7NLzExkQ4dOuTa/OKV1mPOaR3mnNZhzmkd5lxur0Mzi+i2v6DmdxERkZihpC4iIhIjlNRFRERihJK6iIhIjFBSFxERiRFK6iIiIjFCSV1ERCRGKKmLiIjECCV1ERGRGBG1pG5mI8zsDzNbksl4M7NXzGyVmS02s+bRikVERCQeRLNSfw/onMX4LkA9/28A8GYUYxEREYl5Ubv3u3NumpnVymKS7sD7zvtB99lmVtbMqjrnNkUrJhERCZhzgPP+u7Qjj0k3nNnjrKaLxrhjnMfBg6mU27Uc6JBnqzRckD/oUg1YHzac5D93VFI3swF41TyVK1cmMTEx14JITk7O1fnFK63HDPg7unHkIHb4cQbDB5KTmfHfCf5rOHzAOOoxaf5r8V+bhrmwx8exbHNhj4NcdlZxHLXs9NM7Kh84wOpRH2axbDD8AzFgLpPHhw/e/mPS/NdmFuuRA3tmyz4yPf5wWsaPj2le4dOHvbcs3mfW69zROi2V/Ystg2Xjf95hj/1lHX7sf/7esjN5bzFs+uqaXDPqcv5z9ZckntgykBiCTOqWwXMZfuLOueHAcICWLVu63Pz1m8O/ppOWApt+hJT9R759ZfutLS3sG9oxjItk/qGDyvHMP+JlZzLdcYz744/NnOQqHUNc6abLatxxxRX2OKtxOYkr03Ghg5jkLwZmR/5bQthzCbk/LvQY/3FCBtOFxmU0j6zGHVNc6abLapwlsOn3zVStUvUY4gp7nNW4HMYVzLIjnL8/7H7cTslvl5NUbzB/C+iX7oJM6klAjbDh6sDGgGKBFZ/Cl30CW3zussw30AwPADkcZ0apvfvhz025N/+EhOOP6y8HgGi870gP5Mc2btWqX6lbr14gy45o/gVg2dOmT+fc9h0ymYdE4pfERKrqp1cjtm7dTr755lduvLE5514OP3fvwvTp0wKLJ8ikPgG4zcxGA62BnYGeTz+w0/vfbRyUqJT5QT6jA/7xjjuO5Jn9/IM5eM3RbzDnWNLuROo27xB0GAVaWkJRKFQ06DAkTnz22XJuuGECqamO7t1Po1KlEyhUKNgrxaOW1M1sFF5PgYpmlgT8HSgC4Jx7C5gEXAysAvYC/aIVS0Rcqvf/5LZwQuVAQxERkfxr375D3H33N7z55jxatKjK6NG9qFTphKDDAqLb+z3Ltmy/1/ut0Vr+MUvzk7oVCjYOERHJt1JT02jf/j3mzt3I3Xe34Z//PJ+iRfNP3giy+T2f8Ts3JeSfD0dERPIH5xxmRqFCCdxyS0sef7wUXbrUCzqso+g2sSGHK3WtEhEROWLHjv1cddWnfPyxd4PUfv2a5cuEDkrqR4QuQ1Lzu4iI+GbPTqJZs2GMG7eczZv3BB1OtpTUQ5wqdRER8aSlOZ5+egbnnDMCgBkz+jN4cOuAo8qezqmHOHWUExERz3ffrebBB7/jyitPZ9iwrpQtWzzokCKipB7i1FFORCTebdiwi2rVTuTCC+uQmHgd5557ClaAbl6ktuYQdZQTEYlbBw+mcs8931C37qssWfIHAO3b1ypQCR1UqR9xuKOckrqISDxZtWobffqMZd68jQwa1JI6dcoFHdJxU1IPcalK6CIicWbkyMXcfPOXFCmSwLhxV9KjR8OgQ8oRJfUQl6pOciIicWbRos00bVqFkSMvp2bNMkGHk2NK6iEuTZ3kRETiwE8/beLAgVTOPrs6Tz3VETOjcOHYaKmNjXeRG9JS0eoQEYldzjlefnk2Z5/9LnfeORnnHEWKFIqZhA6q1MOoUhcRiVV//rmXfv3GM3HiCi69tD4jRnQvcD3bI6GkHpKmjnIiIrFozZodtGs3gj//3MvLL3fm9ttbxWRCByX1I9RRTkQkJtWsWYZu3eozYEALmjWrGnQ4UaXSNMSlKamLiMSIdet20q3bKJKSdpGQYLz5ZteYT+igSv0IXacuIhITxo1bzg03TCAlJY1ly7ZQvfqJQYeUZ5TFQnRJm4hIgbZv3yEGDfqSnj3HULdueRYuHEinTnWCDitPKamH6JI2EZEC7fHHp/Lmm/O45542zJzZnzp1ygcdUp5T83uIS1WlLiJSwDjn2LnzAGXLFufBB8/h/PNrc+GF8VWdh1NpGqKOciIiBcqOHfu56qpP6dDhPfbvT6FMmeJxndBBSf0IdZQTESkwfvhhPU2bvsVnn/2PPn0aU7SoijJQ8/sRqtRFRPK91NQ0nnlmJo8+OoUaNcowY0Y/WreuHnRY+YaSeogqdRGRfO/QoTQ+/ngpvXo1YtiwrpQpUzzokPIVJfWQNHWUExHJr7799ldat67OiScWY+rU6ylTpljM3uo1J1Sahqj5XUQk3zl4MJW7755Mp04f8vTTMwAoW7a4EnomVKmHqPldRCRfWbVqG717f8r8+Zu49dazePTR9kGHlO8pqYeoUhcRyTcmT15Fr16fUKRIAuPGXUmPHg2DDqlAUFIP0c1nRETyjUaNKtGxY21efbULNWuWCTqcAkPtzSG6TayISKAWLNjEoEFfkpbmqFGjDOPH91ZCP0bKYofpB11ERILgnGPo0NmcffY7jB//C+vX7ww6pAJLze8haeooJyKS17Zs2UO/fuP58suVXHppfUaM6E7FiiWDDqvAUlIPcWmQoNUhIpJXnHN07TqKhQt/55VXOnPbba10qVoOKYuFuFSwYkFHISIS81JS0nDOUaRIIV566SJKlixC06ZVgg4rJqi9OUTXqYuIRN3atTto3/49Hn10CgBt29ZQQs9FymIhTh3lRESiaezYZTRtOoyff97MmWdWDjqcmKTm9xB1lBMRiYp9+w5x552TGTZsPmeddTKjRvWkTp3yQYcVk5TFQnRHORGRqPj11+385z+LuO++tsyY0V8JPYpUqYe4VCV1EZFc4pxj2rS1tG9fi8aNT2LVqtupVu3EoMOKearUQ9RRTkQkV+zYsZ8rr/yUDh3+w3ffrQZQQs8jqtRD1PwuIpJjs2atp2/fsWzYsJtnnrmA886rHXRIcUVJPUSVuohIjgwdOpt77vmGmjXLMGNGP1q3rh50SHFHST1El7SJiORI5concMUVp/PWW5dQpkzxoMOJS0rqIWnqKCcicqwmTVrJH3/s4frrm9Knzxn07t1Yt3oNkNqbQ9T8LiISsQMHUrjrrslccslHvPnmPFJT0wCU0AOmSj1EHeVERCKycuVWevcey4IFm7j11rN4/vlOFCqkoig/UFIPUaUuIpKtP/7YQ4sWwylcOIHPPruKyy5rEHRIEkZJPSQtVR3lREQykZqaRqFCCZx00gk8/3wnunSpS40aZYIOS9JRaRqi5ncRkQwtWLCJxo3fZMaMdQAMGNBCCT2fUlIPUfO7iMhfOOcYOnQ2Z5/9DsnJB1EfuPwvqlnMzDqb2S9mtsrMHshgfE0zm2JmP5nZYjO7OJrxZEmVuojIYVu27KFr11HceedkLr64HgsXDqRdu5pBhyXZiFpSN7NCwOtAF6AR0MfMGqWb7GFgjHOuGdAbeCNa8WRLlbqIyGEfffQz3323mtde68Jnn11FhQolgw5JIhDNjnKtgFXOudUAZjYa6A4sC5vGAaG7/JcBNkYxnqzpV9pEJM6lpKSxZs0eAG6/vTVdutSjfv0KAUclxyKapWk1YH3YcJL/XLjHgKvNLAmYBNwexXiyptvEikgcW7t2B+3bv8eQIQvZvn0fCQmmhF4ARbNSz6hLhUs33Ad4zzn3gpm1AT4ws8bOubS/zMhsADAAoHLlyiQmJuZakMnJySQmJtI+NYV169bzWy7OO56E1qMcP63DnNM6PD5Tp27h+edXkJbmGDSoJosW/Rh0SAVakNthNJN6ElAjbLg6Rzev3wB0BnDO/WBmxYGKwB/hEznnhgPDAVq2bOk6dOiQa0EmJibSoUMHmO84pdapnNIu9+YdTw6vRzluWoc5p3V4bFJS0rjttkkMG7aMVq2qMWpUT9atW6R1mENBbofRbH6fC9Qzs9pmVhSvI9yEdNOsA84HMLOGQHFgSxRjyphzgFNHORGJK4UKGTt3HuC++9oyfXo/Tj21XNAhSQ5FrVJ3zqWY2W3AZKAQMMI5t9TMngDmOecmAHcDb5vZnXhN89c759I30UefS/X+q6OciMQ45xxvv72A9u1P4bTTKjJy5OUkJOgC9FgR1dvEOucm4XWAC3/u0bDHy4B20YwhIqFT+OooJyIxbPv2fdx00xeMHbucwYNb8fLLXZTQY4zu/Q7efd8B3WBPRGLVrFnr6dNnLBs37uaZZy7gnnvaBh2SRIGSOgCq1EUkdk2evIpLLvmImjXLMGNGP1q3rh50SBIlKk3hSKWujnIiEkNCXZTat6/Fvfe25aefBiqhxzhlMVBHORGJOV9+uYK2bUewa9cBihcvzL/+dQFlyhQPOiyJMiV1ONJRTkldRAq4AwdSuPPOr+nadRR79x5i27Z9QYckeUjn1CGsUtd3HBEpuFas2EqfPmNZsGATt9/eimefvZDixXWYjyf6tEGXtIlITLjjjq9Zs2YH48f3plu304IORwKgpA7qKCciBdbu3Qc4eDCVChVKMnz4pQBUr35iNq+SWKUsBuooJyIF0vz5G2nefDjXXfc54CVzJfT4pqQO6ignIgWKc46XXvqBNm3eZf/+FO67L/gbc0r+oOZ3UEc5ESkw/vxzL9dd9zmTJq2ke/fTePfdblSoUDLosCSfUFIHdZQTkQLll1/+5LXXujBo0FmY6d7tcoSSOuje7yKSrx06lMrw4fMZMKAFFSuWZNmyWylaVEWIHE1JHY40v6tSF5F8Zs2aHfTpM5bZs5OoUqUUPXs2UkKXTCmpgzrKiUi+9MknS7nppi9IS3OMGtWTnj0bBR2S5HNqbwZ1lBORfOfJJ6dx5ZWfctppFVm48GZ6924cdEhSAKhSB1XqIpLvdO1anz17DvLEE+dRpIiOTRIZJXVQpS4igXPOMWzYfJYt28Irr3ShadMqNG1aJeiwpIBRFoMjvd/VUU5EArB9+z6uuOITbrnlS1as2MqBAylBhyQFlCp1UPO7iARm5sx19O07jo0bd/Pccxdy111tSEjQtedyfJTUQc3vIhKI3bsPcOmloyhXrgSzZvXnrLOqBR2SFHBK6qBKXUTy1J9/7qVChRKULl2MCRP6cOaZlTnxxGJBhyUxQKUpqFIXkTzz5ZcraNjwdYYNmw/AOefUVEKXXKMsBmG/p65KXUSi48CBFO6882u6dh1FtWql6dChVtAhSQxS8zsA+kEXEYmeFSu20rv3p/z00+/cfnsrnn32QooX1+FXcp+2Kgir1NVwISK5b/Xq7axfv4vx43vTrdtpQYcjMUxZDNRRTkRy3e7dBxg//n8AdO5cl9WrByuhS9QpqYM6yolIrpo3byPNmg3jyis/ZcOGXQCULq3OcBJ9ymIQltRVqYvI8UtLc7z44g+0bfsuBw6k8t//XkO1aicGHZbEEZ1ThyPN7+ooJyLHyTlHjx4fM2HCL1x2WQPefbcb5cuXCDosiTNK6qCOciKSY2ZG+/an0KnTqQwadBZmutWr5D0ldVBHORE5LocOpfL3vyfStm0Nunatz113tQk6JIlzKk1BHeVE5JitWbODc899j3/9awbTpq0NOhwRQJW6Rx3lROQYfPLJUm666Qucg9Gje3LVVY2DDkkEUFL3qKOciERo+vS1XHnlp7RuXY1Ro3pSu3a5oEMSOUxJHY4VzG98AAAgAElEQVR0lNPZCBHJxJ49BznhhKKcc05NRo68nCuuaESRIioEJH9RFgN073cRyYxzjrfemketWi+zYsVWzIy+fc9QQpd8SZU66JI2EcnQ9u37uPHGLxg3bjkXXVSHMmV0VzjJ35TUQR3lROQoM2euo2/fcWzcuJvnn7+QO+9sQ0KCrj2X/E1JHXSduogcZdSoJRQpksCsWf0566xqQYcjEpGI2pvNrKiZ1Y12MIHRdeoiAmzYsIuff94MwHPPXciCBQOV0KVAyTaLmdklwM/At/5wUzP7LNqB5Sld0iYS9yZOXEGTJm9xzTWf4ZyjRIkinHiizqFLwRJJafoE0BrYAeCcWwjEVtWujnIicevAgRTuuONrLr10FDVqlOHjj3vpvu1SYEVyTv2Qc25Huo3cRSmeYKijnEhc2rw5mS5dRvLTT78zeHArnnnmQooXV1cjKbgi2XqXm9mVQIKZ1QaGALOjG1YeU0c5kbhUoUJJatYsw2OPdaBbt9OCDkckxyJpb74NaIF3h5ZxwH68xB471FFOJG7s2nWA22+fxB9/7KFw4QQ+/7y3ErrEjEiy2EXOufudc838vweALtEOLE8drtSV1EVi2bx5G2nefBhvvDGPKVN+CzockVwXSRZ7OIPnHsrtQALlUgEDdY4RiUlpaY4XXphF27bvcvBgKlOnXq9fVpOYlOk5dTO7COgMVDOzF8NGncjhm6XHiLRUXc4mEsP++c/pPPLIFHr0aMA773SjfPkSQYckEhVZdZT7A1iCdw59adjzu4EHohlUnnNp6iQnEoMOHUqlSJFC3HxzS6pWLUX//s10uZrEtEyTunPuJ+AnMxvpnNufhzHlPZeq8+kiMeTQoVQefXQK06atIzHxOipWLMkNNzQPOiyRqIskk1Uzs9FmttjMVoT+Ipm5mXU2s1/MbJWZZVjdm9mVZrbMzJaa2UfHFH1uUaUuEjPWrNnBuee+x9NPz6RRo4qkpMTW2UKRrERynfp7wJPA83i93vsRwTl1MysEvA5cCCQBc81sgnNuWdg09YAHgXbOue1mdtIxv4PcoEpdJCYkJm5h6NC3cA5Gj+6pznASdyLJZCWdc5MBnHO/OuceBs6L4HWtgFXOudXOuYPAaKB7umluAl53zm335/9H5KHnInWUEynwDhxIYcSI32jYsBILFw5UQpe4FEmlfsC8niW/mtnNwAYgkoq6GrA+bDgJ7x7y4eoDmNlMoBDwmHPu6/QzMrMBwACAypUrk5iYGMHiI5OcnMyGbeuplJLGrFycb7xJTk7O1c8lHmkdHp/ffttD1arFKV68EI89VpeaNcuydu0i1q4NOrKCSdthzgW5DiNJ6ncCpYDBwFNAGaB/BK/LqItp+nvGFwbqAR2A6sB0M2vsnNvxlxc5NxwYDtCyZUvXoUOHCBYfmcTERKoVqwJ7ipGb8403iYmJWn85pHV4bJxzvPXWPO66aya33XYWzz13PqB1mFPaDnMuyHWYbVJ3zv3oP9wNXANgZtUjmHcSUCNsuDqwMYNpZjvnDgG/mdkveEl+bgTzzz3qKCdSoGzbto8bb5zAZ5/9j86d63Lvve2CDkkkX8jynLqZnWVml5lZRX/4dDN7n8h+0GUuUM/MaptZUaA3MCHdNJ/jn5/3l1EfWH2M7yHn1FFOpMCYO3cDTZu+xcSJK3j++Qv58su+nHTSCUGHJZIvZJrJzOxfwEjgb8DXZvYQMAVYhH8uPCvOuRS8H4OZDCwHxjjnlprZE2bWzZ9sMrDVzJb5877XObc1J2/ouLhUVeoiBUS5ciWoWLEks2bdwN13tyUhQTeTEQnJqvm9O9DEObfPzMrjNZ03cc79EunMnXOTgEnpnns07LED7vL/guPS1PtdJB9LStrFv//9Ew8/fC5165Zn/vwBujOcSAayanPe75zbB+Cc2wb871gSeoGSpuZ3kfxqwoRfaNLkLZ55ZiYrV24DUEIXyURWlfqpZjbOf2xArbBhnHOXRzWyvKSOciL5zv79Kdx337e8+uocmjWrwqhRPalfv0LQYYnka1kl9Z7phl+LZiCBUkc5kXznsstGM3nyrwwZ0ppnnrmAYsUiuQJXJL5l9YMu3+VlIIFSRzmRfMHrZuM1r999dxtuu60VXbtm2y9XRHz66gvqKCeSD+zadYBbbvmSRo0q8tBD53LhhXWCDkmkwFGbM3gd5bQqRAIzd+4GmjcfxujRS9QJTiQHIq7UzayYc+5ANIMJjip1kSCkpTlefPEHHnzwO6pWLcXUqddzzjk1gw5LpMDKtjw1s1Zm9jOw0h9uYmavRj2yvKRL2kQCsWzZFh544L9ceml9Fi68WQldJIciqdRfAbri3dIV59wiM4vkp1cLDnWUE8lTK1dupV69CjRufBJz595E06ZV1OwukgsiKU8TnHPpf8QwNRrBBEbXqYvkiUOHUnnwwf/SoMHrfPvtrwA0a1ZVCV0kl0RSqa83s1aAM7NCwO3AiuiGlcd0nbpI1P3223b69h3H7NlJ3HRTc9q1U1O7SG6LJKnfgtcEXxPYDPzXfy526JI2kagaO3YZ/ftPwAzGjOnFFVecHnRIIjEpkqSe4pzrHfVIgpSWCoV1yb5ItGzbto9GjSoxalRPatUqG3Q4IjErkjbnuWY2ycyuM7PSUY8oCOooJ5LrFi/ezPjx/wPgxhubM316PyV0kSjLNqk75+oATwItgJ/N7HMzi63KXR3lRHKNc47XX59Dq1Zvc++935KSkoaZUbiw+q2IRFtEe5lzbpZzbjDQHNgFjIxqVHlNHeVEcsW2bfu4/PIx3HbbV3TsWJsZM/ormYvkoWxPJJtZKaA70BtoCIwH2kY5rrylSl0kx7Zv30fTpm/x++/JvPBCJ+6442wSEnSpmkheiqR32BLgC+BZ59z0KMcTDFXqIjlWrlwJBg5sQefOdWnR4uSgwxGJS5FkslOdc7fHbEIHr/e7LmkTOWZJSbvo1OkD5s/fCMBDD52rhC4SoEwrdTN7wTl3NzDWzFz68c65y6MaWV5S87vIMZsw4Rf69RvPgQMprF27U8lcJB/Iqvn9Y///a3kRSKDU/C4Ssf37U7jvvm959dU5NGtWhdGje1G/foWgwxIRskjqzrk5/sOGzrm/JHYzuw34LpqB5SlV6iIRe+edBbz66hyGDGnNM89cQLFiunGTSH4RSXnaP4PnbsjtQAKlSl0kS845fv89GYCbb25JYuJ1DB3aWQldJJ/J6pz6VXiXsdU2s3Fho0oDO6IdWJ5SRzmRTO3adYBbbvmSKVN+4+efb6FChZK0b18r6LBEJANZfc2eA2wFqgOvhz2/G/gpmkHlOTW/i2Ro7twN9OkzljVrdvD44x0oW7Z40CGJSBayOqf+G/Ab3q+yxTY1v4v8RVqa48UXf+DBB7/j5JNLM3Xq9fqpVJECIKvm96nOufZmth0Iv6TNAOecKx/16PKKKnWRvzCDqVPX0q3babzzzqWUK1ci6JBEJAJZNb+f5/+vmBeBBEqVuggA3377K/XqVaBWrbJ8/HEvSpQojJlu9SpSUGSayZxzaf7DGkAh51wq0AYYCJyQB7HlHf30qsS5Q4dSuf/+b+nU6UMeeywRgJIliyihixQwkZSnnwPOzOoA7+P9qMtHUY0qr7k09X6XuLV69XbOOeffPPvsLAYObMEbb1wSdEgicpwiucg0zTl3yMwuB4Y6514xs9jq/Z6m5neJTz/8sJ7OnUdiBp98cgW9ejUKOiQRyYFIknqKmV0BXANc5j9XJHohBUAd5SROnXFGZbp1O41//OM8atUqG3Q4IpJDkd5R7jy8n15dbWa1gVHRDSuPqaOcxJHFizfTq9cY9u49RKlSRfnggx5K6CIxIttM5pxbAgwG5plZA2C9c+6pqEeWl9RRTuKAc47XX59Dq1ZvM3Pmelav3h50SCKSy7Jtfjez/wd8AGzAu0a9ipld45ybGe3g8ow6ykmM27ZtH/37j2f8+F+4+OJ6vPdedypViq2LWEQksnPqLwEXO+eWAZhZQ7wk3zKageUZ57ykHtGZCJGCqX//8UyatJIXX+zEkCFnk5CgS9VEYlEkSb1oKKEDOOeWm1nRKMaUx/yb5alSlxiTmprGvn0plCpVlOef78Qjj5xLixYnBx2WiERRJEl9gZkNw6vOAf5GDP2gi+HfY0cd5SSGJCXt4uqrx1GxYkk++eQK6taNnbs6i0jmIslkNwO/AvcB9wOr8e4qFxMsdOM8dZSTGDFhwi80afIW8+ZtpFu303RXOJE4kmWlbmZnAHWAz5xzz+ZNSHlNSV1iw/79Kdx337e8+uocmjWrwujRvahfv0LQYYlIHsq0Ujez/8O7RezfgG/NrH+eRZWHjlTqan6Xgm3Hjv18/PFS7rijNT/8cIMSukgcyqpS/xtwpnNuj5lVAiYBI/ImrLzkJ3V1lJMCyDnHxIkruPjielSpUorly2+lfHn9TKpIvMqqPD3gnNsD4Jzbks20BZYqdSmodu7cT9++4+jWbTQjR/4MoIQuEueyqtRPNbNx/mMD6oQN45y7PKqR5RF1lJOCaM6cDfTu/Snr1u3kqac68re/nRF0SCKSD2SV1HumG34tmoEER0ldCpZ33lnALbd8ycknl2batH60bVsj6JBEJJ/INKk7577Ly0CCouZ3KWhOP70SPXs25M03L6FcOTW3i8gRkdx8JqZZ6I5yqtQlH/vmm1+ZM2cDDz98Lm3a1KBNG1XnInI0lacu1fuvSl3yoYMHU7nvvm+56KIP+fjjpezdeyjokEQkH4u4UjezYs65A9EMJgimS9okn1q9ejt9+oxlzpwNDBzYghdfvIiSJYsEHZaI5GOR/PRqK+BdoAxQ08yaADc6526PdnB5wZya3yX/2bfvEO3ajWDfvkN88skV9OrVKOiQRKQAiKTN+RWgK7AVwDm3CDgvkpmbWWcz+8XMVpnZA1lM18vMnJkF8HOuan6X/OPAgRQASpQowrBhXVm48GYldBGJWCSZLME5tzbdc6nZvcjMCgGvA12ARkAfMzvq6GRmpYHBwI8RxJLrVKlLfrFqVTJNmw7jgw8WAdCt22nUqlU24KhEpCCJJKmv95vgnZkVMrM7gBURvK4VsMo5t9o5dxAYDXTPYLp/AM8C+yMNOjfpp1claM45XnttDoMGLWDnzv1Uq3Zi0CGJSAEVSUe5W/Ca4GsCm4H/+s9lpxqwPmw4CWgdPoGZNQNqOOcmmtk9mc3IzAYAAwAqV65MYmJiBIuPjO1JBmDJsuX8uSn35htvkpOTc/VziRe7dh3i2Wd/YebMrbRseSIPPXQ6CQlrSUxM3zgmkdB2mHNahzkX5DrMNqk75/4Aeh/HvDP6EWd3eKRZAvAScH0EMQwHhgO0bNnSdejQ4TjCydi8SSsBaHxGE6ibe/ONN4mJieTm5xIvxo//H3PmzObFFzvRtOl+zjsvou4qkglthzmndZhzQa7DSHq/v01YMg5xzg3I5qVJQPgdMqoDG8OGSwONgUQzA6gCTDCzbs65ednFlVt0RznJaykpacydu4E2bWrQvXsDVq0aTM2aZVQdiUiORZLJ/gt85//NBE4CIrlefS5Qz8xqm1lRvGp/Qmikc26nc66ic66Wc64WMBvI04Tu0b3fJe+sX7+Tjh3/Q/v27/Hbb9sBqFmzTMBRiUisiKT5/ePwYTP7APg2gtelmNltwGSgEDDCObfUzJ4A5jnnJmQ9h7yhSl3yyvjx/6N//wkcPJjKiBHdqV27XNAhiUiMOZ57v9cGTolkQufcJGBSuucezWTaDscRS46ZKnWJMuccd9zxNa+8MocWLaoyalRP6tWrEHRYIhKDIjmnvp0j59QTgG1ApjeSKXCcbhMr0WVmlClTnLvuOpt//esCihbVtiYi0ZFlUjevB1sTYIP/VJpz7qhOcwWZrlOXaHDO8e9/L6RWrbJ07Fibxx/vgN8hVEQkarLMZH4C/8w5l+r/xVRCB45U6mp+l1yyc+d++vYdxw03TODdd38CUEIXkTwRSXk6x8yaRz2SgKhSl9z0449JNGs2jE8+WcpTT3Xk/fcvCzokEYkjmTa/m1lh51wKcA5wk5n9CuzBu6mMc87FRKI3VeqSS+bP38g55/ybatVKM21aP9q2rZH9i0REclFW59TnAM2BGC811FFOciY1NY1ChRJo3rwqTz3VkQEDWlC2bPGgwxKROJRVm7MBOOd+zegvj+KLOl2nLjnxzTe/cvrpb7B27Q7MjPvua6eELiKByapSr2Rmd2U20jn3YhTiyXOGfnpVjt3Bg6k8/PD3PPfcLE4/vRL79qUEHZKISJZJvRBQiox/mCV2OP+n4VWpS4RWr95Onz5jmTNnAwMHtuDFFy+iZMkiQYclIpJlUt/knHsizyIJiDrKybF69tmZrFixlU8+uYJevRoFHY6IyGHZnlOPdWp+l0js2XOQNWt2APD8851YuHCgErqI5DtZVern51kUQVLzu2Rj4cLf6d37U4oVK8yCBQMoVaoopUoVDTosEZGjZJrJnHPb8jKQoByu1HVJm6TjnOPVV3+kdet32L37IEOHXkShQvryJyL51/H8SltM0SVtkpGdO/dz7bWfM2HCL3TtWp9//7s7FSuWDDosEZEsxX1Sh1Dzuyp1OaJEiSL8+edehg69iMGDW+ve7SJSIMR9UjenjnLiSUlJY+jQ2dxwQzPKlSvBtGnXq7ldRAoUJXX9oIsA69fv5G9/G8f06es44YQi3HLLWUroIlLgxH1SP9z7XR3l4tbnn/+P/v3Hc+hQGh980IOrrz4z6JBERI5L3Jcih3u/a1XEpTfemEuPHh9z6qnlWLBggBK6iBRocV+pH+79rko9rjjnMDMuu6wBGzfu5tFH21O0qLYBESnYVJ6i28TGE+cc7767gO7dR5OamsbJJ5fmySc7KqGLSEyI+6Su69Tjx86d++nTZyw33vgFe/YcYvfug0GHJCKSq9T8rko9Lvz4YxJ9+oxl3bqdPPVUR+6/v516t4tIzIn7pI4q9ZiXkpLG1Vd/RlqaY9q0frRtWyPokEREoiLuk/rhSl0d5WLO5s3JlC1bnGLFCvP551dRrdqJlC1bPOiwRESiRuWp7igXkyZPXsWZZ77Fww9/D8Dpp5+khC4iMS/uk7odvve77u0dCw4eTOXee7+hc+eRnHTSCfTr1yzokERE8oya312aqvQYsXr1dnr3/pS5czdyyy0teeGFTpQoUSTosERE8kzcJ3VwOp8eI/btO8SGDbsZO/ZKLr+8YdDhiIjkOTW/u1T1fC/AkpMP8u67CwDvvPnq1YOV0EUkbsV9NjOcmt8LqIULf6dly+HcdNMXLF68GYBixdT4JCLxK+6TOi5NlXoB45zjlVd+pHXrd9i9+yDffXctZ55ZOeiwREQCF/dljdf8rkq9ILn++vG8//4iunatz7//3Z2KFUsGHZKISL6gpK7m9wKne/fTaN68CoMHt8Z0KaKIyGFxn9TV/J7/paSk8Y9/TKVChZIMHtxaHeFERDIR99nMSNMlbfnY+vU7Oe+8//DEE9NYsuSPoMMREcnX4r5SN1Xq+dZnny3nhhsmcOhQGh980IOrrz4z6JBERPK1uE/qoDvK5UcrVmylZ88xNG9eldGje1G3bvmgQxIRyffiPqnrNrH5y7Zt+yhfvgT161fgyy/7cv75p1K0qD4fEZFIxH27s6Hm9/zAOcc77yygZs2XmDLlNwC6dKmnhC4icgzivlLHqaNc0Hbs2M/AgRMZM2YpF1xwKg0aVAw6JBGRAinuk7qRhhosgjN7dhJ9+oxl/fqd/Otf53Pffe1ISNC15yIix0NJ3aVBIVXqQZk9OwmAGTP6c/bZ1QOORkSkYFOJqjvK5bnff09m6tQ1AAwZ0prFi29WQhcRyQWq1PXTq3nq669Xce21n1GkSCFWrx5MsWKFKV26WNBhiYjEBGUzVep54uDBVO699xu6dBlJ5cql+Pbba/QzqSIiuSzuj6q6o1z0JScf5Lzz/sO8eRsZNKglzz/fiRIligQdlohIzFFSJ1WXtEVZqVJFadOmOv/3f+fQo4d+jEVEJFpUojo1v0dDcvJBBg78gqVLvR9heeWVLkroIiJRpkpdd5TLdQsX/k7v3p+yYsVWmjWryumnnxR0SCIicSGq2czMOpvZL2a2ysweyGD8XWa2zMwWm9l3ZnZKNOPJMEbd+z3XOOd45ZUfad36HXbvPsj331/HzTe3DDosEZG4EbWkbmaFgNeBLkAjoI+ZNUo32U9AS+fcmcCnwLPRiidzqtRzy7vv/sSQIV/TqVMdFi26mQ4dagUdkohIXIlm83srYJVzbjWAmY0GugPLQhM456aETT8buDqK8WRIlXrO7d17CIBrrjmTYsUKcfXVZ2KmW72KiOS1aJao1YD1YcNJ/nOZuQH4KorxZMjQD7ocr5SUNB555HvOOONNkpNTKFasMNdc00QJXUQkINGs1DM6srsMJzS7GmgJtM9k/ABgAEDlypVJTEzMpRChaeohtm7bwc+5OM94sHnzfp58cjlLluyic+fK7Nu3J1c/l3iUnJysdZhDWoc5p3WYc0Guw2gm9SSgRthwdWBj+onM7ALgIaC9c+5ARjNyzg0HhgO0bNnSdejQIdeC3L3cKFuxErk5z1g3btxybr55AqmpaYwceTl9+55BYmKi1mEOaR3mnNZhzmkd5lyQ6zCaSX0uUM/MagMbgN5A3/AJzKwZMAzo7Jz7I4qxZEp3lDs2zjneeGMudeuWZ/TontSpUz7okERExBe1pO6cSzGz24DJQCFghHNuqZk9Acxzzk0AngNKAZ/452HXOee6RSumjANVR7lILFu2hXLlilO1amk+/rgXpUsXo2hRrTcRkfwkqjefcc5NAiale+7RsMcXRHP5kVBHuaw553jnnQUMGfI13bqdxujRvahQoWTQYYmISAZ0RzmXhu6Wm7EdO/YzYMAXfPLJMi644FSGDu0cdEgiIpKFuE/q4FSpZ2Dp0j/o2nUUSUm7ePrp87n33nYkJOhSNRGR/Czuk7o6ymXs5JNLU6tWWUaN6snZZ1cPOhwREYlA3GczI1Ud5XybNu1myJCvOHgwlXLlSjBlynVK6CIiBUjcJ3X99Krn669X0aTJW7z99gIWLNgUdDgiInIc4j6px/tPrx48mMo993xDly4jqVKlFPPmDVB1LiJSQMX9OXVcfF/SdsMNE/jww8UMGtSS55/vRIkSRYIOSUREjlPcJ/V4rdRTUtIoXDiB++9vx+WXN6BHj4ZBhyQiIjmkpB5nd5RLTj7IrbdOwgzee+8yGjc+icaNTwo6LBERyQXxV6IeJX6S+oIFm2jefBgffriYU04pQ1pahj+aJyIiBZQq9Ti4Tt05x8sv/8j99/+XSpVK8v3319K+fa2gwxIRkVympB4Hlfrvvyfz+ONTueiiOowY0Z2KFXXvdhGRWBT3SZ0YrtR/+mkTTZtWoWrV0sydexN16pTD/zU8ERGJQbGZzY5BLP5KW0pKGo888j0tWgxnxIifAKhbt7wSuohIjIv7Sj3Wer+vW7eTvn3HMnPmevr1a8pVVzUOOiQREckjSuoxdJ36xIkruOaaz0hNTeOjjy6nT58zgg5JRETyUHwndedf0hUjlXrJkkU47bQKjBx5OXXqlA86HBERyWOxUaIeL5fq/S/AlfqyZVt46615AHTsWJsffrhBCV1EJE4V3GyWG9L8pF4AO8o55xg+fD4tWw7n8censmvXAQB1hhMRiWPxndRdmve/gDW/79ixn6uu+pSBAyfSrl1NfvppICeeWCzosEREJGBxfk694DW/HzyYSuvW77B69Xaefvp87r23HQkJqs5FRCTuk3rBqdSdc5gZRYsW4oEH2tGoUSVat9bvnouIyBEFp0SNhgJSqW/cuJsLL/yAceOWA9CvXzMldBEROUr+zmbRFuool48r9UmTVtKkyVvMmrWevXsPBR2OiIjkY/Gd1PGb3/Nh7/cDB1K4667JXHLJR5x8cmnmzx/A1VefGXRYIiKSj8V3Uk/Lv83vX321ipdems2tt57Fjz/eSMOGlYIOSURE8jl1lIN81fy+evV2Tj21HJdd1oC5c2+iZcuTgw5JREQKiPxXoualfNRRLjn5INdd9zmNG7/BypVbAZTQRUTkmMR5pZ4/OsotWLCJ3r0/5ddft/PII+dSu3a5QOMREZGCKc6TevAd5V5+eTb33vstJ510At9/fy3t29cKLBYRESnY4jup54OOcmvW7KBLl3qMGNGNChVKBhaHiIgUfPGd1APqKDdlym+ULFmE1q2r89xznShUyPRDLCJ57NChQyQlJbF///6gQ8lXypQpw/Lly4MOo0A73nVYvHhxqlevTpEiRY572XGe1PO2Uk9JSePxxxN56qnpXHRRXb766m8ULhx8Jz2ReJSUlETp0qWpVauWvlSH2b17N6VLlw46jALteNahc46tW7eSlJRE7dq1j3vZSuqQJ5X62rU76Nt3HLNmrad//6a88kqXqC9TRDK3f/9+JXTJN8yMChUqsGXLlhzNJ86Tet40vy9fvoW2bUeQmprGRx9dTp8+Z0R1eSISGSV0yU9yY3uM86SeN83v9etX4LrrmnD77a2oU6d8VJclIiLxK75P6EbxkralS/+gY8f/sGnTbgoVSmDo0M5K6CLyF4UKFaJp06Y0btyYSy+9lB07dhwet3TpUjp27Ej9+vWpV68e//jHP3DOHR7/1Vdf0bJlSxo2bEiDBg245557MlxGpNNFi3OOjh07smvXrjxd7rGYP38+Z5xxBnXr1mXw4MF/Wc8h27dvp0ePHpx55pm0atWKJUuWHB738ssv07hxY04//XSGDh16+PlHHnmEM888k6ZNm9KpUyc2btwIwMSJE/n73/8elfcS30k9Cpe0OecYNmweLVu+zdKlW1i7dmeuzVtEYkuJEiVYuHAhS5YsoXz58rz++usA7L3c3IYAABjxSURBVNu3j27duvHAAw+wYsUKFi1axKxZs3jjjTcAWLJkCbfddhsffvghy5cvZ8mSJZx66qlHzT/S6TKTmpqa4/c4adIkmjRpwoknnpinyz0Wt9xyC8OHD2flypWsXLmSr7/++qhp/vnPf9K0aVMWL17M+++/z5AhQwBvHb/99tvMmTOHRYsWMXHiRFatWgXAvffey+LFi1m4cCFdu3bliSeeAOCSSy5hwoQJ7N27N9ffi5rfIdfOqW/fvo+bbvqCsWOXc+GFp/L++z2oUqVUrsxbRKJoyh3wx8LcnedJTeG8odlP52vTpg2LFy8G4KOPPqJdu3Z06tQJgJIlS/Laa6/RoUMHbr31Vp599lkeeughGjRoAEDhwoUZNGjQUfPMarrrr7+erl270qtXLwBKlSpFcnIy06dP57nnnqNq1aosXLiQSy+9lFNOOeXw6x577DFKly7N3XffzXPPPceYMWM4cOAAPXr04PHHHz8qhpEjRzJgwIDDw5dddhnr169n//79DBky5PC4UqVKcddddzF58mReeOEFSpQowV133UVycjIVK1bkvffeo2rVqrz99tsMHz6cgwcPUrduXT744ANKljz+e3xs2rSJXbt20aZNGwCuvfZaPv/8c7p0+Wtn5mXLlvHggw8C0KBBA9asWcPmzZtZvnw5Z5999uEY2rdvz8SJE2nWrNlfvsjs2bPn8DlzM6NDhw5MnDiRK6+88rhjz0h8V+q53FHu4Ye/Z/z4X3jmmQv4+uurldBFJCKpqal89913dOvWDfCa3lu0aPGXaerUqUNycjK7du1iyZIlR43PSKTTpTdnzhyeeuopli1bRu/evfn4448PjxszZgxXXHEF33zzDStXrmTOnDksXLiQ+fPnM23atKPmNXPmzL/EMGLECObPn8+8efN45ZVX2LrV+62LPXv20LhxY3788Udat27N7bffzqeffsr8+fPp378/Dz30EACXX345c+fOZdGiRTRs2JB33333qGVOmTKFpk2bHvXXtm3bo6bdsGED1atXPzxcvXp1NmzYcNR0TZo0Ydy4cYfXz9q1a0lKSqJx48ZMmzaNrVu3snfvXiZNmkRSUtLh1z300EPUqFGDkSNHHq7UAVq2bMn06dMz/xCOkyp1yFHze2pqGtu376dixZI8+WRHrruuKa1aVculAEUkTxxDRZ2b9u37/+3deXxU5bnA8d+TBUJIWIqNIqkkFsuFhCwYYrReZJNVxArFaAAjiyJqhF5ooVr0VsFqcbkINVUupogVahThI2jdklaUWBE0ErYIpRipErY4gURCePvHORmzM2EyyWTm+X4+8/nMnPOec555GPLMec+Z9y0jISGBAwcOcPnll3PttdcC1mW8hu6Ebok79pOTk52/lU5MTOTw4cMcOnSI4uJiunbtyiWXXMLSpUt56623SExMBKC0tJTCwkIGDhxYY1/Hjh2r8ZvtpUuXsm7dOgC+/PJLCgsL6datG4GBgYwfPx6APXv2sGPHDmc+Kisr6d69O2B9Ubn//vs5ceIEpaWljBgxok78gwcP5tNPXet5qe/6eX05nj9/Pvfeey8JCQn069ePxMREgoKC6NOnD7/61a+49tprCQsLIz4+nsDA708UFy1axKJFi3jkkUdYtmyZszcjIiLCeY29Ofl5UXfvRrlDhxxMnryOkpJytmyZRteuHbSgK6VcVnVNvaSkhOuuu47ly5eTkZFBTExMnbPe/fv3ExYWRnh4ODExMXzyySfEx8c3uv/G2gUFBXH2rPU30BjD6dOnnes6duxYo+2ECRPIzs7m66+/JjU11bnNggULuOOOOxqNoeo4AQEB5Obm8s4777BlyxZCQ0MZNGiQc0S/kJAQZzE0xhATE8OWLVvq7C89PZ3XXnuN+Ph4srKyyM3NrdMmJyeHOXPm1FkeGhrKhx9+WGNZZGRkjTProqIiLr647gyZnTp14vnnn3fGFx0d7fziM23aNKZNmwbAr3/9ay644II6299yyy2MGTPGWdTLy8vp0KFD3YS5yb+736tulDuPNGzcuJf4+Ezy8oqYNWuAjgynlDpvnTt3ZunSpSxZsoSKigrS0tLYvHkz77zzDmCd0WdkZPDLX/4SsG7AWrx4MXv37gXg7NmzPPHEE3X221i7qKgoPvnkEwDWr19PRUVFg/GlpqayZs0asrOzndfgR4wYwcqVKyktLQWsbuzDhw/X2bZ3797s378fgJKSErp27UpoaCi7d+8mLy+v3uP17t2b4uJiZ1GvqKigoKAAsEZr6969OxUVFbz44ov1bl91pl77UbugA3Tv3p3w8HDy8vIwxrBq1SrGjRtXp92JEyecX3xWrFjBwIEDndfMq973wYMHefXVV505KiwsdG6/YcMG570NAHv37iU2Nrbe+N3h52fqdlFvwpn6d9+dYcGCd3nyyTzi4i5kzZrx9OnzQw8FqJTyF4mJicTHx7NmzRomT57M+vXrueeee7jrrruorKxk8uTJ3H333QDExcXx1FNPcfPNN3Pq1ClEhDFjxtTZZ2PtZsyYwbhx40hOTmbo0KF1zs6ri4mJweFw0KNHD2c3+PDhw9m1a5fzBrOwsDBWr15NREREjW3HjBlDbm4uvXr1YuTIkWRmZhIXF0fv3r1JSUmp93jt2rUjOzubjIwMSkpKOHPmDLNnzyYmJoaHHnqIK664gp49e9KvXz8cDkfTk13LM888Q3p6OmVlZYwaNcp5k1xmZiYAM2fOZNeuXUyZMoXAwED69u1b41r++PHjOXr0KMHBwSxfvpyuXa3ps+fPn8+ePXsICAigZ8+ezv2B1ZvwyCOPuB17HcaYNvW4/PLLTbPZt9GYJRhz6COXNzl58rSJiVlu7r57oykrq2i+WNq4nJyc1g6hzdMcuq8pOdy5c6fnAmnDvv3222bd36FDh8ywYcOadZ/e7lw5/Prrr82QIUPqXVff5xLYalyskXqmDi7dKJedvZORI3sRFtaOjz6aTseO7TwcnFJKtX3du3dnxowZfPvtt036rbovO3jwII8//rhH9u3fF4Jd+Embw/EdU6as4+c/f5lly/4BoAVdKaWaYOLEiVrQqxkwYAAJCQke2beeqUODZ+rbtv2b1NRs9u07zoMPXsO8eXV/46iUUkp5C/8u6mcbvlHu5ZcLSEt7lYiIjuTk3MrAgT1bODillFKqafy7qDfS/T5gQA9uuimWp54aQbdu5z8EoVJKKdVS/Pyaes3u9/fe+yfTp2/AGENUVBdeeOFnWtCVUh7T2Cxt7jhw4IBHfgMNsHr1auLi4oiJiSE+Pp7p06c3W9zKfR4t6iIyUkT2iMgXIjK/nvXtRWStvf4jEYnyZDx12GfqFWeE++57l2HDVvHBB19SXNz8M+copVRtDc3S5q3efPNNnnzySd544w0KCgrYtm0bV111Fd98801rh6ZsHivqIhIILAdGAX2Bm0Wkb61m04DjxphewJPAo56Kp16mkgPHunDNde+wePFmpk5NZOvWGURENDwIg1JKecKVV17pnEiktLSUoUOH0r9/f/r168f69esB6wy8T58+zJgxg5iYGIYPH05ZWRmAczjYK6+8ssaXg/Lycm677TbneOU5OTkAZGVlccMNNzB27Fiio6NZtmwZTzzxBFdffTUpKSkcO3asToyLFi1iyZIl9OhhDYcdGBjI1KlT6d27N2CNUnfkyBEAtm7dyqBBgwBrspapU6cyYMAAEhMTne+noKCA5ORkEhISiIuLo7CwkJMnTzJmzBji4+OJjY2tMZmMOjdPXlNPBr4wxuwHEJE1wDhgZ7U244AH7efZwDIREfvH9h539swZRq9Io6jsBC+9NJ7UVM90VymlvN+gQVl1lk2cGMOsWQM4daqC0aPrDkmanp5AenoCR46cYsKEv9RYl5ub7vKxq2Zpqxo/PCQkhHXr1tGpUyeOHDlCSkqKcwa3wsJCXnrpJZ577jkmTpzIK6+8wqRJk7jtttt4+umnueaaa5g3b55z31UF/vPPP2f37t0MHz7cOWzsjh072L59O+Xl5fTq1YtHH32UzZs3s3DhQlatWsXs2bNrxFlQUED//v1dfl9VFi1axJAhQ1i5ciUnTpwgOTmZYcOGkZmZyb333ktaWhqnT5+msrKSTZs2cfHFF7Nx40bAGlpWuc6TRb0H8GW110XAFQ21McacEZESoBtwpHojEbkduB3gwgsvrHcA//Pxg5Kj/H7KLk72+28iLjrSbPv1R6WlpZo/N2kO3deUHHbu3LnGEKOVlZV12pSXl+NwODh1qqLR9aWlZXXWuzJ8aVlZGXFxcRw8eJCEhARSUlJwOBxUVFQwf/58PvzwQwICAvjqq6/Yt28f5eXl9OzZkx//+Mc4HA5iY2PZs2cPRUVFHD9+nP79++NwOLjxxhvZuHEjDoeD3Nxc7rjjDucwr5GRkc5CfvXVVwPWl4hOnToxePBgKisrueyyy9ixY0ed92CMweFwEBAQQEFBAbfffjsOh4MHHniA8ePHY4yhtLSU9u3bc/LkSSorK3E4HLz55pu89tprPPbYY873vWvXLhISEnj44YfZt28fY8eOpVevXkRHR/P2228zZ84cRo4cyVVXXdUsQ8G2pKr3fT7Ky8vd+jvgyaJe3/yAtc/AXWmDMeZZ4FmApKQkU9Wl475BHOuczJhm25//ys3Npfn+XfyT5tB9Tcnhrl27akwJ+v770xpsGx5+rvXhja5vSIcOHcjPz3fO0rZq1SoyMjLIysqipKSE7du3ExwcTFRUFEFBQYSFhdGhQwdn3KGhoZSWlhIWFkZAQIBzeceOHZ2vAwMDCQ0Nda4LDAykY8eOhISEOGd9q1peNQVqaGhojf1ViY2NpbCwkMGDB5OSkkJ+fr5zPPrw8HCCg4OdxwoMDCQwMJDw8HBEhHXr1jm76askJSUxaNAgNm7cyPjx41mxYgVDhgxh27ZtbNq0iYceeojhw4ezcOHCJue2NTkcjjq5c1VISIhzOtvz4ckb5YqAH1V7HQnUnjzW2UZEgoDOQN0LOUop5cNqz9JWUlJCREQEwcHB5OTk8K9//avR7bt06ULnzp3ZvHkzQI3ZywYOHOh8vXfvXg4ePFinuLpqwYIFzJ07t8ZUpVXX9KHmzG+vvPKKc/mIESN4+umnnXOXb9++HbCmk7300kvJyMjg+uuvJz8/n0OHDhEaGsqkSZOYO3cu27ZtO69Y/ZUnz9Q/Bi4TkWjgKyAVuKVWmw3ArcAWYALwXktdT1dKKW9SfZa2tLQ0xo4dS1JSEgkJCTWm7GzI888/z9SpUwkNDWXEiBHO5bNmzWLmzJn069ePoKAgsrKyaN++/XnFOHr0aIqLixk1ahSVlZV06dKF2NhY5/EeeOABpk2bxuLFi7niiu+vtv7mN79h9uzZxMXF2T8ZjuL1119n7dq1rF69muDgYC666CIWLlzIxx9/zLx58wgICCA4OJhnnnnmvGL1V+LJGioio4GngEBgpTFmkYj8FmvGmQ0iEgK8ACRinaGnVt1Y15CkpCSzdevWZotRuzybh+bRfZpD9zW1+71Pnz6eDagNcqfrWFncyWF9n0sR+cQYk+TK9h4dUc4YswnYVGvZwmrPy4GfezIGpZRSyl/494hySimllA/Roq6UUkr5CC3qSim/pfflKm/SHJ9HLepKKb8UEhLC0aNHtbArr2CM4ejRo4SEhLi1H/+eelUp5bciIyMpKiqiuLi4tUPxKuXl5W4XFn93vjkMCQkhMjLSrWNrUVdK+aXg4GCio6NbOwyvk5ub69aIZqp1c6jd70oppZSP0KKulFJK+Qgt6koppZSP8OgwsZ4gIsVA47MbNM0F1JrqVZ0XzaP7NIfu0xy6T3PovubOYU9jzA9dadjminpzE5Gtro6pqxqmeXSf5tB9mkP3aQ7d15o51O53pZRSykdoUVdKKaV8hBZ1eLa1A/ARmkf3aQ7dpzl0n+bQfa2WQ7+/pq6UUkr5Cj1TV0oppXyE3xR1ERkpIntE5AsRmV/P+vYistZe/5GIRLV8lN7NhRz+QkR2iki+iLwrIj1bI05vdq4cVms3QUSMiOhdyPVwJY8iMtH+PBaIyJ9bOkZv58L/50tEJEdEttv/p0e3RpzeSkRWishhEdnRwHoRkaV2fvNFpH+LBGaM8fkHEAjsAy4F2gGfAX1rtZkFZNrPU4G1rR23Nz1czOFgINR+fqfmsOk5tNuFA38H8oCk1o7b2x4ufhYvA7YDXe3XEa0dtzc9XMzhs8Cd9vO+wIHWjtubHsBAoD+wo4H1o4E3AAFSgI9aIi5/OVNPBr4wxuw3xpwG1gDjarUZB/zJfp4NDBURacEYvd05c2iMyTHGnLJf5gHuTTfke1z5HAI8BDwGlLdkcG2IK3mcASw3xhwHMMYcbuEYvZ0rOTRAJ/t5Z+BQC8bn9YwxfweONdJkHLDKWPKALiLS3dNx+UtR7wF8We11kb2s3jbGmDNACdCtRaJrG1zJYXXTsL6lqu+dM4cikgj8yBjzeksG1sa48ln8CfATEflARPJEZGSLRdc2uJLDB4FJIlIEbALuaZnQfEZT/2Y2C3+ZerW+M+7at/270safuZwfEZkEJAHXeDSitqfRHIpIAPAkkN5SAbVRrnwWg7C64Adh9Ri9LyKxxpgTHo6trXAlhzcDWcaYx0XkSuAFO4dnPR+eT2iVmuIvZ+pFwI+qvY6kbleSs42IBGF1NzXWteJvXMkhIjIMuA+43hjzXQvF1lacK4fhQCyQKyIHsK7DbdCb5epw9f/zemNMhTHmn8AerCKvLK7kcBrwFwBjzBYgBGtMc+Ual/5mNjd/KeofA5eJSLSItMO6EW5DrTYbgFvt5xOA94x9t4MCXMih3XX8R6yCrtcw62o0h8aYEmPMBcaYKGNMFNZ9CdcbY7a2Trhey5X/z69h3biJiFyA1R2/v0Wj9G6u5PAgMBRARPpgFfXiFo2ybdsATLHvgk8BSowx//b0Qf2i+90Yc0ZE7gb+inXX50pjTIGI/BbYaozZAPw/VvfSF1hn6KmtF7H3cTGHvwfCgJftewwPGmOub7WgvYyLOVTn4GIe/woMF5GdQCUwzxhztPWi9i4u5vB/gOdEZA5Wt3G6nuh8T0Rewrq8c4F938EDQDCAMSYT6z6E0cAXwCngthaJS/+NlFJKKd/gL93vSimllM/Toq6UUkr5CC3qSimllI/Qoq6UUkr5CC3qSimllI/Qoq5UCxORShH5tNojqpG2UQ3NAtXEY+baM3J9Zg+d2vs89jFTRKbYz9NF5OJq61aISN9mjvNjEUlwYZvZIhLq7rGV8gVa1JVqeWXGmIRqjwMtdNw0Y0w81sRFv2/qxsaYTGPMKvtlOnBxtXXTjTE7myXK7+P8A67FORvQoq4UWtSV8gr2Gfn7IrLNflxVT5sYEfmHfXafLyKX2csnVVv+RxEJPMfh/g70srcdas+X/bk9P3R7e/nv7LnI80Vkib3sQRGZKyITsMb2f9E+Zgf7DDtJRO4UkceqxZwuIk+fZ5xbqDYBhog8IyJbxZof/X/tZRlYXy5yRCTHXjZcRLbYeXxZRMLOcRylfIYWdaVaXodqXe/r7GWHgWuNMf2Bm4Cl9Ww3E/g/Y0wCVlEtsofvvAn4qb28Ekg7x/HHAp+LSAiQBdxkjOmHNcLknSLyA+BnQIwxJg54uPrGxphsYCvWGXWCMaas2ups4MZqr28C1p5nnCOxhnutcp8xJgmIA64RkThjzFKs8bQHG2MG20PC3g8Ms3O5FfjFOY6jlM/wi2FilfIyZXZhqy4YWGZfQ67EGqu8ti3AfSISCbxqjCkUkaHA5cDH9tC8HbC+INTnRREpAw5gTaPZG/inMWavvf5PwF3AMqy53FeIyEbA5WlgjTHFIrLfHuu60D7GB/Z+mxJnR6zhS/tXWz5RRG7H+rvVHegL5NfaNsVe/oF9nHZYeVPKL2hRV8o7zAG+AeKxetDKazcwxvxZRD4CxgB/FZHpWNM7/skYs8CFY6RVnxxGRLrV18geFzwZazKPVOBuYEgT3staYCKwG1hnjDFiVViX4wQ+A34HLAduFJFoYC4wwBhzXESysCYYqU2At40xNzchXqV8hna/K+UdOgP/tueqnox1llqDiFwK7Le7nDdgdUO/C0wQkQi7zQ9EpKeLx9wNRIlIL/v1ZOBv9jXozsaYTVg3odV3B7oDa6rY+rwK3IA1H/dae1mT4jTGVGB1o6fYXfedgJNAiYhcCIxqIJY84KdV70lEQkWkvl4PpXySFnWlvMMfgFtFJA+r6/1kPW1uAnaIyKfAfwGr7DvO7wfeEpF84G2srulzMsaUY80c9bKIfA6cBTKxCuTr9v7+htWLUFsWkFl1o1yt/R4HdgI9jTH/sJc1OU77Wv3jwFxjzGfAdqAAWInVpV/lWeANEckxxhRj3Zn/kn2cPKxcKeUXdJY2pZRSykfombpSSinlI7SoK6WUUj5Ci7pSSinlI7SoK6WUUj5Ci7pSSinlI7SoK6WUUj5Ci7pSSinlI7SoK6WUUj7iPyghz1mQhCJIAAAAAElFTkSuQmCC\n"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 40,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "confusion_matrix(y_true, y_pred)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 41,
          "data": {
            "text/plain": [
              "array([[1481,   11],\n",
              "       [  10, 1450]])"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 41,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Our model performed very well, with an F1 Score of 99.38%, Accuracy of 99.39%, and an ROC Score of ~99.4%. \n",
        "\n",
        "Before we jump to our feet and run to the sports betting websites, we have to realize that the features in our dataset contain data from the games that we are trying to predict. In other words, we would have to wait through the entire game to collect and aggreagate the data from the game before cleaning it and feeding it to our dense, fully connected network. \n",
        "\nUnless we could do this very efficiently (instantly..) there is little use for this program in terms of actual sports betting. Still, it was a good way to practice using PyTorch to construct a feedforward neural netowrk and perform simple regularization techniques to improve model performance."
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "nteract": {
      "version": "0.12.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}