lesson03/titanic/titanic3/titanic3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Titanic Kaggle Competition: Fast.AI Lesson 3 Excel in Python\n",
    "\n",
    "In this notebook I implement the Excel-based model from [video lecture 3](https://www.youtube.com/watch?v=hBBOjCiFcuo&list=PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU&index=3&t=3862s) to predict the [Titanic](https://www.kaggle.com/c/titanic) survivors in python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the dataset\n",
    "\n",
    "Uncomment the lines below if you need to dataset.\n",
    "\n",
    "For any advice on how to install kaggle, please refer to this blog post LINK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!kaggle competitions download -c titanic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!unzip titanic.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the training data\n",
    "\n",
    "Let's load the training data and have quick look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train = pd.read_csv(\"train.csv\")\n",
    "train_save = train\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',\n",
       "       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing the model\n",
    "\n",
    "Now I am going to follow the same steps as shown in the [video lecture](https://www.youtube.com/watch?v=hBBOjCiFcuo&list=PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU&index=3&t=3862s) to predict the survivors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Remove lines with incomplete data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training set has 891 lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "891"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon quick assessment, there are lines which contain null values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Williams, Mr. Charles Eugene</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>244373</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Masselmani, Mrs. Fatima</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2649</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Emir, Mr. Farred Chehab</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2631</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>O'Dwyer, Miss. Ellen \"Nellie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330959</td>\n",
       "      <td>7.8792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>859</th>\n",
       "      <td>860</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Razi, Mr. Raihed</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2629</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>863</th>\n",
       "      <td>864</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Sage, Miss. Dorothy Edith \"Dolly\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>69.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>868</th>\n",
       "      <td>869</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>van Melkebeke, Mr. Philemon</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>345777</td>\n",
       "      <td>9.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>878</th>\n",
       "      <td>879</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Laleff, Mr. Kristo</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349217</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>177 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass                                      Name  \\\n",
       "5              6         0       3                          Moran, Mr. James   \n",
       "17            18         1       2              Williams, Mr. Charles Eugene   \n",
       "19            20         1       3                   Masselmani, Mrs. Fatima   \n",
       "26            27         0       3                   Emir, Mr. Farred Chehab   \n",
       "28            29         1       3             O'Dwyer, Miss. Ellen \"Nellie\"   \n",
       "..           ...       ...     ...                                       ...   \n",
       "859          860         0       3                          Razi, Mr. Raihed   \n",
       "863          864         0       3         Sage, Miss. Dorothy Edith \"Dolly\"   \n",
       "868          869         0       3               van Melkebeke, Mr. Philemon   \n",
       "878          879         0       3                        Laleff, Mr. Kristo   \n",
       "888          889         0       3  Johnston, Miss. Catherine Helen \"Carrie\"   \n",
       "\n",
       "        Sex  Age  SibSp  Parch      Ticket     Fare Cabin Embarked  \n",
       "5      male  NaN      0      0      330877   8.4583   NaN        Q  \n",
       "17     male  NaN      0      0      244373  13.0000   NaN        S  \n",
       "19   female  NaN      0      0        2649   7.2250   NaN        C  \n",
       "26     male  NaN      0      0        2631   7.2250   NaN        C  \n",
       "28   female  NaN      0      0      330959   7.8792   NaN        Q  \n",
       "..      ...  ...    ...    ...         ...      ...   ...      ...  \n",
       "859    male  NaN      0      0        2629   7.2292   NaN        C  \n",
       "863  female  NaN      8      2    CA. 2343  69.5500   NaN        S  \n",
       "868    male  NaN      0      0      345777   9.5000   NaN        S  \n",
       "878    male  NaN      0      0      349217   7.8958   NaN        S  \n",
       "888  female  NaN      1      2  W./C. 6607  23.4500   NaN        S  \n",
       "\n",
       "[177 rows x 12 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[train['Age'].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the actual lines which contain null values in columns that we care about:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Column Pclass contains 0 null values\n",
      "Column Sex contains 0 null values\n",
      "Column Age contains 177 null values\n",
      "Column SibSp contains 0 null values\n",
      "Column Parch contains 0 null values\n",
      "Column Fare contains 0 null values\n",
      "Column Embarked contains 2 null values\n"
     ]
    }
   ],
   "source": [
    "relevant_columns = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']\n",
    "\n",
    "for col in relevant_columns:\n",
    "    print(f\"Column {col} contains {len(train[train[col].isnull()])} null values\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the time being, let's just get rid of them as suggested by Jeremy [in the video](https://youtu.be/hBBOjCiFcuo?t=4013):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Column Pclass contains 0 null values\n",
      "Column Sex contains 0 null values\n",
      "Column Age contains 0 null values\n",
      "Column SibSp contains 0 null values\n",
      "Column Parch contains 0 null values\n",
      "Column Fare contains 0 null values\n",
      "Column Embarked contains 0 null values\n"
     ]
    }
   ],
   "source": [
    "for col in relevant_columns:\n",
    "    train = train[train[col].notnull()]\n",
    "\n",
    "for col in relevant_columns:\n",
    "    print(f\"Column {col} contains {len(train[train[col].isnull()])} null values\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "712"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, we have 712 lines remaining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_save = train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Feature Engineering\n",
    "\n",
    "The next step is to turn the columns into better features that we can for the predictions.\n",
    "\n",
    "For example, we cannot work with a column called male, so we need to turn it into a number. For categorical values,\n",
    "* we create a new column for each value\n",
    "* put in a `1` if this new column if the value of the source column matches the \"column header\", else we put a `0`.\n",
    "\n",
    "For continuous values, we squish the values into values between 0 and 1 (or close to that)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore how to basically engineer a feature real quick before implementing a general algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['male', 'female'], dtype=object)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sex_values = train.Sex.unique()\n",
    "sex_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sex_male', 'Sex_female']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_col_names = [ \"Sex\" + \"_\" + sex_value for sex_value in sex_values]\n",
    "new_col_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Sex_male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  Sex_male  \n",
       "0      0         A/5 21171   7.2500   NaN        S         1  \n",
       "1      0          PC 17599  71.2833   C85        C         0  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S         0  \n",
       "3      0            113803  53.1000  C123        S         0  \n",
       "4      0            373450   8.0500   NaN        S         1  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['Sex_male'] = [1 if sex == \"male\" else 0 for sex in train['Sex'] ]\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, we have created a new column which indicates if a passenger was a male.\n",
    "\n",
    "Now let's do this in a more general way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_cols = ['Pclass', 'Sex', 'SibSp', 'Parch', 'Embarked']\n",
    "\n",
    "for cat_col in cat_cols:\n",
    "    col_values = train[cat_col].unique()\n",
    "    new_col_names = [ str(cat_col) + \"_\" + str(col_value) for col_value in col_values]\n",
    "    #print(new_col_names)\n",
    "    for new_col_name in new_col_names:\n",
    "        col_name, col_val = new_col_name.split('_')\n",
    "        #print (col, val)\n",
    "        train[new_col_name] = [1 if value == col_val else 0 for value in train[cat_col] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Sex_female</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>female</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sex  Sex_male  Sex_female\n",
       "0     male         1           0\n",
       "1   female         0           1\n",
       "2   female         0           1\n",
       "3   female         0           1\n",
       "4     male         1           0\n",
       "6     male         1           0\n",
       "7     male         1           0\n",
       "8   female         0           1\n",
       "9   female         0           1\n",
       "10  female         0           1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['Sex', 'Sex_male', 'Sex_female']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous variables\n",
    "\n",
    "For continuous variables, we squish the values into values between 0 and 1 (or close to that), i.e. we normalize the data:\n",
    "\n",
    "_\"Normalization is scaling the data to be analyzed to a specific range such as [0.0, 1.0] to provide better results.\"_ [Source](https://www.geeksforgeeks.org/what-is-data-normalization/)\n",
    "\n",
    "Depending on the distribution, we can use different algorithm. For the age, we are dividing by the maximum age, and for the fare, we use the log10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Log_Age</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Log_Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>0.860338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>0.4750</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>1.852988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>0.3250</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>0.898999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>1.725095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>0.905796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>54.0</td>\n",
       "      <td>0.6750</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>1.714853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0250</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>1.323768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>27.0</td>\n",
       "      <td>0.3375</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>1.046624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>14.0</td>\n",
       "      <td>0.1750</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>1.478145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0500</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>1.222716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Age  Log_Age     Fare  Log_Fare\n",
       "0   22.0   0.2750   7.2500  0.860338\n",
       "1   38.0   0.4750  71.2833  1.852988\n",
       "2   26.0   0.3250   7.9250  0.898999\n",
       "3   35.0   0.4375  53.1000  1.725095\n",
       "4   35.0   0.4375   8.0500  0.905796\n",
       "6   54.0   0.6750  51.8625  1.714853\n",
       "7    2.0   0.0250  21.0750  1.323768\n",
       "8   27.0   0.3375  11.1333  1.046624\n",
       "9   14.0   0.1750  30.0708  1.478145\n",
       "10   4.0   0.0500  16.7000  1.222716"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "max_age = train.Age.max()\n",
    "train['Log_Age'] = [age/max_age for age in train.Age] \n",
    "train['Log_Fare'] = [0 if fare == 0 else math.log10(float(fare)) for fare in train.Fare] #preventing \"underflow\"\n",
    "\n",
    "train[['Age', 'Log_Age', 'Fare', 'Log_Fare']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a colum of ones\n",
    "\n",
    "Finally, we add a colum of ones, which will create the bias in our calculation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train['Ones'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ones</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Ones\n",
       "0     1\n",
       "1     1\n",
       "2     1\n",
       "3     1\n",
       "4     1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['Ones']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we have these columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',\n",
       "       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked', 'Sex_male', 'Pclass_3',\n",
       "       'Pclass_1', 'Pclass_2', 'Sex_female', 'SibSp_1', 'SibSp_0', 'SibSp_3',\n",
       "       'SibSp_4', 'SibSp_2', 'SibSp_5', 'Parch_0', 'Parch_1', 'Parch_2',\n",
       "       'Parch_5', 'Parch_3', 'Parch_4', 'Parch_6', 'Embarked_S', 'Embarked_C',\n",
       "       'Embarked_Q', 'Log_Age', 'Log_Fare', 'Ones'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify things, let's create a new dataframe which just contains the columns that we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "train2 = train[['Sex_male', 'Sex_female', \n",
    "                'Pclass_1', 'Pclass_2', #'Pclass_3',\n",
    "                'SibSp_0', 'SibSp_1', 'SibSp_2', 'SibSp_3', 'SibSp_4', #'SibSp_5',\n",
    "                'Parch_0', 'Parch_1', 'Parch_2', 'Parch_3', 'Parch_4', 'Parch_5', #'Parch_6',\n",
    "                'Embarked_C', 'Embarked_Q', #'Embarked_S',\n",
    "                'Log_Age', 'Log_Fare', 'Ones']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have 20 relevant columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train2.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reuse, let's wrap all the feature engineering in a function. We will need it for the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_data(train):\n",
    "    for cat_col in cat_cols:\n",
    "        col_values = train[cat_col].unique()\n",
    "        new_col_names = [ str(cat_col) + \"_\" + str(col_value) for col_value in col_values]\n",
    "        #print(new_col_names)\n",
    "        for new_col_name in new_col_names:\n",
    "            col_name, col_val = new_col_name.split('_')\n",
    "            #print (col, val)\n",
    "            train[new_col_name] = [1 if value == col_val else 0 for value in train[cat_col] ]\n",
    "        \n",
    "    max_age = train.Age.max()\n",
    "    train['Log_Age'] = [age/max_age for age in train.Age] \n",
    "    train['Log_Fare'] = [0 if fare == 0 else math.log10(float(fare)) for fare in train.Fare] #preventing \"underflow\"\n",
    "    \n",
    "    train['Ones'] = 1\n",
    "    \n",
    "    return train[['Sex_male', 'Sex_female', \n",
    "                'Pclass_1', 'Pclass_2', #'Pclass_3',\n",
    "                'SibSp_0', 'SibSp_1', 'SibSp_2', 'SibSp_3', 'SibSp_4', #'SibSp_5',\n",
    "                'Parch_0', 'Parch_1', 'Parch_2', 'Parch_3', 'Parch_4', 'Parch_5', #'Parch_6',\n",
    "                'Embarked_C', 'Embarked_Q', #'Embarked_S',\n",
    "                'Log_Age', 'Log_Fare', 'Ones']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train3 = convert_data(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making sure that the 2 ways of doing feature engineering have the same result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train2.equals(train3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Make Predictions\n",
    "\n",
    "Let's initialize our parameters (weights) with a random number between 0 and 1 and subtract 0.5 to have values between -0,5 and 0,5.\n",
    "\n",
    "Let's use a tensor with 20 elements, so that we can do a nice matrix multiplication later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([712, 20])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "train_tensor = torch.tensor(train2.values)\n",
    "train_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1., dtype=torch.float64)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_tensor[0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.3775, -0.2325, -0.4063,  0.1706, -0.3362,  0.4272, -0.2380, -0.0033,\n",
       "        -0.2370, -0.3825, -0.3306, -0.2900, -0.0110, -0.4436, -0.0240,  0.2606,\n",
       "         0.2759,  0.0243, -0.1286, -0.4574], dtype=torch.float64,\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(27) #https://pytorch.org/docs/stable/notes/randomness.html\n",
    "params = torch.rand(20, dtype=torch.float64)-0.5 #https://pytorch.org/docs/stable/generated/torch.rand.html\n",
    "params.requires_grad_()\n",
    "params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.3775, dtype=torch.float64, grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the predictions by matrix multiplying the training values by the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([712])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = train_tensor@params\n",
    "preds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the loss, we subtract the actual value from the prediction. Square the result and take the mean of all values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.1453, dtype=torch.float64, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "survived_tensor = torch.tensor(train.Survived.values)\n",
    "\n",
    "loss = ((preds-survived_tensor)**2).mean()\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to implement the optimizer with the usual steps:\n",
    "* Forward pass, i.e. calculation of the predictions via matrix multiplication\n",
    "* Calculate the loss\n",
    "* Initialize the gradients\n",
    "* Calculate the gradients (backpropagation)\n",
    "* Update the parameters with the learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_optimizer_step(params, lr=0.01):\n",
    "    preds = train_tensor@params\n",
    "    loss = ((preds-survived_tensor)**2).mean()\n",
    "    \n",
    "    params.grad = None #Initializing gradients (.zero_grad())\n",
    "    loss.backward() #Calculate gradients\n",
    "    with torch.no_grad(): params -= params.grad*lr #update parameters\n",
    "    \n",
    "    return loss    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run that a few times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 1.15\n",
      "Loss: 0.48\n",
      "Loss: 0.38\n",
      "Loss: 0.34\n",
      "Loss: 0.31\n",
      "Loss: 0.28\n",
      "Loss: 0.26\n",
      "Loss: 0.25\n",
      "Loss: 0.23\n",
      "Loss: 0.22\n",
      "Loss: 0.21\n",
      "Loss: 0.20\n",
      "Loss: 0.20\n",
      "Loss: 0.19\n",
      "Loss: 0.19\n",
      "Loss: 0.19\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n"
     ]
    }
   ],
   "source": [
    "for i in range(25):\n",
    "    loss = run_optimizer_step(params, lr=0.1)\n",
    "    print(f\"Loss: {loss:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result we have a set of parameters we can use to make predictions, i.e. we have trained a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Applying Learning on the Test Set\n",
    "\n",
    "Let's have a look at the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  \n",
       "0  34.5      0      0   330911   7.8292   NaN        Q  \n",
       "1  47.0      1      0   363272   7.0000   NaN        S  \n",
       "2  62.0      0      0   240276   9.6875   NaN        Q  \n",
       "3  27.0      0      0   315154   8.6625   NaN        S  \n",
       "4  22.0      1      1  3101298  12.2875   NaN        S  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = pd.read_csv(\"test.csv\")\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a line in which the fare value in NaN (null). Let's select and fix that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>1044</td>\n",
       "      <td>3</td>\n",
       "      <td>Storey, Mr. Thomas</td>\n",
       "      <td>male</td>\n",
       "      <td>60.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3701</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Pclass                Name   Sex   Age  SibSp  Parch Ticket  \\\n",
       "152         1044       3  Storey, Mr. Thomas  male  60.5      0      0   3701   \n",
       "\n",
       "     Fare Cabin Embarked  \n",
       "152   NaN   NaN        S  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select as suggested by https://datatofish.com/rows-with-nan-pandas-dataframe/\n",
    "test[test['Fare'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['Fare'] = test['Fare'].fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, there is no more null value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [PassengerId, Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, Embarked]\n",
       "Index: []"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test[test['Fare'].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the parameters the model has learned, we need to transform the test data the same way, i.e. we need to engineer the features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Pclass_1</th>\n",
       "      <th>Pclass_2</th>\n",
       "      <th>SibSp_0</th>\n",
       "      <th>SibSp_1</th>\n",
       "      <th>SibSp_2</th>\n",
       "      <th>SibSp_3</th>\n",
       "      <th>SibSp_4</th>\n",
       "      <th>Parch_0</th>\n",
       "      <th>Parch_1</th>\n",
       "      <th>Parch_2</th>\n",
       "      <th>Parch_3</th>\n",
       "      <th>Parch_4</th>\n",
       "      <th>Parch_5</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Log_Age</th>\n",
       "      <th>Log_Fare</th>\n",
       "      <th>Ones</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.453947</td>\n",
       "      <td>0.893717</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.618421</td>\n",
       "      <td>0.845098</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.815789</td>\n",
       "      <td>0.986212</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.937643</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.289474</td>\n",
       "      <td>1.089464</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sex_male  Sex_female  Pclass_1  Pclass_2  SibSp_0  SibSp_1  SibSp_2  \\\n",
       "0         1           0         0         0        0        0        0   \n",
       "1         0           1         0         0        0        0        0   \n",
       "2         1           0         0         0        0        0        0   \n",
       "3         1           0         0         0        0        0        0   \n",
       "4         0           1         0         0        0        0        0   \n",
       "\n",
       "   SibSp_3  SibSp_4  Parch_0  Parch_1  Parch_2  Parch_3  Parch_4  Parch_5  \\\n",
       "0        0        0        0        0        0        0        0        0   \n",
       "1        0        0        0        0        0        0        0        0   \n",
       "2        0        0        0        0        0        0        0        0   \n",
       "3        0        0        0        0        0        0        0        0   \n",
       "4        0        0        0        0        0        0        0        0   \n",
       "\n",
       "   Embarked_C  Embarked_Q   Log_Age  Log_Fare  Ones  \n",
       "0           0           1  0.453947  0.893717     1  \n",
       "1           0           0  0.618421  0.845098     1  \n",
       "2           0           1  0.815789  0.986212     1  \n",
       "3           0           0  0.355263  0.937643     1  \n",
       "4           0           0  0.289474  1.089464     1  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_save = test.copy()\n",
    "test2 = convert_data(test)\n",
    "\n",
    "test2_save = test2.copy()\n",
    "test2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([418, 20])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tensor = torch.tensor(test2.values)\n",
    "test_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the predictions, we just need to do the forward pass, i.e. the matrix multiplication of the test values and the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([418])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = test_tensor@params\n",
    "preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.3403, 0.4616, 0.3880, 0.1140, 0.5105], dtype=torch.float64,\n",
       "       grad_fn=<SliceBackward0>)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([418])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.3403, 0.4616, 0.3880, 0.1140, 0.5105], dtype=torch.float64,\n",
       "       grad_fn=<SliceBackward0>)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.squeeze()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['Survived_pred'] = preds.detach().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>...</th>\n",
       "      <th>Parch_6</th>\n",
       "      <th>Parch_5</th>\n",
       "      <th>Parch_9</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Log_Age</th>\n",
       "      <th>Log_Fare</th>\n",
       "      <th>Ones</th>\n",
       "      <th>Survived_pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.453947</td>\n",
       "      <td>0.893717</td>\n",
       "      <td>1</td>\n",
       "      <td>0.340303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.618421</td>\n",
       "      <td>0.845098</td>\n",
       "      <td>1</td>\n",
       "      <td>0.461622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.815789</td>\n",
       "      <td>0.986212</td>\n",
       "      <td>1</td>\n",
       "      <td>0.387979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.937643</td>\n",
       "      <td>1</td>\n",
       "      <td>0.114024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.289474</td>\n",
       "      <td>1.089464</td>\n",
       "      <td>1</td>\n",
       "      <td>0.510544</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin  ... Parch_6  Parch_5  Parch_9  \\\n",
       "0  34.5      0      0   330911   7.8292   NaN  ...       0        0        0   \n",
       "1  47.0      1      0   363272   7.0000   NaN  ...       0        0        0   \n",
       "2  62.0      0      0   240276   9.6875   NaN  ...       0        0        0   \n",
       "3  27.0      0      0   315154   8.6625   NaN  ...       0        0        0   \n",
       "4  22.0      1      1  3101298  12.2875   NaN  ...       0        0        0   \n",
       "\n",
       "   Embarked_Q  Embarked_S  Embarked_C   Log_Age  Log_Fare  Ones  Survived_pred  \n",
       "0           1           0           0  0.453947  0.893717     1       0.340303  \n",
       "1           0           1           0  0.618421  0.845098     1       0.461622  \n",
       "2           1           0           0  0.815789  0.986212     1       0.387979  \n",
       "3           0           1           0  0.355263  0.937643     1       0.114024  \n",
       "4           0           1           0  0.289474  1.089464     1       0.510544  \n",
       "\n",
       "[5 rows x 38 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do these values mean? Let's look at `min` and `max`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived_pred: Min: -0.13200928030085637 Max: 1.2292019860263197\n"
     ]
    }
   ],
   "source": [
    "print(f\"Survived_pred: Min: {test['Survived_pred'].min()} Max: {test['Survived_pred'].max()}\") #test.Survived_pred.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAlOklEQVR4nO3dbXBU9f3+8WtD1iXBJNag2URSCE7wphG1pERC+wNaEkXFOkyrbSjaTrU4aGuasRSati5Vg6bTDG0ptFqLPDCFaS2tM0XIdooRGyw3wtSGetMaEYWYagOJBDcr+f4f7H8XYgLkJLvfk03er5mdcc+e3fM5l5uTi7O7WY8xxggAAMCSFLcHAAAAowvlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVqW4P8FE9PT06dOiQMjIy5PF43B4HAAAMgDFGnZ2dysvLU0rKmc9tDLvycejQIeXn57s9BgAAGISDBw9qwoQJZ1xn2JWPjIwMSVJLS4vOP/98l6dxVzgcVkNDg8rLy+X1et0exzXkEEEOEeRwEllEkEOE2zl0dHQoPz8/9nv8TIZd+Yi+1JKRkaHMzEyXp3FXOBxWenq6MjMzR/0PFDmQQxQ5nEQWEeQQMVxyGMhbJnjDKQAAsIryAQAArKJ8AAAAqygfAADAKsoHAACwivIBAACsonwAAACrKB8AAMAqygcAALCK8gEAAKyifAAAAKsoHwAAwCrKBwAAsIryAQAArEp1ewCcXVFgq0Inzv4VxcPJGw/f4PYIAIBhijMfAADAKsoHAACwivIBAACsonwAAACrKB8AAMAqygcAALCK8gEAAKyifAAAAKsoHwAAwCrKBwAAsIryAQAArKJ8AAAAqxyXj7fffltf+cpXlJ2drfT0dF111VXas2dP7HZjjAKBgPLy8pSWlqbZs2erubk5rkMDAIDk5ah8tLe3a+bMmfJ6vXrmmWe0f/9+/eQnP9F5550XW6e2tlZ1dXVavXq1du3aJb/fr7KyMnV2dsZ7dgAAkIRSnaz8yCOPKD8/X+vWrYstmzRpUuy/jTFatWqVqqurtWDBAknS+vXrlZOTo/r6ei1evDg+UwMAgKTlqHw8/fTTuvbaa/XFL35RjY2Nuuiii7RkyRLdeeedkqSWlha1traqvLw8dh+fz6dZs2apqamp3/IRCoUUCoVi1zs6OiRJ4XBY4XB4UDs1UkT335diXJ7EuXj+v4s+Fs8HcpDI4VRkEUEOEW7n4GS7HmPMgH+zjR07VpJUVVWlL37xi9q5c6cqKyv1q1/9Srfddpuampo0c+ZMvf3228rLy4vd7xvf+IYOHDigrVu39nnMQCCgFStW9FleX1+v9PT0Ae8IAABwT1dXlyoqKnT06FFlZmaecV1HZz56enpUXFysmpoaSdLVV1+t5uZmrV27VrfddltsPY/H0+t+xpg+y6KWL1+uqqqq2PWOjg7l5+drzpw5ys7OdjLeiBMOhxUMBvWD3SkK9fSf32jgSzF6oLgn4Tn8M3Btwh47HqLPh7KyMnm9XrfHcQ05nEQWEeQQ4XYO0VcuBsJR+cjNzdXll1/ea9lll12mp556SpLk9/slSa2trcrNzY2t09bWppycnH4f0+fzyefz9Vnu9XpH9ZPoVKEej0InRm/5iEp0DsnyfONnI4IcTiKLCHKIcCsHJ9t09GmXmTNn6pVXXum17NVXX9XEiRMlSQUFBfL7/QoGg7Hbu7u71djYqNLSUiebAgAAI5SjMx/f/va3VVpaqpqaGt1yyy3auXOnHn30UT366KOSIi+3VFZWqqamRoWFhSosLFRNTY3S09NVUVGRkB0AAADJxVH5+NSnPqVNmzZp+fLl+tGPfqSCggKtWrVKCxcujK2zdOlSHT9+XEuWLFF7e7tKSkrU0NCgjIyMuA8PAACSj6PyIUk33nijbrzxxtPe7vF4FAgEFAgEhjIXAAAYofhuFwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGCVo/IRCATk8Xh6Xfx+f+x2Y4wCgYDy8vKUlpam2bNnq7m5Oe5DAwCA5OX4zMcnPvEJHT58OHZ56aWXYrfV1taqrq5Oq1ev1q5du+T3+1VWVqbOzs64Dg0AAJKX4/KRmpoqv98fu1xwwQWSImc9Vq1aperqai1YsEBFRUVav369urq6VF9fH/fBAQBAckp1eofXXntNeXl58vl8KikpUU1NjSZPnqyWlha1traqvLw8tq7P59OsWbPU1NSkxYsX9/t4oVBIoVAodr2jo0OSFA6HFQ6HnY43okT335diXJ7EXdH9T3QOw/35Fp1vuM+ZaORwEllEkEOE2zk42a7HGDPgI/ozzzyjrq4uTZkyRe+8844efPBBvfzyy2pubtYrr7yimTNn6u2331ZeXl7sPt/4xjd04MABbd26td/HDAQCWrFiRZ/l9fX1Sk9PH/COAAAA93R1damiokJHjx5VZmbmGdd1VD4+6tixY7r44ou1dOlSXXPNNZo5c6YOHTqk3Nzc2Dp33nmnDh48qC1btvT7GP2d+cjPz9fhw4eVnZ092NFGhHA4rGAwqB/sTlGox+P2OK7xpRg9UNyT8Bz+Gbg2YY8dD9HnQ1lZmbxer9vjuIYcTiKLCHKIcDuHjo4OjR8/fkDlw/HLLqcaN26crrjiCr322mu6+eabJUmtra29ykdbW5tycnJO+xg+n08+n6/Pcq/XO6qfRKcK9XgUOjF6y0dUonNIlucbPxsR5HASWUSQQ4RbOTjZ5pD+zkcoFNK//vUv5ebmqqCgQH6/X8FgMHZ7d3e3GhsbVVpaOpTNAACAEcTRmY/77rtP8+fP18c//nG1tbXpwQcfVEdHh26//XZ5PB5VVlaqpqZGhYWFKiwsVE1NjdLT01VRUZGo+QEAQJJxVD7eeustffnLX9a7776rCy64QNdcc41eeOEFTZw4UZK0dOlSHT9+XEuWLFF7e7tKSkrU0NCgjIyMhAwPAACSj6PysWHDhjPe7vF4FAgEFAgEhjITAAAYwfhuFwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFjl6M+rAxheJi37s9sjOPbGwze4PQIAl3HmAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFbx59UBWDXUPwnvG2NUO10qCmxV6IQnTlOdHX8WHogfznwAAACrKB8AAMAqygcAALCK8gEAAKziDafA/zfUN0ImmltvtASAeOPMBwAAsIryAQAArKJ8AAAAqygfAADAKsoHAACwivIBAACsonwAAACrKB8AAMAqygcAALCK8gEAAKyifAAAAKsoHwAAwCrKBwAAsIryAQAArKJ8AAAAqygfAADAKsoHAACwivIBAACsGlL5WLlypTwejyorK2PLjDEKBALKy8tTWlqaZs+erebm5qHOCQAARohBl49du3bp0Ucf1dSpU3str62tVV1dnVavXq1du3bJ7/errKxMnZ2dQx4WAAAkv0GVj/fff18LFy7UY489po997GOx5cYYrVq1StXV1VqwYIGKioq0fv16dXV1qb6+Pm5DAwCA5JU6mDvdfffduuGGGzR37lw9+OCDseUtLS1qbW1VeXl5bJnP59OsWbPU1NSkxYsX93msUCikUCgUu97R0SFJCofDCofDgxlvxIjuvy/FuDyJu6L7Tw7kILmXw3A8HkVnGo6z2UQOEW7n4GS7jsvHhg0b9OKLL2rXrl19bmttbZUk5eTk9Fqek5OjAwcO9Pt4K1eu1IoVK/os37Ztm9LT052ONyI9UNzj9gjDAjlEkEOE7Rw2b95sdXtOBINBt0cYFsghwq0curq6Bryuo/Jx8OBB3XvvvWpoaNDYsWNPu57H4+l13RjTZ1nU8uXLVVVVFbve0dGh/Px8zZkzR9nZ2U7GG3HC4bCCwaB+sDtFoZ7+8xsNfClGDxT3kAM5SHIvh38GrrW2rYGKHiPKysrk9XrdHsc15BDhdg7RVy4GwlH52LNnj9ra2jRt2rTYshMnTui5557T6tWr9corr0iKnAHJzc2NrdPW1tbnbEiUz+eTz+frs9zr9Y7qJ9GpQj0ehU6M3l82UeQQQQ4RtnMYzscjjpcR5BDhVg5OtunoDaef+9zn9NJLL2nfvn2xS3FxsRYuXKh9+/Zp8uTJ8vv9vU75dHd3q7GxUaWlpU42BQAARihHZz4yMjJUVFTUa9m4ceOUnZ0dW15ZWamamhoVFhaqsLBQNTU1Sk9PV0VFRfymBgAASWtQn3Y5k6VLl+r48eNasmSJ2tvbVVJSooaGBmVkZMR7UwAAIAkNuXw8++yzva57PB4FAgEFAoGhPjQAABiB+G4XAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgVarbAwBAMpi07M9uj9CHb4xR7XSpKLBVoROePre/8fANLkwFnB1nPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABglaPysXbtWk2dOlWZmZnKzMzUjBkz9Mwzz8RuN8YoEAgoLy9PaWlpmj17tpqbm+M+NAAASF6OyseECRP08MMPa/fu3dq9e7c++9nP6vOf/3ysYNTW1qqurk6rV6/Wrl275Pf7VVZWps7OzoQMDwAAko+j8jF//nxdf/31mjJliqZMmaKHHnpI5557rl544QUZY7Rq1SpVV1drwYIFKioq0vr169XV1aX6+vpEzQ8AAJLMoN/zceLECW3YsEHHjh3TjBkz1NLSotbWVpWXl8fW8fl8mjVrlpqamuIyLAAASH6O/7z6Sy+9pBkzZuiDDz7Queeeq02bNunyyy+PFYycnJxe6+fk5OjAgQOnfbxQKKRQKBS73tHRIUkKh8MKh8NOxxtRovvvSzEuT+Ku6P6TAzlI5HCqs2UxWo6h0f0cLft7Om7n4GS7HmOMo5/g7u5uvfnmmzpy5Iieeuop/frXv1ZjY6OOHDmimTNn6tChQ8rNzY2tf+edd+rgwYPasmVLv48XCAS0YsWKPsvr6+uVnp7uZDQAAOCSrq4uVVRU6OjRo8rMzDzjuo7Lx0fNnTtXF198sb773e/q4osv1osvvqirr746dvvnP/95nXfeeVq/fn2/9+/vzEd+fr4OHz6s7OzsoYyW9MLhsILBoH6wO0Whnr5fGjVa+FKMHijuIQdykEQOpzpbFv8MXOvCVPZFj5VlZWXyer1uj+Mat3Po6OjQ+PHjB1Q+hvyttsYYhUIhFRQUyO/3KxgMxspHd3e3Ghsb9cgjj5z2/j6fTz6fr89yr9c7qp9Epwr1ePr9xsrRhhwiyCGCHE46XRaj7RjK740It3Jwsk1H5eN73/ue5s2bp/z8fHV2dmrDhg169tlntWXLFnk8HlVWVqqmpkaFhYUqLCxUTU2N0tPTVVFR4XgnAADAyOSofLzzzjtatGiRDh8+rKysLE2dOlVbtmxRWVmZJGnp0qU6fvy4lixZovb2dpWUlKihoUEZGRkJGR4AACQfR+Xj8ccfP+PtHo9HgUBAgUBgKDMBAIARjO92AQAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFY5Kh8rV67Upz71KWVkZOjCCy/UzTffrFdeeaXXOsYYBQIB5eXlKS0tTbNnz1Zzc3NchwYAAMnLUflobGzU3XffrRdeeEHBYFAffvihysvLdezYsdg6tbW1qqur0+rVq7Vr1y75/X6VlZWps7Mz7sMDAIDkk+pk5S1btvS6vm7dOl144YXas2eP/u///k/GGK1atUrV1dVasGCBJGn9+vXKyclRfX29Fi9eHL/JAQBAUhrSez6OHj0qSTr//PMlSS0tLWptbVV5eXlsHZ/Pp1mzZqmpqWkomwIAACOEozMfpzLGqKqqSp/+9KdVVFQkSWptbZUk5eTk9Fo3JydHBw4c6PdxQqGQQqFQ7HpHR4ckKRwOKxwOD3a8ESG6/74U4/Ik7oruPzmQg0QOpzpbFqPlGBrdz9Gyv6fjdg5Otjvo8nHPPffoH//4h55//vk+t3k8nl7XjTF9lkWtXLlSK1as6LN827ZtSk9PH+x4I8oDxT1ujzAskEMEOUSQw0mny2Lz5s2WJ3FXMBh0e4Rhwa0curq6BrzuoMrHN7/5TT399NN67rnnNGHChNhyv98vKXIGJDc3N7a8ra2tz9mQqOXLl6uqqip2vaOjQ/n5+ZozZ46ys7MHM96IEQ6HFQwG9YPdKQr19F/eRgNfitEDxT3kQA6SyOFUIzGLfwaudXyf6LGyrKxMXq83AVMlB7dziL5yMRCOyocxRt/85je1adMmPfvssyooKOh1e0FBgfx+v4LBoK6++mpJUnd3txobG/XII4/0+5g+n08+n6/Pcq/XO6qfRKcK9XgUOjEyDixDQQ4R5BBBDieNpCyGctzn90aEWzk42aaj8nH33Xervr5ef/rTn5SRkRF7j0dWVpbS0tLk8XhUWVmpmpoaFRYWqrCwUDU1NUpPT1dFRYWzvQAAACOSo/Kxdu1aSdLs2bN7LV+3bp2++tWvSpKWLl2q48ePa8mSJWpvb1dJSYkaGhqUkZERl4EBAEByc/yyy9l4PB4FAgEFAoHBzgQAAEYwvtsFAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYFWq2wMAABA1admfHd/HN8aodrpUFNiq0AlPAqY6szcevsH6NpOd4zMfzz33nObPn6+8vDx5PB798Y9/7HW7MUaBQEB5eXlKS0vT7Nmz1dzcHK95AQBAknNcPo4dO6Yrr7xSq1ev7vf22tpa1dXVafXq1dq1a5f8fr/KysrU2dk55GEBAEDyc/yyy7x58zRv3rx+bzPGaNWqVaqurtaCBQskSevXr1dOTo7q6+u1ePHioU0LAACSXlzf89HS0qLW1laVl5fHlvl8Ps2aNUtNTU39lo9QKKRQKBS73tHRIUkKh8MKh8PxHC/pRPffl2JcnsRd0f0nB3KQyOFUZBHhdg7D5XdVdA635nGy3biWj9bWVklSTk5Or+U5OTk6cOBAv/dZuXKlVqxY0Wf5tm3blJ6eHs/xktYDxT1ujzAskEMEOUSQw0lkEeFWDps3b3Zlu6cTDAZd2W5XV9eA103Ip108nt7vNjbG9FkWtXz5clVVVcWud3R0KD8/X3PmzFF2dnYixksa4XBYwWBQP9idolCP/XdwDxe+FKMHinvIgRwkkcOpyCLC7Rz+GbjW+jb7E/2dUVZWJq/Xa3370VcuBiKu5cPv90uKnAHJzc2NLW9ra+tzNiTK5/PJ5/P1We71el0JbzgK9Xhc+fjYcEMOEeQQQQ4nkUWEWzkMt99Vbv3+dLLNuP6RsYKCAvn9/l6nfLq7u9XY2KjS0tJ4bgoAACQpx2c+3n//ff373/+OXW9padG+fft0/vnn6+Mf/7gqKytVU1OjwsJCFRYWqqamRunp6aqoqIjr4AAAIDk5Lh+7d+/WnDlzYtej79e4/fbb9cQTT2jp0qU6fvy4lixZovb2dpWUlKihoUEZGRnxmxoAACQtx+Vj9uzZMub0H2fyeDwKBAIKBAJDmQsAAIxQfLEcAACwivIBAACsonwAAACrKB8AAMAqygcAALCK8gEAAKyifAAAAKsoHwAAwCrKBwAAsIryAQAArHL859UBAMBJk5b92e0RJEm+MUa106WiwFaFTnjOuO4bD99gaar+ceYDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYBXlAwAAWEX5AAAAVqW6PYBtk5b92e0RBsw3xqh2uttTAAAQX5z5AAAAVlE+AACAVZQPAABgFeUDAABYRfkAAABWUT4AAIBVlA8AAGAV5QMAAFhF+QAAAFZRPgAAgFWUDwAAYFXCyseaNWtUUFCgsWPHatq0adq+fXuiNgUAAJJIQsrHxo0bVVlZqerqau3du1ef+cxnNG/ePL355puJ2BwAAEgiCSkfdXV1+vrXv6477rhDl112mVatWqX8/HytXbs2EZsDAABJJDXeD9jd3a09e/Zo2bJlvZaXl5erqampz/qhUEihUCh2/ejRo5Kk//3vf/EeTZKU+uGxhDxuIqT2GHV19Sg1nKITPR63x3ENOUSQQwQ5nEQWEeQQ4SSH9957L+7b7+zslCQZY86+somzt99+20gyf/vb33otf+ihh8yUKVP6rH///fcbSVy4cOHChQuXEXA5ePDgWbtC3M98RHk8vVuXMabPMklavny5qqqqYtePHDmiiRMn6s0331RWVlaixksKHR0dys/P18GDB5WZmen2OK4hhwhyiCCHk8gighwi3M7BGKPOzk7l5eWddd24l4/x48drzJgxam1t7bW8ra1NOTk5fdb3+Xzy+Xx9lmdlZY3qJ9GpMjMzyULkEEUOEeRwEllEkEOEmzkM9KRB3N9wes4552jatGkKBoO9lgeDQZWWlsZ7cwAAIMkk5GWXqqoqLVq0SMXFxZoxY4YeffRRvfnmm7rrrrsSsTkAAJBEElI+br31Vr333nv60Y9+pMOHD6uoqEibN2/WxIkTz3pfn8+n+++/v9+XYkYbsogghwhyiCCHk8gighwikikHjzED+UwMAABAfPDdLgAAwCrKBwAAsIryAQAArKJ8AAAAq4ZF+Whvb9eiRYuUlZWlrKwsLVq0SEeOHDnt+uFwWN/97nd1xRVXaNy4ccrLy9Ntt92mQ4cO2Rs6TtasWaOCggKNHTtW06ZN0/bt28+4fmNjo6ZNm6axY8dq8uTJ+uUvf2lp0sRyksMf/vAHlZWV6YILLlBmZqZmzJihrVu3Wpw2cZw+H6L+9re/KTU1VVdddVViB7TEaQ6hUEjV1dWaOHGifD6fLr74Yv3mN7+xNG1iOc3iySef1JVXXqn09HTl5ubqa1/7WkK+x8Om5557TvPnz1deXp48Ho/++Mc/nvU+I/FY6TSHYX2sjMsXugzRddddZ4qKikxTU5NpamoyRUVF5sYbbzzt+keOHDFz5841GzduNC+//LLZsWOHKSkpMdOmTbM49dBt2LDBeL1e89hjj5n9+/ebe++914wbN84cOHCg3/Vff/11k56ebu69916zf/9+89hjjxmv12t+//vfW548vpzmcO+995pHHnnE7Ny507z66qtm+fLlxuv1mhdffNHy5PHlNIeoI0eOmMmTJ5vy8nJz5ZVX2hk2gQaTw0033WRKSkpMMBg0LS0t5u9//3uf75dKRk6z2L59u0lJSTE//elPzeuvv262b99uPvGJT5ibb77Z8uTxtXnzZlNdXW2eeuopI8ls2rTpjOuP1GOl0xyG87HS9fKxf/9+I8m88MILsWU7duwwkszLL7884MfZuXOnkXTWA/VwMn36dHPXXXf1WnbppZeaZcuW9bv+0qVLzaWXXtpr2eLFi80111yTsBltcJpDfy6//HKzYsWKeI9m1WBzuPXWW833v/99c//994+I8uE0h2eeecZkZWWZ9957z8Z4VjnN4sc//rGZPHlyr2U/+9nPzIQJExI2o20D+aU7Uo+VpxpIDv0ZLsdK11922bFjh7KyslRSUhJbds011ygrK0tNTU0DfpyjR4/K4/HovPPOS8CU8dfd3a09e/aovLy81/Ly8vLT7veOHTv6rH/ttddq9+7dCofDCZs1kQaTw0f19PSos7NT559/fiJGtGKwOaxbt07/+c9/dP/99yd6RCsGk8PTTz+t4uJi1dbW6qKLLtKUKVN033336fjx4zZGTpjBZFFaWqq33npLmzdvljFG77zzjn7/+9/rhhtusDHysDESj5XxMJyOlQn7VtuBam1t1YUXXthn+YUXXtjny+lO54MPPtCyZctUUVGRNF8q9O677+rEiRN9vmwvJyfntPvd2tra7/offvih3n33XeXm5iZs3kQZTA4f9ZOf/ETHjh3TLbfckogRrRhMDq+99pqWLVum7du3KzXV9R/luBhMDq+//rqef/55jR07Vps2bdK7776rJUuW6H//+19Sv+9jMFmUlpbqySef1K233qoPPvhAH374oW666Sb9/Oc/tzHysDESj5XxMJyOlQk78xEIBOTxeM542b17tyTJ4/H0ub8xpt/lHxUOh/WlL31JPT09WrNmTdz3I9E+uo9n2+/+1u9vebJxmkPUb3/7WwUCAW3cuLHfEptsBprDiRMnVFFRoRUrVmjKlCm2xrPGyfOhp6dHHo9HTz75pKZPn67rr79edXV1euKJJ5L+7IfkLIv9+/frW9/6ln74wx9qz5492rJli1paWkbl92qN1GPlYA23Y2XC/rl0zz336Etf+tIZ15k0aZL+8Y9/6J133ulz23//+98+zfWjwuGwbrnlFrW0tOivf/1r0pz1kKTx48drzJgxff4F09bWdtr99vv9/a6fmpqq7OzshM2aSIPJIWrjxo36+te/rt/97neaO3duIsdMOKc5dHZ2avfu3dq7d6/uueceSZFfwsYYpaamqqGhQZ/97GetzB5Pg3k+5Obm6qKLLur1Vd6XXXaZjDF66623VFhYmNCZE2UwWaxcuVIzZ87Ud77zHUnS1KlTNW7cOH3mM5/Rgw8+OGr+xT8Sj5VDMRyPlQk78zF+/HhdeumlZ7yMHTtWM2bM0NGjR7Vz587Yff/+97/r6NGjKi0tPe3jR4vHa6+9pr/85S9J94Q655xzNG3aNAWDwV7Lg8Hgafd7xowZfdZvaGhQcXGxvF5vwmZNpMHkIEVa/Fe/+lXV19ePiNezneaQmZmpl156Sfv27Ytd7rrrLl1yySXat29fr/dQJZPBPB9mzpypQ4cO6f33348te/XVV5WSkqIJEyYkdN5EGkwWXV1dSknpfVgfM2aMpJP/8h8NRuKxcrCG7bHSnfe59nbdddeZqVOnmh07dpgdO3aYK664os9HbS+55BLzhz/8wRhjTDgcNjfddJOZMGGC2bdvnzl8+HDsEgqF3NiFQYl+jO7xxx83+/fvN5WVlWbcuHHmjTfeMMYYs2zZMrNo0aLY+tGPj3372982+/fvN48//viI+PiY0xzq6+tNamqq+cUvftHr//2RI0fc2oW4cJrDR42UT7s4zaGzs9NMmDDBfOELXzDNzc2msbHRFBYWmjvuuMOtXYgbp1msW7fOpKammjVr1pj//Oc/5vnnnzfFxcVm+vTpbu1CXHR2dpq9e/eavXv3Gkmmrq7O7N27N/bpxtFyrHSaw3A+Vg6L8vHee++ZhQsXmoyMDJORkWEWLlxo2tvbe60jyaxbt84YY0xLS4uR1O9l27Zt1ucfil/84hdm4sSJ5pxzzjGf/OQnTWNjY+y222+/3cyaNavX+s8++6y5+uqrzTnnnGMmTZpk1q5da3nixHCSw6xZs/r9f3/77bfbHzzOnD4fTjVSyocxznP417/+ZebOnWvS0tLMhAkTTFVVlenq6rI8dWI4zeJnP/uZufzyy01aWprJzc01CxcuNG+99ZblqeNr27ZtZ/yZHy3HSqc5DOdjpceYUXQuDgAAuM71v/MBAABGF8oHAACwivIBAACsonwAAACrKB8AAMAqygcAALCK8gEAAKyifAAAAKsoHwAAwCrKBwAAsIryAQAArKJ8AAAAq/4f1ivpAZ/ZACIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test.Survived_pred.hist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4379399037518624"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.Survived_pred.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deriving the survived-column, let's just say that any value greater than 0.5 means survived."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['Survived'] = [ 1 if element > 0.5 else 0 for element in preds.squeeze()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>...</th>\n",
       "      <th>Parch_5</th>\n",
       "      <th>Parch_9</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Log_Age</th>\n",
       "      <th>Log_Fare</th>\n",
       "      <th>Ones</th>\n",
       "      <th>Survived_pred</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.453947</td>\n",
       "      <td>0.893717</td>\n",
       "      <td>1</td>\n",
       "      <td>0.340303</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.618421</td>\n",
       "      <td>0.845098</td>\n",
       "      <td>1</td>\n",
       "      <td>0.461622</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.815789</td>\n",
       "      <td>0.986212</td>\n",
       "      <td>1</td>\n",
       "      <td>0.387979</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.937643</td>\n",
       "      <td>1</td>\n",
       "      <td>0.114024</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.289474</td>\n",
       "      <td>1.089464</td>\n",
       "      <td>1</td>\n",
       "      <td>0.510544</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 39 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin  ... Parch_5  Parch_9  \\\n",
       "0  34.5      0      0   330911   7.8292   NaN  ...       0        0   \n",
       "1  47.0      1      0   363272   7.0000   NaN  ...       0        0   \n",
       "2  62.0      0      0   240276   9.6875   NaN  ...       0        0   \n",
       "3  27.0      0      0   315154   8.6625   NaN  ...       0        0   \n",
       "4  22.0      1      1  3101298  12.2875   NaN  ...       0        0   \n",
       "\n",
       "   Embarked_Q  Embarked_S  Embarked_C   Log_Age  Log_Fare  Ones  \\\n",
       "0           1           0           0  0.453947  0.893717     1   \n",
       "1           0           1           0  0.618421  0.845098     1   \n",
       "2           1           0           0  0.815789  0.986212     1   \n",
       "3           0           1           0  0.355263  0.937643     1   \n",
       "4           0           1           0  0.289474  1.089464     1   \n",
       "\n",
       "   Survived_pred  Survived  \n",
       "0       0.340303         0  \n",
       "1       0.461622         0  \n",
       "2       0.387979         0  \n",
       "3       0.114024         0  \n",
       "4       0.510544         1  \n",
       "\n",
       "[5 rows x 39 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a new dataframe with just the columns `PassengerId` and `Survived` (as requested for the submission of the results):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived\n",
       "0          892         0\n",
       "1          893         0\n",
       "2          894         0\n",
       "3          895         0\n",
       "4          896         1"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test[['PassengerId', 'Survived']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = test[['PassengerId', 'Survived']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I uploaded the results, and the result is unbelievably good: **Score: 0.75837**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One more attempt: Let's set the threshold to 0.6:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['Survived'] = [ 1 if element > 0.6 else 0 for element in preds.squeeze()]\n",
    "submission4 = test[['PassengerId', 'Survived']]\n",
    "submission4.to_csv('submission4.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score went down: **Score: 0.72727** (I somehow expected that looking at the histogram.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Re-do with 3 layers of parameters\n",
    "\n",
    "I initially coded it with 2 layers, if you want to get the 2 layer version, feel free to remove the 3rd layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_layer():\n",
    "    layer = torch.rand(20, dtype=torch.float64)-0.5 #https://pytorch.org/docs/stable/generated/torch.rand.html\n",
    "    layer.requires_grad_()\n",
    "    return layer\n",
    "\n",
    "layer1 = create_layer()\n",
    "layer2 = create_layer()\n",
    "layer3 = create_layer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.2657,  0.4991, -0.3225,  0.1319,  0.1734,  0.2937,  0.1468,  0.0825,\n",
       "        -0.0401,  0.2960,  0.4038,  0.4735,  0.1428, -0.1275, -0.2948, -0.4493,\n",
       "        -0.0552,  0.0775,  0.2237,  0.0927], dtype=torch.float64,\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.1783,  0.1441, -0.2199,  0.4132, -0.4671, -0.0144,  0.4927,  0.0895,\n",
       "        -0.2389,  0.4413, -0.3629,  0.3726,  0.0590,  0.3451, -0.3747,  0.4434,\n",
       "        -0.4538, -0.2640, -0.4761,  0.3950], dtype=torch.float64,\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.2419,  0.4471,  0.1690, -0.3446, -0.4179,  0.2309,  0.2764,  0.4769,\n",
       "        -0.4804, -0.4616, -0.0706, -0.1562,  0.0494,  0.3238,  0.4845,  0.1826,\n",
       "         0.4001, -0.1978,  0.2509, -0.4074], dtype=torch.float64,\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimizer essentially contains the same steps as above.\n",
    "\n",
    "True, I could have done a better job at shortening the code, but I think this way it is more explicit what is happening, and it is easy to comment the 3rd layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_optimizer2_step(layer1, layer2, layer3, lr=0.01):\n",
    "    preds1 = train_tensor@layer1\n",
    "    preds1 = preds1.relu()\n",
    "    preds2 = train_tensor@layer2\n",
    "    preds2 = preds2.relu()\n",
    "    preds3 = train_tensor@layer3\n",
    "    preds3 = preds3.relu()\n",
    "    preds = preds1 + preds2\n",
    "    preds += preds3\n",
    "    loss = ((preds-survived_tensor)**2).mean()\n",
    "    \n",
    "    layer1.grad = None #Initializing gradients (.zero_grad())\n",
    "    layer2.grad = None #Initializing gradients (.zero_grad())\n",
    "    layer3.grad = None #Initializing gradients (.zero_grad())\n",
    "    \n",
    "    loss.backward() #Calculate gradients\n",
    "    with torch.no_grad():\n",
    "        layer1 -= layer1.grad*lr #update parameters\n",
    "        layer2 -= layer2.grad*lr #update parameters\n",
    "        layer3 -= layer3.grad*lr #update parameters\n",
    "    \n",
    "    return loss   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.27\n",
      "Loss: 0.19\n",
      "Loss: 0.19\n",
      "Loss: 0.19\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.18\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.17\n",
      "Loss: 0.16\n"
     ]
    }
   ],
   "source": [
    "for i in range(25):\n",
    "    loss = run_optimizer2_step(layer1, layer2, layer3, lr=0.1)\n",
    "    print(f\"Loss: {loss:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, we now have new parameters, we can apply to the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Re-do Predictions with new model\n",
    "\n",
    "Let's recover the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = test_save.copy()\n",
    "test2 = test2_save.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  \n",
       "0  34.5      0      0   330911   7.8292   NaN        Q  \n",
       "1  47.0      1      0   363272   7.0000   NaN        S  \n",
       "2  62.0      0      0   240276   9.6875   NaN        Q  \n",
       "3  27.0      0      0   315154   8.6625   NaN        S  \n",
       "4  22.0      1      1  3101298  12.2875   NaN        S  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Pclass_1</th>\n",
       "      <th>Pclass_2</th>\n",
       "      <th>SibSp_0</th>\n",
       "      <th>SibSp_1</th>\n",
       "      <th>SibSp_2</th>\n",
       "      <th>SibSp_3</th>\n",
       "      <th>SibSp_4</th>\n",
       "      <th>Parch_0</th>\n",
       "      <th>Parch_1</th>\n",
       "      <th>Parch_2</th>\n",
       "      <th>Parch_3</th>\n",
       "      <th>Parch_4</th>\n",
       "      <th>Parch_5</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Log_Age</th>\n",
       "      <th>Log_Fare</th>\n",
       "      <th>Ones</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.453947</td>\n",
       "      <td>0.893717</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.618421</td>\n",
       "      <td>0.845098</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.815789</td>\n",
       "      <td>0.986212</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.355263</td>\n",
       "      <td>0.937643</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.289474</td>\n",
       "      <td>1.089464</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sex_male  Sex_female  Pclass_1  Pclass_2  SibSp_0  SibSp_1  SibSp_2  \\\n",
       "0         1           0         0         0        0        0        0   \n",
       "1         0           1         0         0        0        0        0   \n",
       "2         1           0         0         0        0        0        0   \n",
       "3         1           0         0         0        0        0        0   \n",
       "4         0           1         0         0        0        0        0   \n",
       "\n",
       "   SibSp_3  SibSp_4  Parch_0  Parch_1  Parch_2  Parch_3  Parch_4  Parch_5  \\\n",
       "0        0        0        0        0        0        0        0        0   \n",
       "1        0        0        0        0        0        0        0        0   \n",
       "2        0        0        0        0        0        0        0        0   \n",
       "3        0        0        0        0        0        0        0        0   \n",
       "4        0        0        0        0        0        0        0        0   \n",
       "\n",
       "   Embarked_C  Embarked_Q   Log_Age  Log_Fare  Ones  \n",
       "0           0           1  0.453947  0.893717     1  \n",
       "1           0           0  0.618421  0.845098     1  \n",
       "2           0           1  0.815789  0.986212     1  \n",
       "3           0           0  0.355263  0.937643     1  \n",
       "4           0           0  0.289474  1.089464     1  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the forward pass to do get the predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived_pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>0.250963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.620510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>0.219481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.097688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.639145</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  Survived_pred  \n",
       "0  34.5      0      0   330911   7.8292   NaN        Q       0.250963  \n",
       "1  47.0      1      0   363272   7.0000   NaN        S       0.620510  \n",
       "2  62.0      0      0   240276   9.6875   NaN        Q       0.219481  \n",
       "3  27.0      0      0   315154   8.6625   NaN        S       0.097688  \n",
       "4  22.0      1      1  3101298  12.2875   NaN        S       0.639145  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds1 = test_tensor@layer1\n",
    "preds1 = preds1.relu()\n",
    "preds2 = test_tensor@layer2\n",
    "preds2 = preds2.relu()\n",
    "preds3 = test_tensor@layer3\n",
    "preds3 = preds3.relu()\n",
    "preds = preds1 + preds2\n",
    "preds += preds3\n",
    "\n",
    "test['Survived_pred'] = preds.detach().numpy()\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, we get 2 separate areas in the histogram which separate survived and not survived:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGdCAYAAAA8F1jjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmJUlEQVR4nO3df3Db9X3H8ZccK4qV2un4ZdnEJA5zWqgHZXFjYl9nt8PiQkrL5Uq5mdGwtSWcYcP1dV48r0MuTAHv8LmdSzY4luauuHAtP8qtIbF6GybUYzhpcqNOB6wYAyHCF8fEJk5lEX/2R87ajB2a79f62h/Lz8ed7vh+9NVHb735SN9XvpIsnzHGCAAAwEJZ810AAADA2RBUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWyp7vAj5sYmJC77zzjnJzc+Xz+ea7HAAAcA6MMRodHVVhYaGystJ3HsS6oPLOO++oqKhovssAAAAuvPXWW1q5cmXa5rMuqOTm5ko680Dz8vLSOncymVRXV5fC4bD8fn9a58509M49eucevXOP3rlH79w5fvy4iouLU8fxdLEuqEy+3ZOXl+dJUAkGg8rLy2PxOUTv3KN37tE79+ide/TOnWQyKUlp/9iGozeRVq9eLZ/PN+1yxx13SDrz/lQkElFhYaFycnJUXV2tvr6+tBYMAAAWD0dBpbe3V0ePHk1dYrGYJOnGG2+UJLW2tqqtrU0dHR3q7e1VKBRSTU2NRkdH0185AADIeI6CyoUXXqhQKJS6/Ou//qsuvfRSVVVVyRij9vZ2NTc3a/PmzSotLdWuXbs0Njamzs5Or+oHAAAZzPVnVMbHx/XDH/5QDQ0N8vl8ev311xWPxxUOh1P7BAIBVVVVqaenR1u3bp1xnkQioUQikdoeGRmRdOa9rsn3u9Jlcr50z7sY0Dv36J179M49eucevXPHq365DipPP/203nvvPd16662SpHg8LknKz8+fsl9+fr4GBgbOOs/27dvV0tIybbyrq0vBYNBteR9p8i0rOEfv3KN37tE79+ide/TOmbGxMU/mdR1UHnnkEW3cuFGFhYVTxj/8aV9jzEd+AripqUkNDQ2p7ZGRERUVFSkcDnvyrZ9YLKaamho+ye0QvXOP3rlH79yjd+7RO3eGhoY8mddVUBkYGNDPf/5zPfnkk6mxUCgk6cyZlYKCgtT44ODgtLMs/18gEFAgEJg27vf7PVsgXs6d6eide/TOPXrnHr1zj94541WvXP2N2507d+qiiy7Spk2bUmPFxcUKhUJTTpWNj4+ru7tbFRUVs68UAAAsOo7PqExMTGjnzp3asmWLsrP/7+Y+n0/19fWKRqMqKSlRSUmJotGogsGgamtr01o0AABYHBwHlZ///Od688039ed//ufTrmtsbNSpU6dUV1en4eFhlZeXq6urK+1/ThcAACwOjoNKOByWMWbG63w+nyKRiCKRyGzrAgAAcPcZFQAAgLlAUAEAANYiqAAAAGu5/oNvC1lpZK8Sp9P7M9ReeuO+Tb97JwAAMhBnVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYy3FQOXLkiP70T/9U559/voLBoD796U/rwIEDqeuNMYpEIiosLFROTo6qq6vV19eX1qIBAMDi4CioDA8Pq7KyUn6/X88++6wOHz6sBx54QB//+MdT+7S2tqqtrU0dHR3q7e1VKBRSTU2NRkdH0107AADIcNlOdr7//vtVVFSknTt3psZWr16d+m9jjNrb29Xc3KzNmzdLknbt2qX8/Hx1dnZq69at6akaAAAsCo6CyjPPPKNrr71WN954o7q7u3XxxRerrq5O3/jGNyRJ/f39isfjCofDqdsEAgFVVVWpp6dnxqCSSCSUSCRS2yMjI5KkZDKpZDLp6kGdzeR8gSyT1nm9lu4+zKYGG2pZaOide/TOPXrnHr1zx6t++Ywx53zUXrZsmSSpoaFBN954o1566SXV19frn//5n/XVr35VPT09qqys1JEjR1RYWJi63W233aaBgQHt3bt32pyRSEQtLS3Txjs7OxUMBt08JgAAMMfGxsZUW1urEydOKC8vL23zOjqjMjExobKyMkWjUUnSVVddpb6+Pu3YsUNf/epXU/v5fL4ptzPGTBub1NTUpIaGhtT2yMiIioqKFA6H0/pApTNpLxaL6dv7s5SYmLkeG/0qcu18l5DqXU1Njfx+/3yXs6DQO/fonXv0zj16587Q0JAn8zoKKgUFBbr88sunjF122WV64oknJEmhUEiSFI/HVVBQkNpncHBQ+fn5M84ZCAQUCASmjfv9fs8WSGLCp8TphRNUbHqiePn/JdPRO/fonXv0zj1654xXvXL0rZ/Kykq98sorU8ZeffVVrVq1SpJUXFysUCikWCyWun58fFzd3d2qqKhIQ7kAAGAxcXRG5Zvf/KYqKioUjUb1la98RS+99JIeeughPfTQQ5LOvOVTX1+vaDSqkpISlZSUKBqNKhgMqra21pMHAAAAMpejoPKZz3xGTz31lJqamvSd73xHxcXFam9v180335zap7GxUadOnVJdXZ2Gh4dVXl6urq4u5ebmpr14AACQ2RwFFUn6whe+oC984Qtnvd7n8ykSiSgSicymLgAAAH7rBwAA2IugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwluM/+Ia5t3rbz+a7BAWWGLWul0oje8/pBx3fuG/THFQFAMh0nFEBAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWyp7vAgBgIVi97WfzXYIjgSVGrevnuwpg9jijAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWo6CSiQSkc/nm3IJhUKp640xikQiKiwsVE5Ojqqrq9XX15f2ogEAwOLg+IzKpz71KR09ejR1efnll1PXtba2qq2tTR0dHert7VUoFFJNTY1GR0fTWjQAAFgcHAeV7OxshUKh1OXCCy+UdOZsSnt7u5qbm7V582aVlpZq165dGhsbU2dnZ9oLBwAAmc/xH3x77bXXVFhYqEAgoPLyckWjUa1Zs0b9/f2Kx+MKh8OpfQOBgKqqqtTT06OtW7fOOF8ikVAikUhtj4yMSJKSyaSSyaTT8j7S5HyBLJPWeReDyZ6da+/S/f9uIZvsBT1xzqbeBZYsrNeNyeeqDb1baGxadwuJV/3yGWPO+dn37LPPamxsTGvXrtW7776re++9V//93/+tvr4+vfLKK6qsrNSRI0dUWFiYus1tt92mgYEB7d27d8Y5I5GIWlpapo13dnYqGAy6eEgAAGCujY2Nqba2VidOnFBeXl7a5nUUVD7s5MmTuvTSS9XY2Kirr75alZWVeuedd1RQUJDa5xvf+Ibeeust7dmzZ8Y5ZjqjUlRUpGPHjqX1gUpn0l4sFtO392cpMeFL69yZLpBldE/ZxDn37leRa+egqoVhct3V1NTI7/fPdzkLik29K43M/I8tW00+Z23o3UJj07pbSIaGhlRQUJD2oDKr3/pZvny5/uAP/kCvvfaabrjhBklSPB6fElQGBweVn59/1jkCgYACgcC0cb/f79kCSUz4lDhNUHHjXHvHk3s6L9d0prOhdwv1NcOG3i1U9M4Zr3o1q7+jkkgk9Otf/1oFBQUqLi5WKBRSLBZLXT8+Pq7u7m5VVFTMulAAALD4ODqj8q1vfUvXX3+9LrnkEg0ODuree+/VyMiItmzZIp/Pp/r6ekWjUZWUlKikpETRaFTBYFC1tbVe1Q8AADKYo6Dy9ttv60/+5E907NgxXXjhhbr66qv14osvatWqVZKkxsZGnTp1SnV1dRoeHlZ5ebm6urqUm5vrSfEAACCzOQoqjz322Ede7/P5FIlEFIlEZlMTMsDqbT+b7xIce+O+TfNdAgDgQ/itHwAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGCtWQWV7du3y+fzqb6+PjVmjFEkElFhYaFycnJUXV2tvr6+2dYJAAAWIddBpbe3Vw899JCuuOKKKeOtra1qa2tTR0eHent7FQqFVFNTo9HR0VkXCwAAFhdXQeX999/XzTffrIcffli/93u/lxo3xqi9vV3Nzc3avHmzSktLtWvXLo2NjamzszNtRQMAgMUh282N7rjjDm3atEnXXHON7r333tR4f3+/4vG4wuFwaiwQCKiqqko9PT3aunXrtLkSiYQSiURqe2RkRJKUTCaVTCbdlHdWk/MFskxa510MJnuWyb1L93r78LxezZ/JbOpdYMnCWvuTz1UberfQ2LTuFhKv+uU4qDz22GP65S9/qd7e3mnXxeNxSVJ+fv6U8fz8fA0MDMw43/bt29XS0jJtvKurS8Fg0Gl55+SesglP5l0MMrl3u3fv9nT+WCzm6fyZzIbeta6f7wrcsaF3CxW9c2ZsbMyTeR0Flbfeekt33XWXurq6tGzZsrPu5/P5pmwbY6aNTWpqalJDQ0Nqe2RkREVFRQqHw8rLy3NS3u+UTCYVi8X07f1ZSkzMXA9mFsgyuqdsIqN796vItZ7MO7nuampq5Pf7PbmPTGVT70oje+f1/p2afM7a0LuFxqZ1t5AMDQ15Mq+joHLgwAENDg5q3bp1qbHTp0/r+eefV0dHh1555RVJZ86sFBQUpPYZHBycdpZlUiAQUCAQmDbu9/s9WyCJCZ8SpzPzYOu1TO6d1y9IXq7pTGdD7xbqurehdwsVvXPGq145+jDtH//xH+vll1/WoUOHUpeysjLdfPPNOnTokNasWaNQKDTldNn4+Li6u7tVUVGR9uIBAEBmc3RGJTc3V6WlpVPGli9frvPPPz81Xl9fr2g0qpKSEpWUlCgajSoYDKq2tjZ9VQMAgEXB1bd+PkpjY6NOnTqluro6DQ8Pq7y8XF1dXcrNzU33XQEAgAw366Dy3HPPTdn2+XyKRCKKRCKznRoAACxy/NYPAACwFkEFAABYi6ACAACsRVABAADWSvu3fgDgo6ze9rNz3jewxKh1/Zm/CrtQ/+AagNnhjAoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1nIUVHbs2KErrrhCeXl5ysvL04YNG/Tss8+mrjfGKBKJqLCwUDk5OaqurlZfX1/aiwYAAIuDo6CycuVK3Xfffdq/f7/279+vz3/+8/rSl76UCiOtra1qa2tTR0eHent7FQqFVFNTo9HRUU+KBwAAmc1RULn++ut13XXXae3atVq7dq3+/u//Xh/72Mf04osvyhij9vZ2NTc3a/PmzSotLdWuXbs0Njamzs5Or+oHAAAZLNvtDU+fPq0f//jHOnnypDZs2KD+/n7F43GFw+HUPoFAQFVVVerp6dHWrVtnnCeRSCiRSKS2R0ZGJEnJZFLJZNJteTOanC+QZdI672Iw2bNM7l2619uH5/Vq/oUmsOTc19BiWHdemewZ6845nrPueNUvnzHG0SvAyy+/rA0bNui3v/2tPvaxj6mzs1PXXXedenp6VFlZqSNHjqiwsDC1/2233aaBgQHt3bt3xvkikYhaWlqmjXd2dioYDDp8OAAAYD6MjY2ptrZWJ06cUF5eXtrmdXxG5ROf+IQOHTqk9957T0888YS2bNmi7u7u1PU+n2/K/saYaWP/X1NTkxoaGlLbIyMjKioqUjgcTusDlc6kvVgspm/vz1Ji4uw1YbpAltE9ZRMZ3btfRa71ZN7JdVdTUyO/3+/JfSwkpZGZ/9Eyk8Ww7rwy2TvWnXM8Z90ZGhryZF7HQWXp0qX6/d//fUlSWVmZent79d3vfld//dd/LUmKx+MqKChI7T84OKj8/PyzzhcIBBQIBKaN+/1+zxZIYsKnxGle9NzI5N55/YLk5ZpeSNysn0xed15j3blH75zxqlez/jsqxhglEgkVFxcrFAopFoulrhsfH1d3d7cqKipmezcAAGARcnRG5W/+5m+0ceNGFRUVaXR0VI899piee+457dmzRz6fT/X19YpGoyopKVFJSYmi0aiCwaBqa2u9qh8AAGQwR0Hl3Xff1S233KKjR49qxYoVuuKKK7Rnzx7V1NRIkhobG3Xq1CnV1dVpeHhY5eXl6urqUm5urifFAwCAzOYoqDzyyCMfeb3P51MkElEkEplNTQAAAJL4rR8AAGAxggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYK3u+CwBssXrbzzyZN7DEqHW9VBrZq8RpX1rnfuO+TWmdD5nHi3XnJdY0PowzKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKzlKKhs375dn/nMZ5Sbm6uLLrpIN9xwg1555ZUp+xhjFIlEVFhYqJycHFVXV6uvry+tRQMAgMXBUVDp7u7WHXfcoRdffFGxWEwffPCBwuGwTp48mdqntbVVbW1t6ujoUG9vr0KhkGpqajQ6Opr24gEAQGbLdrLznj17pmzv3LlTF110kQ4cOKA/+qM/kjFG7e3tam5u1ubNmyVJu3btUn5+vjo7O7V169b0VQ4AADKeo6DyYSdOnJAknXfeeZKk/v5+xeNxhcPh1D6BQEBVVVXq6emZMagkEgklEonU9sjIiCQpmUwqmUzOprxpJucLZJm0zrsYTPaM3jnnZe/S/RyZC4El594H1p17C7V3NqzpyRpsqGUh8apfPmOMq1VsjNGXvvQlDQ8Pa9++fZKknp4eVVZW6siRIyosLEzte9ttt2lgYEB79+6dNk8kElFLS8u08c7OTgWDQTelAQCAOTY2Nqba2lqdOHFCeXl5aZvX9RmVO++8U//1X/+lF154Ydp1Pp9vyrYxZtrYpKamJjU0NKS2R0ZGVFRUpHA4nNYHKp1Je7FYTN/en6XExMz1YGaBLKN7yibonQte9u5XkWvTOt9cKI1M/wfL2bDu3FuovbNhTU8eK2pqauT3++e7nAVjaGjIk3ldBZW/+Iu/0DPPPKPnn39eK1euTI2HQiFJUjweV0FBQWp8cHBQ+fn5M84VCAQUCASmjfv9fs8WSGLCp8TphfPEtQm9c8+L3i3EF1E3PWDdubfQemfTmvbyOJSJvOqVo2/9GGN055136sknn9S//du/qbi4eMr1xcXFCoVCisViqbHx8XF1d3eroqIiPRUDAIBFw9EZlTvuuEOdnZ366U9/qtzcXMXjcUnSihUrlJOTI5/Pp/r6ekWjUZWUlKikpETRaFTBYFC1tbWePAAAAJC5HAWVHTt2SJKqq6unjO/cuVO33nqrJKmxsVGnTp1SXV2dhoeHVV5erq6uLuXm5qalYAAAsHg4Cirn8gUhn8+nSCSiSCTitiYAAABJ/NYPAACwGEEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC3HQeX555/X9ddfr8LCQvl8Pj399NNTrjfGKBKJqLCwUDk5OaqurlZfX1+66gUAAIuI46By8uRJXXnllero6Jjx+tbWVrW1tamjo0O9vb0KhUKqqanR6OjorIsFAACLS7bTG2zcuFEbN26c8TpjjNrb29Xc3KzNmzdLknbt2qX8/Hx1dnZq69ats6sWAAAsKmn9jEp/f7/i8bjC4XBqLBAIqKqqSj09Pem8KwAAsAg4PqPyUeLxuCQpPz9/ynh+fr4GBgZmvE0ikVAikUhtj4yMSJKSyaSSyWQ6y0vNF8gyaZ13MZjsGb1zzsvepfs5MhcCS869D6w79xZq72xY05M12FDLQuJVv9IaVCb5fL4p28aYaWOTtm/frpaWlmnjXV1dCgaDXpSne8omPJl3MaB37nnRu927d6d9Tq+1rnd+G9adewutdzat6VgsNt8lLChjY2OezJvWoBIKhSSdObNSUFCQGh8cHJx2lmVSU1OTGhoaUtsjIyMqKipSOBxWXl5eOstTMplULBbTt/dnKTExc3DCzAJZRveUTdA7F7zs3a8i16Z1vrlQGtl7zvuy7txbqL2zYU1PHitqamrk9/vnu5wFY2hoyJN50xpUiouLFQqFFIvFdNVVV0mSxsfH1d3drfvvv3/G2wQCAQUCgWnjfr/fswWSmPApcXrhPHFtQu/c86J3C/FF1E0PWHfuLbTe2bSmvTwOZSKveuU4qLz//vv6n//5n9R2f3+/Dh06pPPOO0+XXHKJ6uvrFY1GVVJSopKSEkWjUQWDQdXW1qa1cAAAkPkcB5X9+/frc5/7XGp78m2bLVu26Ac/+IEaGxt16tQp1dXVaXh4WOXl5erq6lJubm76qgYAAIuC46BSXV0tY87+KXKfz6dIJKJIJDKbugAAAPitHwAAYC+CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADWIqgAAABrZc93AQDcW73tZ/NdApBWNqzpwBKj1vVSaWSvEqd9812OJ964b9N8l3DOOKMCAACsRVABAADWIqgAAABrEVQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYC2CCgAAsBZBBQAAWIugAgAArEVQAQAA1iKoAAAAaxFUAACAtQgqAADAWgQVAABgLYIKAACwFkEFAABYi6ACAACsRVABAADW8iyoPPjggyouLtayZcu0bt067du3z6u7AgAAGcqToPL444+rvr5ezc3NOnjwoD772c9q48aNevPNN724OwAAkKE8CSptbW362te+pq9//eu67LLL1N7erqKiIu3YscOLuwMAABkqO90Tjo+P68CBA9q2bduU8XA4rJ6enmn7JxIJJRKJ1PaJEyckScePH1cymUxrbclkUmNjY8pOZun0hC+tc2e67AmjsbEJeucCvXOP3rlH79xbDL0bGhpK+5zHjx+XJBlj0jpv2oPKsWPHdPr0aeXn508Zz8/PVzwen7b/9u3b1dLSMm28uLg43aVhlmrnu4AFjN65R+/co3fuZXrvLnjAu7mHhoa0YsWKtM2X9qAyyeebmkKNMdPGJKmpqUkNDQ2p7YmJCR0/flznn3/+jPvPxsjIiIqKivTWW28pLy8vrXNnOnrnHr1zj965R+/co3funDhxQpdcconOO++8tM6b9qBywQUXaMmSJdPOngwODk47yyJJgUBAgUBgytjHP/7xdJc1RV5eHovPJXrnHr1zj965R+/co3fuZGWl9+Ovaf8w7dKlS7Vu3TrFYrEp47FYTBUVFem+OwAAkME8eeunoaFBt9xyi8rKyrRhwwY99NBDevPNN3X77bd7cXcAACBDeRJUbrrpJg0NDek73/mOjh49qtLSUu3evVurVq3y4u7OWSAQ0N133z3trSb8bvTOPXrnHr1zj965R+/c8apvPpPu7xEBAACkCb/1AwAArEVQAQAA1iKoAAAAaxFUAACAtTIuqDz44IMqLi7WsmXLtG7dOu3bt+8j9+/u7ta6deu0bNkyrVmzRv/0T/80R5Xax0nvnnzySdXU1OjCCy9UXl6eNmzYoL17985htXZxuu4m/eIXv1B2drY+/elPe1ugxZz2LpFIqLm5WatWrVIgENCll16qf/mXf5mjau3htG+PPvqorrzySgWDQRUUFOjP/uzPPPm9F9s9//zzuv7661VYWCifz6enn376d96G48QZTnuXtuOEySCPPfaY8fv95uGHHzaHDx82d911l1m+fLkZGBiYcf/XX3/dBINBc9ddd5nDhw+bhx9+2Pj9fvOTn/xkjiuff057d9ddd5n777/fvPTSS+bVV181TU1Nxu/3m1/+8pdzXPn8c9q7Se+9955Zs2aNCYfD5sorr5ybYi3jpndf/OIXTXl5uYnFYqa/v9/853/+p/nFL34xh1XPP6d927dvn8nKyjLf/e53zeuvv2727dtnPvWpT5kbbrhhjiuff7t37zbNzc3miSeeMJLMU0899ZH7c5z4P057l67jREYFlfXr15vbb799ytgnP/lJs23bthn3b2xsNJ/85CenjG3dutVcffXVntVoK6e9m8nll19uWlpa0l2a9dz27qabbjJ/+7d/a+6+++5FG1Sc9u7ZZ581K1asMENDQ3NRnrWc9u0f/uEfzJo1a6aMfe973zMrV670rMaF4FwOthwnZnYuvZuJm+NExrz1Mz4+rgMHDigcDk8ZD4fD6unpmfE2//Ef/zFt/2uvvVb79+9XMpn0rFbbuOndh01MTGh0dDTtP0ZlO7e927lzp37zm9/o7rvv9rpEa7np3TPPPKOysjK1trbq4osv1tq1a/Wtb31Lp06dmouSreCmbxUVFXr77be1e/duGWP07rvv6ic/+Yk2bdo0FyUvaBwn0sftccKzX0+ea8eOHdPp06en/fBhfn7+tB9InBSPx2fc/4MPPtCxY8dUUFDgWb02cdO7D3vggQd08uRJfeUrX/GiRGu56d1rr72mbdu2ad++fcrOzpinoGNuevf666/rhRde0LJly/TUU0/p2LFjqqur0/HjxxfN51Tc9K2iokKPPvqobrrpJv32t7/VBx98oC9+8Yv6x3/8x7koeUHjOJE+bo8TGXNGZZLP55uybYyZNva79p9pfDFw2rtJP/rRjxSJRPT444/roosu8qo8q51r706fPq3a2lq1tLRo7dq1c1We1Zysu4mJCfl8Pj366KNav369rrvuOrW1tekHP/jBojqrIjnr2+HDh/WXf/mX+ru/+zsdOHBAe/bsUX9/P7+/do44TszebI4TGfPPuQsuuEBLliyZ9i+KwcHBaWl4UigUmnH/7OxsnX/++Z7Vahs3vZv0+OOP62tf+5p+/OMf65prrvGyTCs57d3o6Kj279+vgwcP6s4775R05uBrjFF2dra6urr0+c9/fk5qn29u1l1BQYEuvvhirVixIjV22WWXyRijt99+WyUlJZ7WbAM3fdu+fbsqKyv1V3/1V5KkK664QsuXL9dnP/tZ3XvvvZwV+AgcJ2ZvtseJjDmjsnTpUq1bt06xWGzKeCwWU0VFxYy32bBhw7T9u7q6VFZWJr/f71mttnHTO+lMQr711lvV2dm5aN/rdtq7vLw8vfzyyzp06FDqcvvtt+sTn/iEDh06pPLy8rkqfd65WXeVlZV655139P7776fGXn31VWVlZWnlypWe1msLN30bGxtTVtbUl/slS5ZI+r+zA5gZx4nZSctxwvFHdi02+ZW9Rx55xBw+fNjU19eb5cuXmzfeeMMYY8y2bdvMLbfcktp/8mtn3/zmN83hw4fNI488smi/dua0d52dnSY7O9t8//vfN0ePHk1d3nvvvfl6CPPGae8+bDF/68dp70ZHR83KlSvNl7/8ZdPX12e6u7tNSUmJ+frXvz5fD2FeOO3bzp07TXZ2tnnwwQfNb37zG/PCCy+YsrIys379+vl6CPNmdHTUHDx40Bw8eNBIMm1tbebgwYOpr3ZznDg7p71L13Eio4KKMcZ8//vfN6tWrTJLly41f/iHf2i6u7tT123ZssVUVVVN2f+5554zV111lVm6dKlZvXq12bFjxxxXbA8nvauqqjKSpl22bNky94VbwOm6+/8Wc1Axxnnvfv3rX5trrrnG5OTkmJUrV5qGhgYzNjY2x1XPP6d9+973vmcuv/xyk5OTYwoKCszNN99s3n777Tmuev79+7//+0e+dnGcODunvUvXccJnDOf9AACAnTLmMyoAACDzEFQAAIC1CCoAAMBaBBUAAGAtggoAALAWQQUAAFiLoAIAAKxFUAEAANYiqAAAAGsRVAAAgLUIKgAAwFoEFQAAYK3/Bbk/u51yEjhbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test.Survived_pred.hist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.41907883962730547"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.Survived_pred.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the survived column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived_pred</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>892</td>\n",
       "      <td>3</td>\n",
       "      <td>Kelly, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>34.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330911</td>\n",
       "      <td>7.8292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>0.250963</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>893</td>\n",
       "      <td>3</td>\n",
       "      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>363272</td>\n",
       "      <td>7.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.620510</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>894</td>\n",
       "      <td>2</td>\n",
       "      <td>Myles, Mr. Thomas Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>240276</td>\n",
       "      <td>9.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>0.219481</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>895</td>\n",
       "      <td>3</td>\n",
       "      <td>Wirz, Mr. Albert</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>315154</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.097688</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>896</td>\n",
       "      <td>3</td>\n",
       "      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3101298</td>\n",
       "      <td>12.2875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.639145</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Pclass                                          Name     Sex  \\\n",
       "0          892       3                              Kelly, Mr. James    male   \n",
       "1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n",
       "2          894       2                     Myles, Mr. Thomas Francis    male   \n",
       "3          895       3                              Wirz, Mr. Albert    male   \n",
       "4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n",
       "\n",
       "    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  Survived_pred  \\\n",
       "0  34.5      0      0   330911   7.8292   NaN        Q       0.250963   \n",
       "1  47.0      1      0   363272   7.0000   NaN        S       0.620510   \n",
       "2  62.0      0      0   240276   9.6875   NaN        Q       0.219481   \n",
       "3  27.0      0      0   315154   8.6625   NaN        S       0.097688   \n",
       "4  22.0      1      1  3101298  12.2875   NaN        S       0.639145   \n",
       "\n",
       "   Survived  \n",
       "0         0  \n",
       "1         1  \n",
       "2         0  \n",
       "3         0  \n",
       "4         1  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['Survived'] = [ 1 if element > 0.5 else 0 for element in preds.squeeze()]\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the submission file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = test[['PassengerId', 'Survived']]\n",
    "submission.to_csv('submission-3layers.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can I say, a new High Score:\n",
    "* **Score: 0.77033** (2-layers)\n",
    "* **Score: 0.77272** (3-layers)\n",
    "\n",
    "![score077272](score077272.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is quite surprising: The self-written algorithm is better than the Fast.AI one...\n",
    "\n",
    "Nonetheless it seems to hit a ceiling.\n",
    "\n",
    "I see additional ways to optimize this, for example, the layers are actually not a neural net, because they do not feed into each other, but they are in parallel.\n",
    "\n",
    "But my goal is not to optimize competition, but to participate in my first kaggle competition, and to re-visit the topic of gradient descent.\n",
    "\n",
    "I will sure return to this dataset, but that is for another time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "d9da906b64701e68312bc07fbc15a3a13814f930718c2c6b0e41a29d035806a3"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}