0102_Pandas_Exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb0915a2",
   "metadata": {},
   "source": [
    "# Pandas: Exercise\n",
    "\n",
    "This is an exercise that is intended to complement the script. Idea is that students go through the script and then try to apply the learned content.\n",
    "\n",
    "Though one possible solution will be provided it is important to note that there might be multiple approaches to achieve the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "83a38b84",
   "metadata": {},
   "outputs": [],
   "source": [
    "#from IPython.core.display import HTML\n",
    "#HTML('<style>.input, .jp-InputArea {display: none !important}</style>')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c3f9cc",
   "metadata": {},
   "source": [
    "## Panda Basics\n",
    "\n",
    "Learning the basics of Pandas.\n",
    "\n",
    "First import Numpy and Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fb0ecd28",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b78186",
   "metadata": {},
   "source": [
    "Load/Download the 'Titanic' data set from the following GitHub repo: https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv. A description of the data can be found here: https://www.kaggle.com/c/titanic/data\n",
    "\n",
    "Use the `pd.read_csv()` function to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ce639148",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ead3c2d",
   "metadata": {},
   "source": [
    "Inspect the imported data frame: Print the first 5 rows of the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8e9a38cc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S  Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C  First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S  Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S  First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S  Third   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \n",
       "0    man        True  NaN  Southampton    no  False  \n",
       "1  woman       False    C    Cherbourg   yes  False  \n",
       "2  woman       False  NaN  Southampton   yes   True  \n",
       "3  woman       False    C  Southampton   yes  False  \n",
       "4    man        True  NaN  Southampton    no   True  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "197c9f14",
   "metadata": {},
   "source": [
    "Check the data types (dtypes) of the different columns, see how many non-null values there are per column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e4ce3e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "survived         int64\n",
       "pclass           int64\n",
       "sex             object\n",
       "age            float64\n",
       "sibsp            int64\n",
       "parch            int64\n",
       "fare           float64\n",
       "embarked        object\n",
       "class           object\n",
       "who             object\n",
       "adult_male        bool\n",
       "deck            object\n",
       "embark_town     object\n",
       "alive           object\n",
       "alone             bool\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d67bcc23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "survived         0\n",
       "pclass           0\n",
       "sex              0\n",
       "age            177\n",
       "sibsp            0\n",
       "parch            0\n",
       "fare             0\n",
       "embarked         2\n",
       "class            0\n",
       "who              0\n",
       "adult_male       0\n",
       "deck           688\n",
       "embark_town      2\n",
       "alive            0\n",
       "alone            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d3f55340",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 15 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   survived     891 non-null    int64  \n",
      " 1   pclass       891 non-null    int64  \n",
      " 2   sex          891 non-null    object \n",
      " 3   age          714 non-null    float64\n",
      " 4   sibsp        891 non-null    int64  \n",
      " 5   parch        891 non-null    int64  \n",
      " 6   fare         891 non-null    float64\n",
      " 7   embarked     889 non-null    object \n",
      " 8   class        891 non-null    object \n",
      " 9   who          891 non-null    object \n",
      " 10  adult_male   891 non-null    bool   \n",
      " 11  deck         203 non-null    object \n",
      " 12  embark_town  889 non-null    object \n",
      " 13  alive        891 non-null    object \n",
      " 14  alone        891 non-null    bool   \n",
      "dtypes: bool(2), float64(2), int64(4), object(7)\n",
      "memory usage: 92.4+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3e1763",
   "metadata": {},
   "source": [
    "Remove all entries where the age is unknonw and then select the columns `age`, `sex` `pclass`, and `survived` and assign it to a separate variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "55791f9d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>885</th>\n",
       "      <td>39.0</td>\n",
       "      <td>female</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>29.1250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>27.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>19.0</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>26.0</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>32.0</td>\n",
       "      <td>male</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>714 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      age     sex  pclass  survived     fare\n",
       "0    22.0    male       3         0   7.2500\n",
       "1    38.0  female       1         1  71.2833\n",
       "2    26.0  female       3         1   7.9250\n",
       "3    35.0  female       1         1  53.1000\n",
       "4    35.0    male       3         0   8.0500\n",
       "..    ...     ...     ...       ...      ...\n",
       "885  39.0  female       3         0  29.1250\n",
       "886  27.0    male       2         0  13.0000\n",
       "887  19.0  female       1         1  30.0000\n",
       "889  26.0    male       1         1  30.0000\n",
       "890  32.0    male       3         0   7.7500\n",
       "\n",
       "[714 rows x 5 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Median Age of all passangers\n",
    "df_sub = df[df['age'].notna()]\n",
    "# df.dropna(subset=['age'])   Alternative to .notna() \n",
    "\n",
    "df_sub = df_sub[['age', 'sex', 'pclass', 'survived', 'fare']]\n",
    "df_sub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95394111",
   "metadata": {},
   "source": [
    "Select the fourth and fifth row, but display only columns age and sex. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cd20e038",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age     sex\n",
       "3  35.0  female\n",
       "4  35.0    male"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sub.iloc[3:5, :2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b056d1",
   "metadata": {},
   "source": [
    "Now calculate:\n",
    "* The median age of all passangers\n",
    "* The share of passangers per class (`pclass`)\n",
    "* The average age of all males, females\n",
    "* The average age for all males, females, but this time per class \n",
    "* The average fare paid by all passangers between 50 and 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3da3b2bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Median age of all passangers\n",
    "df_sub['age'].median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "088bf926",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    0.497199\n",
       "1    0.260504\n",
       "2    0.242297\n",
       "Name: pclass, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Share of passangers per class\n",
    "df_sub['pclass'].value_counts() / df_sub['pclass'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "26c0164b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex\n",
       "female    27.915709\n",
       "male      30.726645\n",
       "Name: age, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Average age of all males, females\n",
    "df_sub.groupby(['sex']).mean()['age']\n",
    "# df_sub.groupby(['sex'])['age'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d4ef4580",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex     pclass\n",
       "female  1         34.611765\n",
       "        2         28.722973\n",
       "        3         21.750000\n",
       "male    1         41.281386\n",
       "        2         30.740707\n",
       "        3         26.507589\n",
       "Name: age, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Average age of all males, females by class\n",
    "df_sub.groupby(['sex', 'pclass']).mean()['age']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a5b72696",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.69846578947369"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sub[(df_sub['age'] > 50) & (df_sub['age'] < 60)]['fare'].mean()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}