subject_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis of Subject Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Content under Creative Commons license CC-BY-NC-SA 4.0   \n",
    "Code under GNU-GPL v3 License  \n",
    "By [Serena Bonaretti](https://sbonaretti.github.io/)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of this notebook is to calculate descriptive statistics for a group of subjects from tabular data  \n",
    "This notebook can be attached to the *Material* paragraph of your paper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Structure of the subject dataframe (i.e. table)*  \n",
    "Each *row* corresponds to a *subject*, and each *column* corresponds to a *characteristic*. E.g.:\n",
    "\n",
    "| subject_id |      age      |  gender | anatomy | laterality\n",
    "|:----------:|:-------------:|:-------:| :------:| :---------:\n",
    "| ID_01      | 66            | f       | hip     | r\n",
    "| ID_02      | 53            | m       | ankle   | l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What you do:**  \n",
    "To use your data, you can modify the table template following these steps:\n",
    "- Download `subjects_templates.csv` (open-source version) or `subjects_templates.xlsx` (proprietary version) \n",
    "- Do not change the column headers (e.g. `subject_id`, `age`, etc.), but\n",
    "- Change the table content according to your own data. You can also add new columns, and add new queries by adapting the code below\n",
    "- Add the path to the dataframe in the code below (look for the arrow ->)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What the notebook does:**\n",
    "- Reads a table containing data about subjects, such as gender, age, anatomy, anatomy laterality. Data are organized in a tabular file, which can be in `.csv` (open file format) or `.xlsx` (proprietary file format)  \n",
    "- Gets:\n",
    "  - Number of subject\n",
    "  - Average, standard deviation, max, and min age \n",
    "  - Number of subjects per gender\n",
    "  - Anatomies\n",
    "  - Number of anatomies per laterality (left/right)  \n",
    "- Prints out dependencies for reproducibility\n",
    "\n",
    "To read and query the data, it uses the python package `pandas`  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the tabular data using the pandas function `read_csv` (or `read_excel`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"./data/tabular/subjects_template.csv\")\n",
    "\n",
    "# if your table is in .xlsx, use the command below and comment out the command above\n",
    "#df = pd.read_excel(\"subjects_template.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the table. Please note that the index starts from 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject_id</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>anatomy</th>\n",
       "      <th>laterality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ID_01</td>\n",
       "      <td>66</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ID_02</td>\n",
       "      <td>53</td>\n",
       "      <td>m</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ID_03</td>\n",
       "      <td>50</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ID_04</td>\n",
       "      <td>52</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ID_05</td>\n",
       "      <td>73</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ID_06</td>\n",
       "      <td>67</td>\n",
       "      <td>m</td>\n",
       "      <td>ankle</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ID_07</td>\n",
       "      <td>71</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ID_08</td>\n",
       "      <td>56</td>\n",
       "      <td>f</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ID_09</td>\n",
       "      <td>60</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ID_10</td>\n",
       "      <td>78</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ID_11</td>\n",
       "      <td>76</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>ID_12</td>\n",
       "      <td>72</td>\n",
       "      <td>m</td>\n",
       "      <td>ankle</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>ID_13</td>\n",
       "      <td>54</td>\n",
       "      <td>f</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>ID_14</td>\n",
       "      <td>79</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ID_15</td>\n",
       "      <td>51</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ID_16</td>\n",
       "      <td>59</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ID_17</td>\n",
       "      <td>62</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ID_18</td>\n",
       "      <td>63</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ID_19</td>\n",
       "      <td>65</td>\n",
       "      <td>f</td>\n",
       "      <td>ankle</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>ID_20</td>\n",
       "      <td>58</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>ID_21</td>\n",
       "      <td>61</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>ID_22</td>\n",
       "      <td>75</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>ID_23</td>\n",
       "      <td>64</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>ID_24</td>\n",
       "      <td>57</td>\n",
       "      <td>f</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>ID_25</td>\n",
       "      <td>55</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>ID_26</td>\n",
       "      <td>69</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>ID_27</td>\n",
       "      <td>72</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>ID_28</td>\n",
       "      <td>68</td>\n",
       "      <td>f</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>ID_29</td>\n",
       "      <td>74</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>ID_30</td>\n",
       "      <td>77</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subject_id  age gender anatomy laterality\n",
       "0       ID_01   66      f     hip          r\n",
       "1       ID_02   53      m   ankle          l\n",
       "2       ID_03   50      f     hip          r\n",
       "3       ID_04   52      f     hip          l\n",
       "4       ID_05   73      m     hip          l\n",
       "5       ID_06   67      m   ankle          r\n",
       "6       ID_07   71      m     hip          r\n",
       "7       ID_08   56      f   ankle          l\n",
       "8       ID_09   60      m     hip          l\n",
       "9       ID_10   78      f     hip          r\n",
       "10      ID_11   76      m     hip          r\n",
       "11      ID_12   72      m   ankle          r\n",
       "12      ID_13   54      f   ankle          l\n",
       "13      ID_14   79      f     hip          l\n",
       "14      ID_15   51      f     hip          r\n",
       "15      ID_16   59      m     hip          l\n",
       "16      ID_17   62      m     hip          r\n",
       "17      ID_18   63      m     hip          l\n",
       "18      ID_19   65      f   ankle          r\n",
       "19      ID_20   58      m     hip          l\n",
       "20      ID_21   61      f     hip          l\n",
       "21      ID_22   75      f     hip          r\n",
       "22      ID_23   64      m     hip          r\n",
       "23      ID_24   57      f   ankle          l\n",
       "24      ID_25   55      m     hip          l\n",
       "25      ID_26   69      f     hip          r\n",
       "26      ID_27   72      m     hip          r\n",
       "27      ID_28   68      f   ankle          l\n",
       "28      ID_29   74      m     hip          r\n",
       "29      ID_30   77      f     hip          l"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the table is too long, you can:\n",
    "- Make the table scollable by right-clicking on the table, and then `Enable scolling to output`, or\n",
    "- Show only the first five rows with the command: `df.head()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject_id</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>anatomy</th>\n",
       "      <th>laterality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ID_01</td>\n",
       "      <td>66</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ID_02</td>\n",
       "      <td>53</td>\n",
       "      <td>m</td>\n",
       "      <td>ankle</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ID_03</td>\n",
       "      <td>50</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ID_04</td>\n",
       "      <td>52</td>\n",
       "      <td>f</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ID_05</td>\n",
       "      <td>73</td>\n",
       "      <td>m</td>\n",
       "      <td>hip</td>\n",
       "      <td>l</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  subject_id  age gender anatomy laterality\n",
       "0      ID_01   66      f     hip          r\n",
       "1      ID_02   53      m   ankle          l\n",
       "2      ID_03   50      f     hip          r\n",
       "3      ID_04   52      f     hip          l\n",
       "4      ID_05   73      m     hip          l"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get number of subjects\n",
    "The number of subjects coincides with the number of rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    }
   ],
   "source": [
    "n_of_rows = df.shape[0]\n",
    "print (n_of_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Age: Get average, standard deviation, max, and min \n",
    "Calculate average, standard deviation, max, and min for the values in the column `age`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average age is: 64.57\n",
      "The standard deviation is: 8.85\n",
      "The max age is: 79\n",
      "The min age is: 50\n"
     ]
    }
   ],
   "source": [
    "# average\n",
    "average_age = df[\"age\"].mean()\n",
    "print (\"The average age is: \" + str(round(average_age,2))) # round(average_age,2) rounds to 2 decimals \n",
    "\n",
    "# standard deviation\n",
    "stddev_age = df[\"age\"].std()\n",
    "print (\"The standard deviation is: \" + str(round(stddev_age,2))) \n",
    "\n",
    "# max\n",
    "max_age = df[\"age\"].max()\n",
    "print (\"The max age is: \" + str(round(max_age,2))) \n",
    "\n",
    "# min\n",
    "min_age = df[\"age\"].min()\n",
    "print (\"The min age is: \" + str(round(min_age,2))) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gender: Get number of female and male\n",
    "Count the unique values in the column `gender`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m    15\n",
      "f    15\n",
      "Name: gender, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "n_of_sex = df[\"gender\"].value_counts()\n",
    "print (n_of_sex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Anatomy: Get organ \n",
    "Count the unique values in the column `anatomy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hip      22\n",
      "ankle     8\n",
      "Name: anatomy, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "n_of_anatomy = df[\"anatomy\"].value_counts()\n",
    "print(n_of_anatomy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Laterality: Get number of right and left\n",
    "Count the unique values in the column `laterality`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r    15\n",
      "l    15\n",
      "Name: laterality, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "n_of_laterality = df[\"laterality\"].value_counts()\n",
    "print (n_of_laterality)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "## Dependencies\n",
    "Dependencies keep track of the computational environment, so that we can make our workflows reproducible.  \n",
    "Here we use the package watermark. If you haven't installed it yet, go to your terminal, and type `pip install watermark`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPython 3.7.6\n",
      "IPython 7.13.0\n",
      "\n",
      "pandas 1.0.3\n",
      "watermark 2.0.2\n",
      "\n",
      "compiler   : Clang 4.0.1 (tags/RELEASE_401/final)\n",
      "system     : Darwin\n",
      "release    : 19.4.0\n",
      "machine    : x86_64\n",
      "processor  : i386\n",
      "CPU cores  : 4\n",
      "interpreter: 64bit\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -v -m -p pandas,watermark"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}