utf-8''AI4M_C1_W1_lecture_ex_04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AI for Medicine Course 1 Week 1 lecture exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Patient Overlap and Data Leakage\n",
    "\n",
    "Patient overlap in medical data is a part of a more general problem in machine learning called **data leakage**.  To identify patient overlap in this week's graded assignment, you'll check to see if a patient's ID appears in both the training set and the test set. You should also verify that you don't have patient overlap in the training and validation sets, which is what you'll do here.\n",
    "\n",
    "Below is a simple example showing how you can check for and remove patient overlap in your training and validations sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import os\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in the data from a csv file\n",
    "\n",
    "First, you'll read in your training and validation datasets from csv files. Run the next two cells to read these csvs into `pandas` dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1000 rows and 16 columns in the training dataframe\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Image</th>\n",
       "      <th>Atelectasis</th>\n",
       "      <th>Cardiomegaly</th>\n",
       "      <th>Consolidation</th>\n",
       "      <th>Edema</th>\n",
       "      <th>Effusion</th>\n",
       "      <th>Emphysema</th>\n",
       "      <th>Fibrosis</th>\n",
       "      <th>Hernia</th>\n",
       "      <th>Infiltration</th>\n",
       "      <th>Mass</th>\n",
       "      <th>Nodule</th>\n",
       "      <th>PatientId</th>\n",
       "      <th>Pleural_Thickening</th>\n",
       "      <th>Pneumonia</th>\n",
       "      <th>Pneumothorax</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00008270_015.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8270</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00029855_001.png</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29855</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00001297_000.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1297</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00012359_002.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12359</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00017951_001.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17951</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Image  Atelectasis  Cardiomegaly  Consolidation  Edema  \\\n",
       "0  00008270_015.png            0             0              0      0   \n",
       "1  00029855_001.png            1             0              0      0   \n",
       "2  00001297_000.png            0             0              0      0   \n",
       "3  00012359_002.png            0             0              0      0   \n",
       "4  00017951_001.png            0             0              0      0   \n",
       "\n",
       "   Effusion  Emphysema  Fibrosis  Hernia  Infiltration  Mass  Nodule  \\\n",
       "0         0          0         0       0             0     0       0   \n",
       "1         1          0         0       0             1     0       0   \n",
       "2         0          0         0       0             0     0       0   \n",
       "3         0          0         0       0             0     0       0   \n",
       "4         0          0         0       0             1     0       0   \n",
       "\n",
       "   PatientId  Pleural_Thickening  Pneumonia  Pneumothorax  \n",
       "0       8270                   0          0             0  \n",
       "1      29855                   0          0             0  \n",
       "2       1297                   1          0             0  \n",
       "3      12359                   0          0             0  \n",
       "4      17951                   0          0             0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read csv file containing training data\n",
    "train_df = pd.read_csv(\"nih/train-small.csv\")\n",
    "# Print first 5 rows\n",
    "print(f'There are {train_df.shape[0]} rows and {train_df.shape[1]} columns in the training dataframe')\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 109 rows and 16 columns in the validation dataframe\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Image</th>\n",
       "      <th>Atelectasis</th>\n",
       "      <th>Cardiomegaly</th>\n",
       "      <th>Consolidation</th>\n",
       "      <th>Edema</th>\n",
       "      <th>Effusion</th>\n",
       "      <th>Emphysema</th>\n",
       "      <th>Fibrosis</th>\n",
       "      <th>Hernia</th>\n",
       "      <th>Infiltration</th>\n",
       "      <th>Mass</th>\n",
       "      <th>Nodule</th>\n",
       "      <th>PatientId</th>\n",
       "      <th>Pleural_Thickening</th>\n",
       "      <th>Pneumonia</th>\n",
       "      <th>Pneumothorax</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00027623_007.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27623</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00028214_000.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28214</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00022764_014.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22764</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00020649_001.png</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20649</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00022283_023.png</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22283</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Image  Atelectasis  Cardiomegaly  Consolidation  Edema  \\\n",
       "0  00027623_007.png            0             0              0      1   \n",
       "1  00028214_000.png            0             0              0      0   \n",
       "2  00022764_014.png            0             0              0      0   \n",
       "3  00020649_001.png            1             0              0      0   \n",
       "4  00022283_023.png            0             0              0      0   \n",
       "\n",
       "   Effusion  Emphysema  Fibrosis  Hernia  Infiltration  Mass  Nodule  \\\n",
       "0         1          0         0       0             0     0       0   \n",
       "1         0          0         0       0             0     0       0   \n",
       "2         0          0         0       0             0     0       0   \n",
       "3         1          0         0       0             0     0       0   \n",
       "4         0          0         0       0             0     0       0   \n",
       "\n",
       "   PatientId  Pleural_Thickening  Pneumonia  Pneumothorax  \n",
       "0      27623                   0          0             0  \n",
       "1      28214                   0          0             0  \n",
       "2      22764                   0          0             0  \n",
       "3      20649                   0          0             0  \n",
       "4      22283                   0          0             0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read csv file containing validation data\n",
    "valid_df = pd.read_csv(\"nih/valid-small.csv\")\n",
    "# Print first 5 rows\n",
    "print(f'There are {valid_df.shape[0]} rows and {valid_df.shape[1]} columns in the validation dataframe')\n",
    "valid_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract and compare the PatientId columns from the train and validation sets\n",
    "By running the next four cells you will do the following:\n",
    "1. Extract patient IDs from the train and validation sets\n",
    "2. Convert these arrays of numbers into `set()` datatypes for easy comparison\n",
    "3. Identify patient overlap in the intersection of the two sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract patient id's for the training set\n",
    "ids_train = train_df.PatientId.values\n",
    "# Extract patient id's for the validation set\n",
    "ids_valid = valid_df.PatientId.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 928 unique Patient IDs in the training set\n",
      "There are 97 unique Patient IDs in the training set\n"
     ]
    }
   ],
   "source": [
    "# Create a \"set\" datastructure of the training set id's to identify unique id's\n",
    "ids_train_set = set(ids_train)\n",
    "print(f'There are {len(ids_train_set)} unique Patient IDs in the training set')\n",
    "# Create a \"set\" datastructure of the validation set id's to identify unique id's\n",
    "ids_valid_set = set(ids_valid)\n",
    "print(f'There are {len(ids_valid_set)} unique Patient IDs in the training set')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 11 Patient IDs in both the training and validation sets\n",
      "\n",
      "These patients are in both the training and validation datasets:\n",
      "[20290, 27618, 9925, 10888, 22764, 19981, 18253, 4461, 28208, 8760, 7482]\n"
     ]
    }
   ],
   "source": [
    "# Identify patient overlap by looking at the intersection between the sets\n",
    "patient_overlap = list(ids_train_set.intersection(ids_valid_set))\n",
    "n_overlap = len(patient_overlap)\n",
    "print(f'There are {n_overlap} Patient IDs in both the training and validation sets')\n",
    "print('')\n",
    "print(f'These patients are in both the training and validation datasets:')\n",
    "print(f'{patient_overlap}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify rows (indices) of overlapping patients and remove from either the train or validation set\n",
    "Run the next two cells to do the following:\n",
    "1. Create lists of the overlapping row numbers in both the training and validation sets. \n",
    "2. Drop the overlapping patient records from the validation set (could also choose to drop from train set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are the indices of overlapping patients in the training set: \n",
      "[306, 186, 797, 98, 408, 917, 327, 913, 10, 51, 276]\n",
      "These are the indices of overlapping patients in the validation set: \n",
      "[104, 88, 65, 13, 2, 41, 56, 70, 26, 75, 20, 52, 55]\n"
     ]
    }
   ],
   "source": [
    "train_overlap_idxs = []\n",
    "valid_overlap_idxs = []\n",
    "for idx in range(n_overlap):\n",
    "    train_overlap_idxs.extend(train_df.index[train_df['PatientId'] == patient_overlap[idx]].tolist())\n",
    "    valid_overlap_idxs.extend(valid_df.index[valid_df['PatientId'] == patient_overlap[idx]].tolist())\n",
    "    \n",
    "print(f'These are the indices of overlapping patients in the training set: ')\n",
    "print(f'{train_overlap_idxs}')\n",
    "print(f'These are the indices of overlapping patients in the validation set: ')\n",
    "print(f'{valid_overlap_idxs}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the overlapping rows from the validation set\n",
    "valid_df.drop(valid_overlap_idxs, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check that everything worked as planned by rerunning the patient ID comparison between train and validation sets.\n",
    "\n",
    "When you run the next two cells you should see that there are now fewer records in the validation set and that the overlap problem has been removed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 86 unique Patient IDs in the training set\n"
     ]
    }
   ],
   "source": [
    "# Extract patient id's for the validation set\n",
    "ids_valid = valid_df.PatientId.values\n",
    "# Create a \"set\" datastructure of the validation set id's to identify unique id's\n",
    "ids_valid_set = set(ids_valid)\n",
    "print(f'There are {len(ids_valid_set)} unique Patient IDs in the training set')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 0 Patient IDs in both the training and validation sets\n"
     ]
    }
   ],
   "source": [
    "# Identify patient overlap by looking at the intersection between the sets\n",
    "patient_overlap = list(ids_train_set.intersection(ids_valid_set))\n",
    "n_overlap = len(patient_overlap)\n",
    "print(f'There are {n_overlap} Patient IDs in both the training and validation sets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations! You removed overlapping patients from the validation set! \n",
    "\n",
    "You could have just as well removed them from the training set. \n",
    "\n",
    "Always be sure to check for patient overlap in your train, validation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}