feature_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "\n",
    "This notebook takes the raw clinical data and builds corresponding features. There are a mix of continuous and categorical variables from the clinical data, and some contain more missing values than others. \n",
    "\n",
    "The general strategy is to window the data into 10 hour blocks, with a one hour prediction of sepsis/no sepsis. For each window, the following variables are retained as time series:\n",
    "<br />-HR\n",
    "<br />-MAP\n",
    "<br />-O2Sat\n",
    "<br />-SBP\n",
    "<br />-Resp\n",
    "\n",
    "The remainder of the variables are summarized as a single value, the median of the ten values in that window. This is a strategy to deal with the fact that there may be > 90% missing data for some variables.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pdb\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "from google_drive_downloader import GoogleDriveDownloader as gdd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the feats folder, or remove it if it exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    if os.path.exists('feats'):\n",
    "        shutil.rmtree('feats')\n",
    "    os.makedirs('feats')\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading 1AmIJQ2oo7Cy1w32T8d1v-rXiJKM0wZE- into ./combined.pkl... Done.\n"
     ]
    }
   ],
   "source": [
    "#the link to download combined.pkl\n",
    "file_id = '1AmIJQ2oo7Cy1w32T8d1v-rXiJKM0wZE-'\n",
    "\n",
    "#load in the data and labels\n",
    "gdd.download_file_from_google_drive(file_id=file_id, dest_path='./combined.pkl')\n",
    "\n",
    "df = pd.read_pickle('combined.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage Missing:\n",
      "HR                  0.098826\n",
      "O2Sat               0.130611\n",
      "Temp                0.661627\n",
      "SBP                 0.145770\n",
      "MAP                 0.124513\n",
      "DBP                 0.313459\n",
      "Resp                0.153546\n",
      "EtCO2               0.962868\n",
      "BaseExcess          0.945790\n",
      "HCO3                0.958106\n",
      "FiO2                0.916658\n",
      "pH                  0.930697\n",
      "PaCO2               0.944401\n",
      "SaO2                0.965494\n",
      "AST                 0.983776\n",
      "BUN                 0.931344\n",
      "Alkalinephos        0.983932\n",
      "Calcium             0.941161\n",
      "Chloride            0.954603\n",
      "Creatinine          0.939044\n",
      "Bilirubin_direct    0.998074\n",
      "Glucose             0.828943\n",
      "Lactate             0.973299\n",
      "Magnesium           0.936896\n",
      "Phosphate           0.959863\n",
      "Potassium           0.906891\n",
      "Bilirubin_total     0.985092\n",
      "TroponinI           0.990477\n",
      "Hct                 0.911460\n",
      "Hgb                 0.926176\n",
      "PTT                 0.970559\n",
      "WBC                 0.935932\n",
      "Fibrinogen          0.993402\n",
      "Platelets           0.940595\n",
      "Age                 0.000000\n",
      "Gender              0.000000\n",
      "Unit1               0.394251\n",
      "Unit2               0.394251\n",
      "HospAdmTime         0.000005\n",
      "ICULOS              0.000000\n",
      "SepsisLabel         0.000000\n",
      "patient             0.000000\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "#get the percentage missing for each column\n",
    "print('Percentage Missing:')\n",
    "print(df.isna().sum()/len(df))\n",
    "\n",
    "#columns to drop\n",
    "#drop Unit2 because Unit1 and Unit2 are mutually exclusive\n",
    "#drop ICULOS as it's basically just an index\n",
    "cols_to_drop = ['Unit2', 'ICULOS']\n",
    "df = df.drop(cols_to_drop, axis=1)\n",
    "\n",
    "#columns with < 15% missing data, and continuous data. these will be retained as time series\n",
    "cols_cont = ['HR', 'MAP', 'O2Sat', 'SBP', 'Resp']\n",
    "\n",
    "#columns with continuous data and > 15% missing data\n",
    "cols_to_bin = ['Unit1', 'Gender', 'HospAdmTime', 'Age', 'DBP', 'Temp', 'Glucose', 'Potassium', 'Hct', 'FiO2', 'Hgb', 'pH', 'BUN', 'WBC', 'Magnesium', 'Creatinine', 'Platelets', 'Calcium', 'PaCO2', 'BaseExcess', 'Chloride', 'HCO3', 'Phosphate', 'EtCO2', 'SaO2', 'PTT', 'Lactate', 'AST', 'Alkalinephos', 'Bilirubin_total', 'TroponinI', 'Fibrinogen', 'Bilirubin_direct']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the mean/std for standardization for each variable. Leave out a random 8000 patients as the test set. In other words don't include a random 4000 patients when calculating the mean/std scaling parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_training_data = df['patient'].unique()\n",
    "np.random.shuffle(patients_training_data)\n",
    "patients_training_data = patients_training_data[0:-6000]\n",
    "\n",
    "df_mean_std = df[df['patient'].isin(patients_training_data)].describe().loc[['mean', 'std']]\n",
    "df_mean_std.to_pickle('mean_std_scaling.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive training examples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "23770"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Number of positive training examples:')\n",
    "sum(df[df['patient'].isin(patients_training_data)]['SepsisLabel']==1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop through each subject and grab a window of 10 hours, with an output label associated with the 11th hour (ie predict one hour ahead). Note that you will need to create a directory called \"feats\" for this to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "patient 500\n",
      "patient 1000\n",
      "patient 1500\n",
      "patient 2000\n",
      "patient 2500\n",
      "patient 3000\n",
      "patient 3500\n",
      "patient 4000\n",
      "patient 4500\n",
      "patient 5000\n",
      "patient 5500\n",
      "patient 6000\n",
      "patient 6500\n",
      "patient 7000\n",
      "patient 7500\n",
      "patient 8000\n",
      "patient 8500\n",
      "patient 9000\n",
      "patient 9500\n",
      "patient 10000\n",
      "patient 10500\n",
      "patient 11000\n",
      "patient 11500\n",
      "patient 12000\n",
      "patient 12500\n",
      "patient 13000\n",
      "patient 13500\n",
      "patient 14000\n",
      "patient 14500\n",
      "patient 15000\n",
      "patient 15500\n",
      "patient 16000\n",
      "patient 16500\n",
      "patient 17000\n",
      "patient 17500\n",
      "patient 18000\n",
      "patient 18500\n",
      "patient 19000\n",
      "patient 19500\n",
      "patient 20000\n",
      "patient 20500\n",
      "patient 500\n",
      "patient 1000\n",
      "patient 1500\n",
      "patient 2000\n",
      "patient 2500\n",
      "patient 3000\n",
      "patient 3500\n",
      "patient 4000\n",
      "patient 4500\n",
      "patient 5000\n",
      "patient 5500\n",
      "patient 6000\n",
      "patient 6500\n",
      "patient 7000\n",
      "patient 7500\n",
      "patient 8000\n",
      "patient 8500\n",
      "patient 9000\n",
      "patient 9500\n",
      "patient 10000\n",
      "patient 10500\n",
      "patient 11000\n",
      "patient 11500\n",
      "patient 12000\n",
      "patient 12500\n",
      "patient 13000\n",
      "patient 13500\n",
      "patient 14000\n",
      "patient 14500\n",
      "patient 15000\n",
      "patient 15500\n",
      "patient 16000\n",
      "patient 16500\n",
      "patient 17000\n",
      "patient 17500\n",
      "patient 18000\n",
      "patient 18500\n",
      "patient 19000\n",
      "patient 19500\n",
      "patient 20000\n"
     ]
    }
   ],
   "source": [
    "#loop through each patient at a time\n",
    "save_count = 0\n",
    "windowed_df_list = []\n",
    "grouped_by_patient = df.groupby('patient')\n",
    "for patient, group in grouped_by_patient:\n",
    "    #print(patient)\n",
    "    group = group.reset_index(drop=True)\n",
    "\n",
    "    #backfill any missing values for the continuous variables with < 15% missing data\n",
    "    group = group.assign(HR=group['HR'].fillna(method='bfill').fillna(method='ffill'))\n",
    "    group = group.assign(MAP=group['MAP'].fillna(method='bfill').fillna(method='ffill'))\n",
    "    group = group.assign(O2Sat=group['O2Sat'].fillna(method='bfill').fillna(method='ffill'))\n",
    "    group = group.assign(SBP=group['SBP'].fillna(method='bfill').fillna(method='ffill'))\n",
    "    group = group.assign(Resp=group['Resp'].fillna(method='bfill').fillna(method='ffill'))\n",
    "    \n",
    "    #standardize the continous data\n",
    "    group = group.assign(HR=(group['HR']-df_mean_std['HR']['mean'])/(df_mean_std['HR']['std']))\n",
    "    group = group.assign(MAP=(group['MAP']-df_mean_std['MAP']['mean'])/(df_mean_std['MAP']['std']))\n",
    "    group = group.assign(O2Sat=(group['O2Sat']-df_mean_std['O2Sat']['mean'])/(df_mean_std['O2Sat']['std']))\n",
    "    group = group.assign(SBP=(group['SBP']-df_mean_std['SBP']['mean'])/(df_mean_std['SBP']['std']))\n",
    "    group = group.assign(Resp=(group['Resp']-df_mean_std['Resp']['mean'])/(df_mean_std['Resp']['std']))\n",
    "\n",
    "    #generate windows of 10 hours, predicting one sample into the future\n",
    "    windowed_data = []\n",
    "    N = len(group)\n",
    "    win_len = 10\n",
    "    pred_len = 1\n",
    "    i = 0\n",
    "    while(i+win_len+pred_len <= N):\n",
    "        tmp_data = group.iloc[i:i+win_len]\n",
    "        tmp_label = group.iloc[i+win_len:i+win_len+pred_len]\n",
    "        tmp_label = int(any(tmp_label['SepsisLabel']))\n",
    "        tmp_patient = patient\n",
    "\n",
    "        #slide the window forward\n",
    "        i = i+1\n",
    "\n",
    "        #get all the continuous variables into one group\n",
    "        X_cont = tmp_data[cols_cont]\n",
    "        X_cont = X_cont.values\n",
    "\n",
    "        #if any of the continuous variables is nan (in other words, there wasn't even a single value to \n",
    "        #backfill/forwardfill) then just skip this window\n",
    "        if np.isnan(X_cont).any(): continue\n",
    "\n",
    "        #process each of the variables to be binned\n",
    "        X_binned_dict = {}\n",
    "        for col_to_bin in cols_to_bin:\n",
    "            tmp_val = tmp_data[col_to_bin].median()\n",
    "            if col_to_bin not in ['Gender', 'Unit1']:\n",
    "                tmp_val = (tmp_val-df_mean_std[col_to_bin]['mean'])/df_mean_std[col_to_bin]['std']\n",
    "                \n",
    "            X_binned_dict[col_to_bin] = tmp_val\n",
    "        \n",
    "        #package it all into a dictionary\n",
    "        tmp_dict = X_binned_dict\n",
    "        tmp_dict['X_cont'] = X_cont\n",
    "        tmp_dict['label'] = tmp_label\n",
    "        tmp_dict['patient'] = tmp_patient\n",
    "        windowed_data.append(tmp_dict)\n",
    "        \n",
    "    #append the dataframe to the list of dataframes\n",
    "    windowed_data_df = pd.DataFrame(windowed_data)\n",
    "    windowed_df_list.append(windowed_data_df)\n",
    "\n",
    "    #periodically save every 500 patients\n",
    "    if (int(patient[-5:]) % 500) == 0:\n",
    "        print('patient %i' % int(patient[-5:]))\n",
    "        windowed_df = pd.concat(windowed_df_list).reset_index(drop=True)\n",
    "        train = windowed_df[windowed_df['patient'].isin(patients_training_data)].drop('patient', axis=1)\n",
    "        test = windowed_df[~windowed_df['patient'].isin(patients_training_data)].drop('patient', axis=1)\n",
    "\n",
    "        train.to_pickle('feats/train_%i.pkl' % save_count)\n",
    "        test.to_pickle('feats/test_%i.pkl' % save_count)\n",
    "\n",
    "        windowed_df_list = []\n",
    "        save_count = save_count+1\n",
    "\n",
    "#save any remaining data\n",
    "if len(windowed_df_list) > 0:\n",
    "    #separate the training and testing data\n",
    "    windowed_df = pd.concat(windowed_df_list).reset_index(drop=True)\n",
    "    train = windowed_df[windowed_df['patient'].isin(patients_training_data)].drop('patient', axis=1)\n",
    "    test = windowed_df[~windowed_df['patient'].isin(patients_training_data)].drop('patient', axis=1)\n",
    "\n",
    "    train.to_pickle('feats/train_%i.pkl' % save_count)\n",
    "    test.to_pickle('feats/test_%i.pkl' % save_count)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}