model-selection/sklearn-group-cross-validation-split.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-Learn Group Cross-Validation Splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook explains how to generate K-folds for cross-validation with groups using `scikit-learn` for evaluation of machine learning models with out of sample data.\n",
    "\n",
    "This notebook will work with flights in and out of NYC in 2013.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial uses:\n",
    "* [pandas](https://pandas.pydata.org/docs/)\n",
    "* [statsmodels](https://www.statsmodels.org/stable/index.html)\n",
    "    * [statsmodels.api](https://www.statsmodels.org/stable/api.html#statsmodels-api)\n",
    "* [numpy](https://numpy.org/doc/stable/)\n",
    "* [scikit-learn](https://scikit-learn.org/stable/)\n",
    "    * [sklearn.model_selection](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import GroupKFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is from `rdatasets` imported using the Python package `statsmodels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 336776 entries, 0 to 336775\n",
      "Data columns (total 19 columns):\n",
      " #   Column          Non-Null Count   Dtype  \n",
      "---  ------          --------------   -----  \n",
      " 0   year            336776 non-null  int64  \n",
      " 1   month           336776 non-null  int64  \n",
      " 2   day             336776 non-null  int64  \n",
      " 3   dep_time        328521 non-null  float64\n",
      " 4   sched_dep_time  336776 non-null  int64  \n",
      " 5   dep_delay       328521 non-null  float64\n",
      " 6   arr_time        328063 non-null  float64\n",
      " 7   sched_arr_time  336776 non-null  int64  \n",
      " 8   arr_delay       327346 non-null  float64\n",
      " 9   carrier         336776 non-null  object \n",
      " 10  flight          336776 non-null  int64  \n",
      " 11  tailnum         334264 non-null  object \n",
      " 12  origin          336776 non-null  object \n",
      " 13  dest            336776 non-null  object \n",
      " 14  air_time        327346 non-null  float64\n",
      " 15  distance        336776 non-null  int64  \n",
      " 16  hour            336776 non-null  int64  \n",
      " 17  minute          336776 non-null  int64  \n",
      " 18  time_hour       336776 non-null  object \n",
      "dtypes: float64(5), int64(9), object(5)\n",
      "memory usage: 48.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df = sm.datasets.get_rdataset('flights', 'nycflights13').data\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handle null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "year                 0\n",
       "month                0\n",
       "day                  0\n",
       "dep_time          8255\n",
       "sched_dep_time       0\n",
       "dep_delay         8255\n",
       "arr_time          8713\n",
       "sched_arr_time       0\n",
       "arr_delay         9430\n",
       "carrier              0\n",
       "flight               0\n",
       "tailnum           2512\n",
       "origin               0\n",
       "dest                 0\n",
       "air_time          9430\n",
       "distance             0\n",
       "hour                 0\n",
       "minute               0\n",
       "time_hour            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As this model will predict arrival delay, the `Null` values are caused by flights did were cancelled or diverted. These can be excluded from this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(inplace=True)\n",
    "df.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the times from floats or ints to hour and minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['arr_hour'] = df.arr_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['arr_minute'] = df.arr_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df['sched_arr_hour'] = df.sched_arr_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['sched_arr_minute'] = df.sched_arr_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df['sched_dep_hour'] = df.sched_dep_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['sched_dep_minute'] = df.sched_dep_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df.rename(columns={'hour': 'dep_hour',\n",
    "                   'minute': 'dep_minute'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-validation splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn's `GroupKFold` will randomly sample the data into **N** folds (default of 5) that can be used to perform cross-validation during machine learning training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, group records by individual planes as once a plane is late, the subsequent flights are more likely to be delayed as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "group = df.tailnum.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the features and target before running cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'arr_delay'\n",
    "y = df[target]\n",
    "X = df.drop(columns=[target, 'flight', 'time_hour', 'year', 'dep_time', 'sched_dep_time', 'arr_time', 'sched_arr_time', 'dep_delay'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: [     0      1      2 ... 327340 327342 327345] Test: [     8     10     14 ... 327341 327343 327344]\n",
      "Train: [     0      2      4 ... 327341 327343 327344] Test: [     1      3      6 ... 327339 327342 327345]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [    26     57     73 ... 327314 327317 327325]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [    22     51     71 ... 327326 327332 327340]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [     9     33     35 ... 327321 327331 327338]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [     7     15     30 ... 327278 327313 327330]\n",
      "Train: [     1      2      3 ... 327343 327344 327345] Test: [     0     11     12 ... 327300 327312 327322]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [     4      5     17 ... 327276 327299 327307]\n",
      "Train: [     0      1      2 ... 327343 327344 327345] Test: [    13     16     34 ... 327316 327327 327333]\n",
      "Train: [     0      1      3 ... 327343 327344 327345] Test: [     2     24     29 ... 327328 327335 327337]\n"
     ]
    }
   ],
   "source": [
    "gkf = GroupKFold(n_splits=10)\n",
    "for train_index, test_index in gkf.split(X, groups=group):\n",
    "    print(\"Train:\", train_index, \"Test:\", test_index)\n",
    "    X_train = X.iloc[train_index, :]\n",
    "    y_train = y[train_index]\n",
    "    X_test = X.iloc[test_index, :]\n",
    "    y_test = y[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}