t81_558_class_13_04_retrain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 13: Advanced/Other Topics**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 13 Video Material\n",
    "\n",
    "* Part 13.1: Flask and Deep Learning Web Services [[Video]](https://www.youtube.com/watch?v=H73m9XvKHug&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_01_flask.ipynb)\n",
    "* Part 13.2: Deploying a Model to AWS  [[Video]](https://www.youtube.com/watch?v=8ygCyvRZ074&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_02_cloud.ipynb)\n",
    "* Part 13.3: Using a Keras Deep Neural Network with a Web Application  [[Video]](https://www.youtube.com/watch?v=OBbw0e-UroI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_03_web.ipynb)\n",
    "* **Part 13.4: When to Retrain Your Neural Network** [[Video]](https://www.youtube.com/watch?v=K2Tjdx_1v9g&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_04_retrain.ipynb)\n",
    "* Part 13.5: AI at the Edge: Using Keras on a Mobile Device  [[Video]]() [[Notebook]](t81_558_class_13_05_edge.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 13.4: When to Retrain Your Neural Network\n",
    "\n",
    "* Dataset Shift\n",
    "* Covariate Shift\n",
    "\n",
    "![Covariate Shift](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/cov-shift.png \"Covariate Shift\")\n",
    "[(graph source)](http://iwann.ugr.es/2011/pdf/InvitedTalk-FHerrera-IWANN11.pdf)\n",
    "\n",
    "### Measures of Drift\n",
    "\n",
    "* Drift Detection Method (DDM)  [10], \n",
    "* Early Drift Detection Method (EDDM)  [11], \n",
    "* Page-Hinkley Test (PHT) [12], \n",
    "* Adaptive Windowing (ADWIN)  [13], \n",
    "* Paired Learners [14], \n",
    "* EWMA for Concept Drift Detection (ECDD) [15], \n",
    "* Degree of Drift (DOF) [16], and \n",
    "* Statistical Test of Equal Proportions (STEPD) [17]\n",
    "\n",
    "Others.\n",
    "\n",
    "* KOLMOGOROV SMIRNOV TWO SAMPLE\n",
    "* https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence\n",
    "\n",
    "### Academic/Other Sources\n",
    "\n",
    "* [A unifying view on dataset shift in classification (paper)](https://rtg.cis.upenn.edu/cis700-2019/papers/dataset-shift/dataset-shift-terminology.pdf)\n",
    "* [A unifying view on dataset shift in classification (presentation)](http://reframe-d2k.org/img_auth.php/7/7e/Presentation_DatasetShift.pdf)\n",
    "* [A Comparative Study on Concept Drift Detectors](https://www.researchgate.net/publication/264081451_A_Comparative_Study_on_Concept_Drift_Detectors)\n",
    "* [Covariate Shift – Unearthing hidden problems in Real World Data Science](https://www.analyticsvidhya.com/blog/2017/07/covariate-shift-the-hidden-problem-of-real-world-data-science/)\n",
    "\n",
    "\n",
    "Kaggle data set:\n",
    "\n",
    "* [Sberbank Russian Housing Market](https://www.kaggle.com/c/sberbank-russian-housing-market/data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "PATH = \"/Users/jheaton/Downloads/sberbank-russian-housing-market\"\n",
    "\n",
    "\n",
    "train_df = pd.read_csv(os.path.join(PATH,\"train.csv\"))\n",
    "test_df = pd.read_csv(os.path.join(PATH,\"test.csv\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(df):\n",
    "    for i in df.columns:\n",
    "        if df[i].dtype == 'object':\n",
    "            df[i] = df[i].fillna(df[i].mode().iloc[0])\n",
    "        elif (df[i].dtype == 'int' or df[i].dtype == 'float'):\n",
    "            df[i] = df[i].fillna(np.nanmedian(df[i]))\n",
    "\n",
    "    enc = LabelEncoder()\n",
    "    for i in df.columns:\n",
    "        if (df[i].dtype == 'object'):\n",
    "            df[i] = enc.fit_transform(df[i].astype('str'))\n",
    "            df[i] = df[i].astype('object')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess(train_df)\n",
    "preprocess(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.drop('price_doc',axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KS-Statistic\n",
    "\n",
    "We will use the KS-Statistic to look at the difference in distribution between columns in the training and test sets.  Just as a baseline, consider if we compare the same field to itself.  I this case we are comparing the **kitch_sq** in training set. Because there is no difference in distribution between a field in itself, the p-value is 1.0 and the KS-Statistic statistic is 0. The P-Value is the probability that there is no difference between two distributions.  Typically some lower threshold is used for how low of a P-Value is needed to reject the null hypothesis and assume there is a difference.  The value of 0.05 is a common threshold for p-values.  Because the p-value is NOT below 0.05 in this case we can assume the two distributions are the same.  If the p-value were below the threshold then the **statistic** value becomes interesting.  This value tells you how different the two distributions are.  A value of 0.0 in this case means no differences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ks_2sampResult(statistic=-0.0, pvalue=1.0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "stats.ks_2samp(train_df['kitch_sq'], train_df['kitch_sq'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets do something more interesting.  We will compare the same field **kitch_sq** between the test and training sets.  In this case, the p-value is below 0.05, so the **statistic** value now contains the amount of difference detected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ks_2sampResult(statistic=0.25829078867676714, pvalue=0.0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.ks_2samp(train_df['kitch_sq'], test_df['kitch_sq'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we pull the KS-Stat for every field.  We also establish a boundary of what is the maximum p-value to display and how much of a difference is needed before we display the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id: Ks_2sampResult(statistic=1.0, pvalue=0.0)\n",
      "timestamp: Ks_2sampResult(statistic=0.8982081426022823, pvalue=0.0)\n",
      "life_sq: Ks_2sampResult(statistic=0.2255084471628891, pvalue=7.29401465948424e-271)\n",
      "max_floor: Ks_2sampResult(statistic=0.17313454154786817, pvalue=7.82000315371674e-160)\n",
      "build_year: Ks_2sampResult(statistic=0.3176883950430345, pvalue=0.0)\n",
      "num_room: Ks_2sampResult(statistic=0.1226755470309048, pvalue=1.8622542043144584e-80)\n",
      "kitch_sq: Ks_2sampResult(statistic=0.25829078867676714, pvalue=0.0)\n",
      "state: Ks_2sampResult(statistic=0.13641341252952505, pvalue=2.1968159319271184e-99)\n",
      "preschool_quota: Ks_2sampResult(statistic=0.2364160801236304, pvalue=1.1710777340471466e-297)\n",
      "school_quota: Ks_2sampResult(statistic=0.25657342859882415, pvalue=0.0)\n",
      "raion_build_count_with_material_info: Ks_2sampResult(statistic=0.19083554469945835, pvalue=4.2830715478540455e-194)\n",
      "build_count_block: Ks_2sampResult(statistic=0.2085099875571384, pvalue=1.2494725963839073e-231)\n",
      "build_count_wood: Ks_2sampResult(statistic=0.20880892257287548, pvalue=2.7121209564049356e-232)\n",
      "build_count_brick: Ks_2sampResult(statistic=0.20707302040295733, pvalue=1.8727526843617417e-228)\n",
      "build_count_monolith: Ks_2sampResult(statistic=0.1779053203005685, pvalue=9.707337049927237e-169)\n",
      "raion_build_count_with_builddate_info: Ks_2sampResult(statistic=0.19083554469945835, pvalue=4.2830715478540455e-194)\n",
      "build_count_1946-1970: Ks_2sampResult(statistic=0.2120848849003817, pvalue=1.2611967894038327e-239)\n",
      "build_count_1971-1995: Ks_2sampResult(statistic=0.19318717795964874, pvalue=6.749018351478723e-199)\n",
      "build_count_after_1995: Ks_2sampResult(statistic=0.1899788049625577, pvalue=2.326885447792135e-192)\n",
      "cafe_sum_500_min_price_avg: Ks_2sampResult(statistic=0.43416513954613944, pvalue=0.0)\n",
      "cafe_avg_price_500: Ks_2sampResult(statistic=0.4337483600913839, pvalue=0.0)\n",
      "cafe_sum_1000_min_price_avg: Ks_2sampResult(statistic=0.20459725920896277, pvalue=4.926006714051654e-223)\n",
      "cafe_sum_1000_max_price_avg: Ks_2sampResult(statistic=0.2025042643599122, pvalue=1.673760717424549e-218)\n",
      "cafe_avg_price_1000: Ks_2sampResult(statistic=0.20399591976318965, pvalue=9.979898113798837e-222)\n",
      "cafe_sum_1500_min_price_avg: Ks_2sampResult(statistic=0.13264580849650315, pvalue=5.402810864486807e-94)\n",
      "cafe_avg_price_1500: Ks_2sampResult(statistic=0.13316992134991978, pvalue=9.81160594279586e-95)\n",
      "cafe_sum_2000_min_price_avg: Ks_2sampResult(statistic=0.10808898569668424, pvalue=1.4808059430311726e-62)\n",
      "cafe_sum_2000_max_price_avg: Ks_2sampResult(statistic=0.10732529051140638, pvalue=1.1100804327460878e-61)\n",
      "cafe_avg_price_2000: Ks_2sampResult(statistic=0.1081218037860151, pvalue=1.3575759911857293e-62)\n"
     ]
    }
   ],
   "source": [
    "for col in train_df.columns:\n",
    "    ks = stats.ks_2samp(train_df[col], test_df[col])\n",
    "    if ks.pvalue < 0.05 and ks.statistic>0.1:\n",
    "        print(f'{col}: {ks}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting Drift between Training and Testing Datasets by Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample the training and test into smaller sets to train.  We would like 10K elements from each; however, the test set only has 7,662, so as a result we only sample that amount from each side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7662\n"
     ]
    }
   ],
   "source": [
    "SAMPLE_SIZE = min(len(train_df),len(test_df))\n",
    "SAMPLE_SIZE = min(SAMPLE_SIZE,10000)\n",
    "print(SAMPLE_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take the random samples from the training and test sets and also add a flag called **source_training** so we can tell the two apart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_sample = train_df.sample(SAMPLE_SIZE, random_state=49)\n",
    "testing_sample = test_df.sample(SAMPLE_SIZE, random_state=48)\n",
    "\n",
    "# Is the data from the training set?\n",
    "training_sample['source_training'] = 1\n",
    "testing_sample['source_training'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we combine the data that we sampled from the training and test data sets and shuffle the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build combined training set\n",
    "combined = testing_sample.append(training_sample)\n",
    "combined.reset_index(inplace=True, drop=True)\n",
    "\n",
    "# Now randomize\n",
    "combined = combined.reindex(np.random.permutation(combined.index))\n",
    "combined.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now generate $x$ and $y$ to train.  We are attempting to predict the **source_training** value as $y$, that indicates if the data came from train or test.  If the model is very successful at using the data to predict if it came from train or test then there is likely drift.  Ideally the train and test data should be indistinguishable.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get ready to train\n",
    "y = combined['source_training'].values\n",
    "combined.drop('source_training',axis=1,inplace=True)\n",
    "x = combined.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, ..., 1, 0, 0])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will consider anything above a 0.75 AUC as having a good chance of drift."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id 1.0\n",
      "timestamp 0.9601862111975688\n",
      "full_sq 0.7966785611424911\n",
      "life_sq 0.8724218330166038\n",
      "build_year 0.8004825176688191\n",
      "kitch_sq 0.9070093804672634\n",
      "cafe_sum_500_min_price_avg 0.8435920036035689\n",
      "cafe_avg_price_500 0.8453533835344671\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "model = RandomForestClassifier(n_estimators = 60, max_depth = 7,min_samples_leaf = 5)\n",
    "lst = []\n",
    "\n",
    "for i in combined.columns:\n",
    "    score = cross_val_score(model,pd.DataFrame(combined[i]),y,cv=2,scoring='roc_auc')\n",
    "    if (np.mean(score) > 0.75):\n",
    "        lst.append(i)\n",
    "        print(i,np.mean(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}