cisco_telemetry_rnn_modeling_20190401.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Maintenance using Deep Learning\n",
    "40 relevant attributes out of 85 were identified to be used as input to a recurrent neural network. We trained a baseline model that would “detect” anomalous events on each node as soon as they occur with a 10-fold cross validated AUC of 0.97. \n",
    "\n",
    "| #  | Traffic load | No. Anomalies | Duration | Description                        |\n",
    "|----|--------------|---------------|----------|------------------------------------|\n",
    "| 0  | 0            | 0             | 1h       | Baseline (no amolies)              |\n",
    "| 1  | 500Gbps      | 0             | 1h       | Baseline (no anomalies)            |\n",
    "| 2  | 1Tbps        | 11            | 1h       | BGP Clear                          |\n",
    "| 3  | 1Tbps        | 8             | 0.55h    | BGP Clear                          |\n",
    "| 4  | 1Tbps        | 5             | 0.72h    | Port Flap                          |\n",
    "| 5  | 1Tbps        | 12            | 2h       | BGP Clear                          |\n",
    "| 6  | 0            | 12            | 2h       | BGP Clear                          |\n",
    "| 7  | 0            | 130           | 72h      | (VIRL) BGP Clear                   |\n",
    "| 8  | 0            | 238           | 262h     | (VIRL) BGP Clear                   |\n",
    "| 9  | 2.9Tbps      | 5             | .75h     | Port Admin Shut                    |\n",
    "| 10 | 2Tbps        | 5             | .55h     | Port Transceiver Pull and Reinsert |\n",
    "\n",
    "Data source: [Cisco real-world telemetry anomaly dataset](https://github.com/cisco-ie/telemetry) which includes timeseries of 85 signals from 12 nodes under 0 to 1Tbps traffic of Blue Ray/Youtube4K movie streaming, skype video calls and other activities. BGP Clear events from dataset 2, 3, 5, 6 are used for modeling for this work.\n",
    "\n",
    "The preprocessed data is available in [here in s3](https://s3.amazonaws.com/cisco-anomaly-detection/preprocessed_20190401.pkl) by this [notebook](./cisco_telemetry_data_preprocessing_20190401.ipynb). Implementation and training was performed on an AWS EC2 p2.xlarge instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import sys, os, warnings, pickle\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "%matplotlib inline\n",
    "\n",
    "import sklearn.model_selection as skms\n",
    "import sklearn.metrics as skm\n",
    "import sklearn.preprocessing as skp\n",
    "\n",
    "import tensorflow as tf\n",
    "# Here we would like to restrict the GPU memory usage by tensorflow which by default \n",
    "# would allocate all memory available to this routine. We ask it to allocate what is \n",
    "# necessary during runtime and allow it to grow. Note that this configuration has to\n",
    "# be set prior to importing keras to have the intended effect. \n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "session = tf.Session(config=config)\n",
    "\n",
    "import keras.backend as K\n",
    "from keras.layers import Dense, Dropout, LSTM, TimeDistributed\n",
    "from keras.optimizers import Adam\n",
    "from keras.callbacks import TensorBoard, EarlyStopping, Callback\n",
    "from keras.models import Sequential\n",
    "\n",
    "exp='nn_keras_cv_20190401'\n",
    "logdir='/home/ubuntu/Projects/cisco/logs/%s' % exp\n",
    "\n",
    "try:\n",
    "    os.makedirs(logdir)\n",
    "except:\n",
    "    pass\n",
    "\n",
    "np.random.seed(1)\n",
    "tf.set_random_seed(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting the preprocessed data from pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "d_df_resample=pickle.load(open('../Data/cisco_telemetry/preprocessed_20190401.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['active-routes-count', 'as', 'backup-routes-count', 'bandwidth',\n",
       "       'bytes-received', 'bytes-sent', 'carrier-transitions', 'crc-errors',\n",
       "       'deleted-routes-count', 'free-application-memory',\n",
       "       'free-physical-memory', 'global__established-neighbors-count-total',\n",
       "       'global__neighbors-count-total', 'global__nexthop-count',\n",
       "       'input-data-rate', 'input-drops', 'input-errors', 'input-load',\n",
       "       'input-packet-rate', 'load-interval', 'no-route-packets',\n",
       "       'output-data-rate', 'output-drops', 'output-load', 'output-packet-rate',\n",
       "       'packets-received', 'packets-sent', 'paths-count',\n",
       "       'performance-statistics__global__configuration-items-processed',\n",
       "       'performance-statistics__vrf__inbound-update-messages',\n",
       "       'protocol-route-memory', 'reliability', 'routes-counts',\n",
       "       'total-cpu-fifteen-minute', 'total-cpu-five-minute',\n",
       "       'total-cpu-one-minute', 'vrf__neighbors-count', 'vrf__network-count',\n",
       "       'vrf__path-count', 'vrf__update-messages-received', 'time',\n",
       "       'Experiment', 'Anomaly', 'Anomaly_orig', 'Anomaly_shift_1',\n",
       "       'Anomaly_shift_2', 'Anomaly_shift_3', 'Anomaly_shift_4',\n",
       "       'Anomaly_shift_5', 'Anomaly_shift_6', 'Partition'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_df_resample['2_leaf1_0'].columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the feature columns and target columns. There are shifted anomaly labels in the dataset for other purposes in the future. For now we are going to model labels based on actual anomaly, `Anomaly`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols=d_df_resample['2_leaf1_0'].columns[:-11]\n",
    "target_cols=[i for i in d_df_resample['2_leaf1_0'].columns if 'Anomaly' in i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['active-routes-count', 'as', 'backup-routes-count', 'bandwidth',\n",
       "       'bytes-received', 'bytes-sent', 'carrier-transitions', 'crc-errors',\n",
       "       'deleted-routes-count', 'free-application-memory',\n",
       "       'free-physical-memory', 'global__established-neighbors-count-total',\n",
       "       'global__neighbors-count-total', 'global__nexthop-count',\n",
       "       'input-data-rate', 'input-drops', 'input-errors', 'input-load',\n",
       "       'input-packet-rate', 'load-interval', 'no-route-packets',\n",
       "       'output-data-rate', 'output-drops', 'output-load', 'output-packet-rate',\n",
       "       'packets-received', 'packets-sent', 'paths-count',\n",
       "       'performance-statistics__global__configuration-items-processed',\n",
       "       'performance-statistics__vrf__inbound-update-messages',\n",
       "       'protocol-route-memory', 'reliability', 'routes-counts',\n",
       "       'total-cpu-fifteen-minute', 'total-cpu-five-minute',\n",
       "       'total-cpu-one-minute', 'vrf__neighbors-count', 'vrf__network-count',\n",
       "       'vrf__path-count', 'vrf__update-messages-received'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_timesteps = d_df_resample['2_leaf1_0'].shape[0]\n",
    "n_features = len(feature_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Profiling the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(85, 40)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_timesteps, n_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    67\n",
       "True     18\n",
       "Name: Anomaly, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_df_resample['2_leaf1_0']['Anomaly'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time\n",
       "2017-06-29 16:34:30    False\n",
       "2017-06-29 16:34:40    False\n",
       "2017-06-29 16:34:50    False\n",
       "2017-06-29 16:35:00    False\n",
       "2017-06-29 16:35:10    False\n",
       "Freq: 10S, Name: Anomaly, dtype: bool"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_df_resample['2_leaf1_0']['Anomaly'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter samples by having anomaly or not into two dictionary. We will be using positive samples (with anomalies) for training and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "d_df_resample_positive={}\n",
    "d_df_resample_negative={}\n",
    "for key, df in d_df_resample.items():\n",
    "    #print(key)\n",
    "    if df['Anomaly'].any():\n",
    "        d_df_resample_positive[key]=df\n",
    "    else:\n",
    "        d_df_resample_negative[key]=df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating input dataset, `X` and `Y`, and `group` for storing dataset information which will later be used to stratify the data during the cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=np.zeros((len(d_df_resample_positive.keys()), n_timesteps, n_features))\n",
    "Y=np.zeros((len(d_df_resample_positive.keys()), n_timesteps))\n",
    "Y_orig=np.zeros_like(Y)\n",
    "target='Anomaly'\n",
    "group=[]\n",
    "\n",
    "for i, node in enumerate(d_df_resample_positive.keys()):\n",
    "    X[i] = d_df_resample_positive[node][feature_cols].values\n",
    "    Y[i] = d_df_resample_positive[node][target].values\n",
    "    Y_orig[i] = d_df_resample_positive[node]['Anomaly_orig'].values\n",
    "    group.append(node.split('_')[0])\n",
    "group=np.array(group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((48, 85, 40), (48, 85), (48, 85), '0_0', '6_leaf3_8')"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, Y.shape, Y_orig.shape, i, node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "        1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n",
       " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "        1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]))"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_orig[0], Y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '3',\n",
       "       '3', '3', '3', '3', '3', '3', '3', '5', '5', '5', '5', '5', '5',\n",
       "       '5', '5', '5', '5', '5', '5', '5', '5', '5', '6', '6', '6', '6',\n",
       "       '6', '6', '6', '6', '6', '6', '6', '6', '6'], dtype='<U1')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize_prediction_clustermap(df, columns=None, **kw):\n",
    "    row_colors=df[[target, 'Pred']].replace(to_replace={True: 'r', False: 'b'})\n",
    "    g=sns.clustermap(df[columns].T, col_colors=row_colors, figsize=(16,10), vmin=-8, vmax=8, **kw)\n",
    "    return g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Evaluation using 10-fold cross-validation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 fold\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_8 (LSTM)                (None, 85, 100)           56400     \n",
      "_________________________________________________________________\n",
      "dropout_15 (Dropout)         (None, 85, 100)           0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 85, 80)            8080      \n",
      "_________________________________________________________________\n",
      "dropout_16 (Dropout)         (None, 85, 80)            0         \n",
      "_________________________________________________________________\n",
      "time_distributed_8 (TimeDist (None, 85, 1)             81        \n",
      "=================================================================\n",
      "Total params: 64,561\n",
      "Trainable params: 64,561\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 37 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 6s - loss: 0.6155 - mean_squared_error: 0.2134 - acc: 0.6343 - val_loss: 0.4727 - val_mean_squared_error: 0.1456 - val_acc: 0.9059\n",
      "Epoch 2/150\n",
      " - 5s - loss: 0.3580 - mean_squared_error: 0.1047 - acc: 0.8712 - val_loss: 0.4033 - val_mean_squared_error: 0.1203 - val_acc: 0.8147\n",
      "Epoch 3/150\n",
      " - 5s - loss: 0.2244 - mean_squared_error: 0.0599 - acc: 0.9281 - val_loss: 0.3306 - val_mean_squared_error: 0.1054 - val_acc: 0.8029\n",
      "Epoch 4/150\n",
      " - 5s - loss: 0.1607 - mean_squared_error: 0.0431 - acc: 0.9463 - val_loss: 0.1736 - val_mean_squared_error: 0.0482 - val_acc: 0.9471\n",
      "Epoch 5/150\n",
      " - 5s - loss: 0.1356 - mean_squared_error: 0.0363 - acc: 0.9529 - val_loss: 0.2247 - val_mean_squared_error: 0.0755 - val_acc: 0.9059\n",
      "Epoch 6/150\n",
      " - 5s - loss: 0.1224 - mean_squared_error: 0.0336 - acc: 0.9558 - val_loss: 0.2688 - val_mean_squared_error: 0.0943 - val_acc: 0.8000\n",
      "Epoch 7/150\n",
      " - 5s - loss: 0.1151 - mean_squared_error: 0.0321 - acc: 0.9603 - val_loss: 0.2033 - val_mean_squared_error: 0.0618 - val_acc: 0.9265\n",
      "Epoch 8/150\n",
      " - 5s - loss: 0.1142 - mean_squared_error: 0.0324 - acc: 0.9577 - val_loss: 0.2125 - val_mean_squared_error: 0.0710 - val_acc: 0.9176\n",
      "Epoch 9/150\n",
      " - 5s - loss: 0.1036 - mean_squared_error: 0.0282 - acc: 0.9641 - val_loss: 0.3607 - val_mean_squared_error: 0.1241 - val_acc: 0.7941\n",
      "Epoch 10/150\n",
      " - 5s - loss: 0.0900 - mean_squared_error: 0.0254 - acc: 0.9657 - val_loss: 0.2395 - val_mean_squared_error: 0.0761 - val_acc: 0.9088\n",
      "Epoch 11/150\n",
      " - 5s - loss: 0.0898 - mean_squared_error: 0.0260 - acc: 0.9647 - val_loss: 0.4374 - val_mean_squared_error: 0.1424 - val_acc: 0.7971\n",
      "Epoch 12/150\n",
      " - 5s - loss: 0.0842 - mean_squared_error: 0.0240 - acc: 0.9682 - val_loss: 0.2305 - val_mean_squared_error: 0.0705 - val_acc: 0.9382\n",
      "Epoch 13/150\n",
      " - 5s - loss: 0.0824 - mean_squared_error: 0.0231 - acc: 0.9701 - val_loss: 0.2602 - val_mean_squared_error: 0.0725 - val_acc: 0.9353\n",
      "Epoch 14/150\n",
      " - 5s - loss: 0.0862 - mean_squared_error: 0.0257 - acc: 0.9647 - val_loss: 0.9673 - val_mean_squared_error: 0.2079 - val_acc: 0.7735\n",
      "Epoch 00014: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9760, 0.9148, 0.0707, 0.8933, 0.7092, 0.7907, 0.9064\n",
      "\ttest: 0.9379, 0.8504, 0.0829, 0.8558, 0.5460, 0.6667, 0.8525\n",
      "2 fold\n",
      "Train on 37 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 6s - loss: 0.5038 - mean_squared_error: 0.1648 - acc: 0.7876 - val_loss: 0.4704 - val_mean_squared_error: 0.1430 - val_acc: 0.8971\n",
      "Epoch 2/150\n",
      " - 5s - loss: 0.3283 - mean_squared_error: 0.0970 - acc: 0.8734 - val_loss: 0.3107 - val_mean_squared_error: 0.0814 - val_acc: 0.9118\n",
      "Epoch 3/150\n",
      " - 5s - loss: 0.2075 - mean_squared_error: 0.0572 - acc: 0.9281 - val_loss: 0.1983 - val_mean_squared_error: 0.0485 - val_acc: 0.9471\n",
      "Epoch 4/150\n",
      " - 5s - loss: 0.1577 - mean_squared_error: 0.0423 - acc: 0.9459 - val_loss: 0.1358 - val_mean_squared_error: 0.0319 - val_acc: 0.9706\n",
      "Epoch 5/150\n",
      " - 5s - loss: 0.1376 - mean_squared_error: 0.0377 - acc: 0.9536 - val_loss: 0.0744 - val_mean_squared_error: 0.0176 - val_acc: 0.9794\n",
      "Epoch 6/150\n",
      " - 5s - loss: 0.1171 - mean_squared_error: 0.0335 - acc: 0.9577 - val_loss: 0.0895 - val_mean_squared_error: 0.0230 - val_acc: 0.9735\n",
      "Epoch 7/150\n",
      " - 5s - loss: 0.1018 - mean_squared_error: 0.0294 - acc: 0.9603 - val_loss: 0.0770 - val_mean_squared_error: 0.0199 - val_acc: 0.9765\n",
      "Epoch 8/150\n",
      " - 5s - loss: 0.0938 - mean_squared_error: 0.0266 - acc: 0.9650 - val_loss: 0.0838 - val_mean_squared_error: 0.0230 - val_acc: 0.9706\n",
      "Epoch 9/150\n",
      " - 5s - loss: 0.0849 - mean_squared_error: 0.0244 - acc: 0.9707 - val_loss: 0.1096 - val_mean_squared_error: 0.0293 - val_acc: 0.9647\n",
      "Epoch 10/150\n",
      " - 5s - loss: 0.0818 - mean_squared_error: 0.0233 - acc: 0.9669 - val_loss: 0.0718 - val_mean_squared_error: 0.0205 - val_acc: 0.9735\n",
      "Epoch 11/150\n",
      " - 5s - loss: 0.0799 - mean_squared_error: 0.0235 - acc: 0.9685 - val_loss: 0.1372 - val_mean_squared_error: 0.0418 - val_acc: 0.9529\n",
      "Epoch 12/150\n",
      " - 5s - loss: 0.0810 - mean_squared_error: 0.0247 - acc: 0.9657 - val_loss: 0.0678 - val_mean_squared_error: 0.0170 - val_acc: 0.9794\n",
      "Epoch 13/150\n",
      " - 5s - loss: 0.0764 - mean_squared_error: 0.0221 - acc: 0.9714 - val_loss: 0.2535 - val_mean_squared_error: 0.0923 - val_acc: 0.8412\n",
      "Epoch 14/150\n",
      " - 5s - loss: 0.0821 - mean_squared_error: 0.0243 - acc: 0.9663 - val_loss: 0.1586 - val_mean_squared_error: 0.0521 - val_acc: 0.9382\n",
      "Epoch 15/150\n",
      " - 5s - loss: 0.0805 - mean_squared_error: 0.0227 - acc: 0.9688 - val_loss: 0.1652 - val_mean_squared_error: 0.0535 - val_acc: 0.9235\n",
      "Epoch 16/150\n",
      " - 5s - loss: 0.0634 - mean_squared_error: 0.0189 - acc: 0.9707 - val_loss: 0.1216 - val_mean_squared_error: 0.0368 - val_acc: 0.9618\n",
      "Epoch 17/150\n",
      " - 5s - loss: 0.0536 - mean_squared_error: 0.0155 - acc: 0.9771 - val_loss: 0.0966 - val_mean_squared_error: 0.0273 - val_acc: 0.9706\n",
      "Epoch 18/150\n",
      " - 5s - loss: 0.0483 - mean_squared_error: 0.0143 - acc: 0.9803 - val_loss: 0.1111 - val_mean_squared_error: 0.0293 - val_acc: 0.9706\n",
      "Epoch 19/150\n",
      " - 5s - loss: 0.0579 - mean_squared_error: 0.0166 - acc: 0.9762 - val_loss: 0.1452 - val_mean_squared_error: 0.0439 - val_acc: 0.9647\n",
      "Epoch 20/150\n",
      " - 5s - loss: 0.0505 - mean_squared_error: 0.0149 - acc: 0.9793 - val_loss: 0.1012 - val_mean_squared_error: 0.0303 - val_acc: 0.9588\n",
      "Epoch 21/150\n",
      " - 5s - loss: 0.0543 - mean_squared_error: 0.0153 - acc: 0.9793 - val_loss: 0.1166 - val_mean_squared_error: 0.0346 - val_acc: 0.9559\n",
      "Epoch 22/150\n",
      " - 5s - loss: 0.0530 - mean_squared_error: 0.0153 - acc: 0.9797 - val_loss: 0.1176 - val_mean_squared_error: 0.0346 - val_acc: 0.9647\n",
      "Epoch 00022: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9978, 0.9831, 0.0140, 0.9460, 0.9597, 0.9528, 0.9686\n",
      "\ttest: 0.9956, 0.9681, 0.0220, 0.8529, 0.9560, 0.9016, 0.9224\n",
      "3 fold\n",
      "Train on 38 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 7s - loss: 0.5564 - mean_squared_error: 0.1863 - acc: 0.7554 - val_loss: 0.4515 - val_mean_squared_error: 0.1367 - val_acc: 0.9118\n",
      "Epoch 2/150\n",
      " - 5s - loss: 0.3314 - mean_squared_error: 0.0975 - acc: 0.8759 - val_loss: 0.3183 - val_mean_squared_error: 0.0844 - val_acc: 0.9706\n",
      "Epoch 3/150\n",
      " - 5s - loss: 0.1993 - mean_squared_error: 0.0543 - acc: 0.9344 - val_loss: 0.2406 - val_mean_squared_error: 0.0726 - val_acc: 0.9118\n",
      "Epoch 4/150\n",
      " - 5s - loss: 0.1489 - mean_squared_error: 0.0402 - acc: 0.9502 - val_loss: 0.3403 - val_mean_squared_error: 0.1153 - val_acc: 0.8029\n",
      "Epoch 5/150\n",
      " - 5s - loss: 0.1261 - mean_squared_error: 0.0346 - acc: 0.9579 - val_loss: 0.2827 - val_mean_squared_error: 0.0949 - val_acc: 0.8294\n",
      "Epoch 6/150\n",
      " - 5s - loss: 0.1214 - mean_squared_error: 0.0349 - acc: 0.9548 - val_loss: 0.3788 - val_mean_squared_error: 0.1257 - val_acc: 0.8147\n",
      "Epoch 7/150\n",
      " - 5s - loss: 0.1173 - mean_squared_error: 0.0339 - acc: 0.9548 - val_loss: 0.5356 - val_mean_squared_error: 0.1666 - val_acc: 0.7971\n",
      "Epoch 8/150\n",
      " - 5s - loss: 0.1086 - mean_squared_error: 0.0315 - acc: 0.9601 - val_loss: 0.3013 - val_mean_squared_error: 0.1048 - val_acc: 0.8235\n",
      "Epoch 9/150\n",
      " - 5s - loss: 0.0920 - mean_squared_error: 0.0260 - acc: 0.9659 - val_loss: 0.4671 - val_mean_squared_error: 0.1555 - val_acc: 0.7971\n",
      "Epoch 10/150\n",
      " - 5s - loss: 0.0838 - mean_squared_error: 0.0245 - acc: 0.9678 - val_loss: 0.4215 - val_mean_squared_error: 0.1447 - val_acc: 0.8000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11/150\n",
      " - 5s - loss: 0.0896 - mean_squared_error: 0.0264 - acc: 0.9659 - val_loss: 0.4049 - val_mean_squared_error: 0.1376 - val_acc: 0.8088\n",
      "Epoch 12/150\n",
      " - 5s - loss: 0.0857 - mean_squared_error: 0.0256 - acc: 0.9647 - val_loss: 0.5247 - val_mean_squared_error: 0.1602 - val_acc: 0.8118\n",
      "Epoch 13/150\n",
      " - 5s - loss: 0.0678 - mean_squared_error: 0.0199 - acc: 0.9746 - val_loss: 0.6467 - val_mean_squared_error: 0.1721 - val_acc: 0.8088\n",
      "Epoch 00013: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9910, 0.9608, 0.0321, 0.9045, 0.8858, 0.8951, 0.9390\n",
      "\ttest: 0.9843, 0.9529, 0.0405, 0.8611, 0.8158, 0.8378, 0.9146\n",
      "4 fold\n",
      "Train on 39 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 7s - loss: 0.5779 - mean_squared_error: 0.1969 - acc: 0.6887 - val_loss: 0.4050 - val_mean_squared_error: 0.1165 - val_acc: 0.8824\n",
      "Epoch 2/150\n",
      " - 5s - loss: 0.3224 - mean_squared_error: 0.0947 - acc: 0.8706 - val_loss: 0.2671 - val_mean_squared_error: 0.0678 - val_acc: 0.9618\n",
      "Epoch 3/150\n",
      " - 6s - loss: 0.1968 - mean_squared_error: 0.0518 - acc: 0.9388 - val_loss: 0.2017 - val_mean_squared_error: 0.0538 - val_acc: 0.9353\n",
      "Epoch 4/150\n",
      " - 5s - loss: 0.1594 - mean_squared_error: 0.0429 - acc: 0.9502 - val_loss: 0.1205 - val_mean_squared_error: 0.0301 - val_acc: 0.9706\n",
      "Epoch 5/150\n",
      " - 5s - loss: 0.1244 - mean_squared_error: 0.0339 - acc: 0.9584 - val_loss: 0.1909 - val_mean_squared_error: 0.0609 - val_acc: 0.9147\n",
      "Epoch 6/150\n",
      " - 5s - loss: 0.1136 - mean_squared_error: 0.0314 - acc: 0.9584 - val_loss: 0.2771 - val_mean_squared_error: 0.0970 - val_acc: 0.8088\n",
      "Epoch 7/150\n",
      " - 5s - loss: 0.1089 - mean_squared_error: 0.0303 - acc: 0.9608 - val_loss: 0.0610 - val_mean_squared_error: 0.0167 - val_acc: 0.9765\n",
      "Epoch 8/150\n",
      " - 5s - loss: 0.1065 - mean_squared_error: 0.0286 - acc: 0.9626 - val_loss: 0.3288 - val_mean_squared_error: 0.1133 - val_acc: 0.8118\n",
      "Epoch 9/150\n",
      " - 5s - loss: 0.0881 - mean_squared_error: 0.0245 - acc: 0.9662 - val_loss: 0.1318 - val_mean_squared_error: 0.0393 - val_acc: 0.9706\n",
      "Epoch 10/150\n",
      " - 5s - loss: 0.0751 - mean_squared_error: 0.0211 - acc: 0.9732 - val_loss: 0.0945 - val_mean_squared_error: 0.0212 - val_acc: 0.9853\n",
      "Epoch 11/150\n",
      " - 5s - loss: 0.0763 - mean_squared_error: 0.0219 - acc: 0.9701 - val_loss: 0.1462 - val_mean_squared_error: 0.0458 - val_acc: 0.9471\n",
      "Epoch 12/150\n",
      " - 5s - loss: 0.0733 - mean_squared_error: 0.0210 - acc: 0.9713 - val_loss: 0.1701 - val_mean_squared_error: 0.0604 - val_acc: 0.8853\n",
      "Epoch 13/150\n",
      " - 5s - loss: 0.0681 - mean_squared_error: 0.0191 - acc: 0.9762 - val_loss: 0.2630 - val_mean_squared_error: 0.0932 - val_acc: 0.8676\n",
      "Epoch 14/150\n",
      " - 5s - loss: 0.0598 - mean_squared_error: 0.0169 - acc: 0.9792 - val_loss: 0.1991 - val_mean_squared_error: 0.0726 - val_acc: 0.8735\n",
      "Epoch 15/150\n",
      " - 5s - loss: 0.0688 - mean_squared_error: 0.0188 - acc: 0.9768 - val_loss: 0.2500 - val_mean_squared_error: 0.0855 - val_acc: 0.8765\n",
      "Epoch 16/150\n",
      " - 5s - loss: 0.0701 - mean_squared_error: 0.0204 - acc: 0.9744 - val_loss: 0.0320 - val_mean_squared_error: 0.0095 - val_acc: 0.9824\n",
      "Epoch 17/150\n",
      " - 5s - loss: 0.0631 - mean_squared_error: 0.0180 - acc: 0.9780 - val_loss: 0.0299 - val_mean_squared_error: 0.0084 - val_acc: 0.9912\n",
      "Epoch 18/150\n",
      " - 5s - loss: 0.0502 - mean_squared_error: 0.0139 - acc: 0.9849 - val_loss: 0.0323 - val_mean_squared_error: 0.0092 - val_acc: 0.9882\n",
      "Epoch 19/150\n",
      " - 5s - loss: 0.0426 - mean_squared_error: 0.0122 - acc: 0.9849 - val_loss: 0.0337 - val_mean_squared_error: 0.0085 - val_acc: 0.9882\n",
      "Epoch 20/150\n",
      " - 5s - loss: 0.0613 - mean_squared_error: 0.0159 - acc: 0.9816 - val_loss: 0.0345 - val_mean_squared_error: 0.0075 - val_acc: 0.9912\n",
      "Epoch 21/150\n",
      " - 5s - loss: 0.0639 - mean_squared_error: 0.0181 - acc: 0.9777 - val_loss: 0.0460 - val_mean_squared_error: 0.0121 - val_acc: 0.9853\n",
      "Epoch 22/150\n",
      " - 5s - loss: 0.0512 - mean_squared_error: 0.0141 - acc: 0.9837 - val_loss: 0.0379 - val_mean_squared_error: 0.0088 - val_acc: 0.9912\n",
      "Epoch 23/150\n",
      " - 5s - loss: 0.0393 - mean_squared_error: 0.0110 - acc: 0.9864 - val_loss: 0.0293 - val_mean_squared_error: 0.0072 - val_acc: 0.9912\n",
      "Epoch 24/150\n",
      " - 5s - loss: 0.0381 - mean_squared_error: 0.0109 - acc: 0.9873 - val_loss: 0.0448 - val_mean_squared_error: 0.0108 - val_acc: 0.9882\n",
      "Epoch 25/150\n",
      " - 5s - loss: 0.0353 - mean_squared_error: 0.0102 - acc: 0.9882 - val_loss: 0.0535 - val_mean_squared_error: 0.0137 - val_acc: 0.9882\n",
      "Epoch 26/150\n",
      " - 5s - loss: 0.0415 - mean_squared_error: 0.0118 - acc: 0.9858 - val_loss: 0.0453 - val_mean_squared_error: 0.0106 - val_acc: 0.9912\n",
      "Epoch 27/150\n",
      " - 5s - loss: 0.0380 - mean_squared_error: 0.0108 - acc: 0.9864 - val_loss: 0.0694 - val_mean_squared_error: 0.0181 - val_acc: 0.9765\n",
      "Epoch 28/150\n",
      " - 5s - loss: 0.0503 - mean_squared_error: 0.0151 - acc: 0.9801 - val_loss: 0.1134 - val_mean_squared_error: 0.0372 - val_acc: 0.9324\n",
      "Epoch 29/150\n",
      " - 5s - loss: 0.0408 - mean_squared_error: 0.0118 - acc: 0.9852 - val_loss: 0.1326 - val_mean_squared_error: 0.0360 - val_acc: 0.9706\n",
      "Epoch 30/150\n",
      " - 5s - loss: 0.0344 - mean_squared_error: 0.0099 - acc: 0.9888 - val_loss: 0.5122 - val_mean_squared_error: 0.1591 - val_acc: 0.8088\n",
      "Epoch 31/150\n",
      " - 5s - loss: 0.0281 - mean_squared_error: 0.0076 - acc: 0.9910 - val_loss: 0.1601 - val_mean_squared_error: 0.0591 - val_acc: 0.8824\n",
      "Epoch 32/150\n",
      " - 5s - loss: 0.0325 - mean_squared_error: 0.0090 - acc: 0.9894 - val_loss: 0.0560 - val_mean_squared_error: 0.0135 - val_acc: 0.9882\n",
      "Epoch 33/150\n",
      " - 5s - loss: 0.0280 - mean_squared_error: 0.0077 - acc: 0.9900 - val_loss: 0.1327 - val_mean_squared_error: 0.0471 - val_acc: 0.8941\n",
      "Epoch 00033: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9988, 0.9828, 0.0103, 0.9591, 0.9428, 0.9509, 0.9734\n",
      "\ttest: 0.9877, 0.9647, 0.0278, 0.8854, 0.9551, 0.9189, 0.9366\n",
      "5 fold\n",
      "Train on 39 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 7s - loss: 0.6251 - mean_squared_error: 0.2175 - acc: 0.6630 - val_loss: 0.4655 - val_mean_squared_error: 0.1416 - val_acc: 0.9176\n",
      "Epoch 2/150\n",
      " - 5s - loss: 0.3430 - mean_squared_error: 0.0987 - acc: 0.8884 - val_loss: 0.3103 - val_mean_squared_error: 0.0818 - val_acc: 0.9500\n",
      "Epoch 3/150\n",
      " - 6s - loss: 0.2022 - mean_squared_error: 0.0548 - acc: 0.9370 - val_loss: 0.2134 - val_mean_squared_error: 0.0664 - val_acc: 0.9118\n",
      "Epoch 4/150\n",
      " - 6s - loss: 0.1483 - mean_squared_error: 0.0401 - acc: 0.9523 - val_loss: 0.2816 - val_mean_squared_error: 0.0957 - val_acc: 0.8324\n",
      "Epoch 5/150\n",
      " - 5s - loss: 0.1323 - mean_squared_error: 0.0371 - acc: 0.9526 - val_loss: 0.1795 - val_mean_squared_error: 0.0577 - val_acc: 0.9206\n",
      "Epoch 6/150\n",
      " - 5s - loss: 0.1247 - mean_squared_error: 0.0347 - acc: 0.9551 - val_loss: 0.1718 - val_mean_squared_error: 0.0535 - val_acc: 0.9294\n",
      "Epoch 7/150\n",
      " - 5s - loss: 0.1181 - mean_squared_error: 0.0343 - acc: 0.9572 - val_loss: 0.2436 - val_mean_squared_error: 0.0870 - val_acc: 0.8176\n",
      "Epoch 8/150\n",
      " - 5s - loss: 0.1029 - mean_squared_error: 0.0287 - acc: 0.9638 - val_loss: 0.1288 - val_mean_squared_error: 0.0393 - val_acc: 0.9529\n",
      "Epoch 9/150\n",
      " - 5s - loss: 0.1007 - mean_squared_error: 0.0279 - acc: 0.9650 - val_loss: 0.1958 - val_mean_squared_error: 0.0659 - val_acc: 0.9294\n",
      "Epoch 10/150\n",
      " - 5s - loss: 0.0880 - mean_squared_error: 0.0250 - acc: 0.9683 - val_loss: 0.1508 - val_mean_squared_error: 0.0462 - val_acc: 0.9529\n",
      "Epoch 11/150\n",
      " - 5s - loss: 0.0842 - mean_squared_error: 0.0242 - acc: 0.9683 - val_loss: 0.3545 - val_mean_squared_error: 0.1213 - val_acc: 0.8118\n",
      "Epoch 12/150\n",
      " - 5s - loss: 0.0857 - mean_squared_error: 0.0253 - acc: 0.9659 - val_loss: 0.1329 - val_mean_squared_error: 0.0398 - val_acc: 0.9647\n",
      "Epoch 13/150\n",
      " - 6s - loss: 0.0764 - mean_squared_error: 0.0224 - acc: 0.9704 - val_loss: 0.2181 - val_mean_squared_error: 0.0720 - val_acc: 0.9088\n",
      "Epoch 14/150\n",
      " - 6s - loss: 0.0933 - mean_squared_error: 0.0266 - acc: 0.9647 - val_loss: 0.1679 - val_mean_squared_error: 0.0534 - val_acc: 0.9382\n",
      "Epoch 15/150\n",
      " - 5s - loss: 0.0760 - mean_squared_error: 0.0223 - acc: 0.9707 - val_loss: 0.2468 - val_mean_squared_error: 0.0837 - val_acc: 0.8441\n",
      "Epoch 16/150\n",
      " - 5s - loss: 0.0753 - mean_squared_error: 0.0213 - acc: 0.9725 - val_loss: 0.0960 - val_mean_squared_error: 0.0262 - val_acc: 0.9765\n",
      "Epoch 17/150\n",
      " - 5s - loss: 0.0640 - mean_squared_error: 0.0191 - acc: 0.9738 - val_loss: 0.0972 - val_mean_squared_error: 0.0269 - val_acc: 0.9706\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18/150\n",
      " - 5s - loss: 0.0612 - mean_squared_error: 0.0179 - acc: 0.9762 - val_loss: 0.0868 - val_mean_squared_error: 0.0230 - val_acc: 0.9765\n",
      "Epoch 19/150\n",
      " - 5s - loss: 0.0551 - mean_squared_error: 0.0163 - acc: 0.9777 - val_loss: 0.0940 - val_mean_squared_error: 0.0250 - val_acc: 0.9735\n",
      "Epoch 20/150\n",
      " - 5s - loss: 0.0896 - mean_squared_error: 0.0249 - acc: 0.9716 - val_loss: 0.1026 - val_mean_squared_error: 0.0260 - val_acc: 0.9735\n",
      "Epoch 21/150\n",
      " - 5s - loss: 0.0748 - mean_squared_error: 0.0218 - acc: 0.9710 - val_loss: 0.0402 - val_mean_squared_error: 0.0104 - val_acc: 0.9853\n",
      "Epoch 22/150\n",
      " - 5s - loss: 0.0679 - mean_squared_error: 0.0198 - acc: 0.9741 - val_loss: 0.2715 - val_mean_squared_error: 0.0898 - val_acc: 0.8500\n",
      "Epoch 23/150\n",
      " - 5s - loss: 0.0756 - mean_squared_error: 0.0222 - acc: 0.9698 - val_loss: 0.1242 - val_mean_squared_error: 0.0385 - val_acc: 0.9588\n",
      "Epoch 24/150\n",
      " - 5s - loss: 0.0691 - mean_squared_error: 0.0200 - acc: 0.9738 - val_loss: 0.1267 - val_mean_squared_error: 0.0396 - val_acc: 0.9588\n",
      "Epoch 25/150\n",
      " - 5s - loss: 0.0536 - mean_squared_error: 0.0153 - acc: 0.9801 - val_loss: 0.1488 - val_mean_squared_error: 0.0496 - val_acc: 0.9294\n",
      "Epoch 26/150\n",
      " - 5s - loss: 0.0456 - mean_squared_error: 0.0132 - acc: 0.9828 - val_loss: 0.1004 - val_mean_squared_error: 0.0293 - val_acc: 0.9735\n",
      "Epoch 27/150\n",
      " - 5s - loss: 0.0444 - mean_squared_error: 0.0132 - acc: 0.9813 - val_loss: 0.0801 - val_mean_squared_error: 0.0216 - val_acc: 0.9824\n",
      "Epoch 28/150\n",
      " - 6s - loss: 0.0406 - mean_squared_error: 0.0122 - acc: 0.9831 - val_loss: 0.0911 - val_mean_squared_error: 0.0247 - val_acc: 0.9765\n",
      "Epoch 29/150\n",
      " - 6s - loss: 0.0374 - mean_squared_error: 0.0110 - acc: 0.9849 - val_loss: 0.0701 - val_mean_squared_error: 0.0184 - val_acc: 0.9824\n",
      "Epoch 30/150\n",
      " - 7s - loss: 0.0510 - mean_squared_error: 0.0151 - acc: 0.9783 - val_loss: 0.0414 - val_mean_squared_error: 0.0098 - val_acc: 0.9882\n",
      "Epoch 31/150\n",
      " - 8s - loss: 0.0697 - mean_squared_error: 0.0174 - acc: 0.9771 - val_loss: 0.0399 - val_mean_squared_error: 0.0108 - val_acc: 0.9882\n",
      "Epoch 32/150\n",
      " - 8s - loss: 0.0695 - mean_squared_error: 0.0188 - acc: 0.9756 - val_loss: 0.1364 - val_mean_squared_error: 0.0270 - val_acc: 0.9706\n",
      "Epoch 33/150\n",
      " - 8s - loss: 0.0631 - mean_squared_error: 0.0176 - acc: 0.9777 - val_loss: 0.0973 - val_mean_squared_error: 0.0276 - val_acc: 0.9765\n",
      "Epoch 34/150\n",
      " - 8s - loss: 0.0410 - mean_squared_error: 0.0119 - acc: 0.9843 - val_loss: 0.0422 - val_mean_squared_error: 0.0116 - val_acc: 0.9853\n",
      "Epoch 35/150\n",
      " - 8s - loss: 0.0379 - mean_squared_error: 0.0100 - acc: 0.9873 - val_loss: 0.0480 - val_mean_squared_error: 0.0128 - val_acc: 0.9824\n",
      "Epoch 36/150\n",
      " - 8s - loss: 0.0360 - mean_squared_error: 0.0105 - acc: 0.9861 - val_loss: 0.0470 - val_mean_squared_error: 0.0131 - val_acc: 0.9824\n",
      "Epoch 37/150\n",
      " - 9s - loss: 0.0421 - mean_squared_error: 0.0117 - acc: 0.9843 - val_loss: 0.0663 - val_mean_squared_error: 0.0174 - val_acc: 0.9824\n",
      "Epoch 38/150\n",
      " - 8s - loss: 0.0482 - mean_squared_error: 0.0130 - acc: 0.9831 - val_loss: 0.0473 - val_mean_squared_error: 0.0116 - val_acc: 0.9882\n",
      "Epoch 39/150\n",
      " - 8s - loss: 0.0393 - mean_squared_error: 0.0115 - acc: 0.9834 - val_loss: 0.0411 - val_mean_squared_error: 0.0105 - val_acc: 0.9882\n",
      "Epoch 40/150\n",
      " - 8s - loss: 0.0297 - mean_squared_error: 0.0080 - acc: 0.9900 - val_loss: 0.1008 - val_mean_squared_error: 0.0296 - val_acc: 0.9618\n",
      "Epoch 00040: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9995, 0.9912, 0.0077, 0.9834, 0.9688, 0.9761, 0.9882\n",
      "\ttest: 0.9976, 0.9765, 0.0185, 0.8676, 0.9833, 0.9219, 0.9324\n",
      "6 fold\n",
      "Train on 40 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 12s - loss: 0.6185 - mean_squared_error: 0.2150 - acc: 0.6600 - val_loss: 0.4484 - val_mean_squared_error: 0.1333 - val_acc: 0.9029\n",
      "Epoch 2/150\n",
      " - 11s - loss: 0.3465 - mean_squared_error: 0.1016 - acc: 0.8715 - val_loss: 0.2878 - val_mean_squared_error: 0.0736 - val_acc: 0.9324\n",
      "Epoch 3/150\n",
      " - 12s - loss: 0.2157 - mean_squared_error: 0.0591 - acc: 0.9306 - val_loss: 0.2025 - val_mean_squared_error: 0.0595 - val_acc: 0.9176\n",
      "Epoch 4/150\n",
      " - 12s - loss: 0.1698 - mean_squared_error: 0.0472 - acc: 0.9376 - val_loss: 0.2093 - val_mean_squared_error: 0.0659 - val_acc: 0.9147\n",
      "Epoch 5/150\n",
      " - 12s - loss: 0.1496 - mean_squared_error: 0.0429 - acc: 0.9435 - val_loss: 0.2174 - val_mean_squared_error: 0.0733 - val_acc: 0.9088\n",
      "Epoch 6/150\n",
      " - 14s - loss: 0.1214 - mean_squared_error: 0.0341 - acc: 0.9574 - val_loss: 0.1367 - val_mean_squared_error: 0.0426 - val_acc: 0.9412\n",
      "Epoch 7/150\n",
      " - 15s - loss: 0.1072 - mean_squared_error: 0.0311 - acc: 0.9597 - val_loss: 0.1749 - val_mean_squared_error: 0.0576 - val_acc: 0.9294\n",
      "Epoch 8/150\n",
      " - 17s - loss: 0.0991 - mean_squared_error: 0.0282 - acc: 0.9644 - val_loss: 0.1891 - val_mean_squared_error: 0.0626 - val_acc: 0.9265\n",
      "Epoch 9/150\n",
      " - 16s - loss: 0.0939 - mean_squared_error: 0.0280 - acc: 0.9635 - val_loss: 0.0663 - val_mean_squared_error: 0.0174 - val_acc: 0.9765\n",
      "Epoch 10/150\n",
      " - 19s - loss: 0.0903 - mean_squared_error: 0.0268 - acc: 0.9635 - val_loss: 0.1054 - val_mean_squared_error: 0.0315 - val_acc: 0.9618\n",
      "Epoch 11/150\n",
      " - 16s - loss: 0.0959 - mean_squared_error: 0.0282 - acc: 0.9618 - val_loss: 0.4275 - val_mean_squared_error: 0.1480 - val_acc: 0.8059\n",
      "Epoch 12/150\n",
      " - 20s - loss: 0.0991 - mean_squared_error: 0.0260 - acc: 0.9676 - val_loss: 0.1628 - val_mean_squared_error: 0.0549 - val_acc: 0.9265\n",
      "Epoch 13/150\n",
      " - 24s - loss: 0.0821 - mean_squared_error: 0.0236 - acc: 0.9685 - val_loss: 0.1083 - val_mean_squared_error: 0.0281 - val_acc: 0.9735\n",
      "Epoch 14/150\n",
      " - 24s - loss: 0.0713 - mean_squared_error: 0.0214 - acc: 0.9709 - val_loss: 0.2194 - val_mean_squared_error: 0.0768 - val_acc: 0.8441\n",
      "Epoch 15/150\n",
      " - 22s - loss: 0.0770 - mean_squared_error: 0.0233 - acc: 0.9676 - val_loss: 0.3007 - val_mean_squared_error: 0.1146 - val_acc: 0.8029\n",
      "Epoch 16/150\n",
      " - 28s - loss: 0.0687 - mean_squared_error: 0.0208 - acc: 0.9703 - val_loss: 0.1606 - val_mean_squared_error: 0.0566 - val_acc: 0.8882\n",
      "Epoch 17/150\n",
      " - 28s - loss: 0.0619 - mean_squared_error: 0.0185 - acc: 0.9750 - val_loss: 0.3484 - val_mean_squared_error: 0.1272 - val_acc: 0.8088\n",
      "Epoch 18/150\n",
      " - 28s - loss: 0.0629 - mean_squared_error: 0.0186 - acc: 0.9765 - val_loss: 0.2500 - val_mean_squared_error: 0.0911 - val_acc: 0.8559\n",
      "Epoch 19/150\n",
      " - 28s - loss: 0.0595 - mean_squared_error: 0.0175 - acc: 0.9779 - val_loss: 0.2853 - val_mean_squared_error: 0.1061 - val_acc: 0.8412\n",
      "Epoch 00019: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9954, 0.9693, 0.0221, 0.9256, 0.9024, 0.9139, 0.9521\n",
      "\ttest: 0.9753, 0.9559, 0.0386, 0.9178, 0.8816, 0.8993, 0.9421\n",
      "7 fold\n",
      "Train on 40 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 35s - loss: 0.4966 - mean_squared_error: 0.1608 - acc: 0.8071 - val_loss: 0.4042 - val_mean_squared_error: 0.1175 - val_acc: 0.8706\n",
      "Epoch 2/150\n",
      " - 29s - loss: 0.3002 - mean_squared_error: 0.0882 - acc: 0.8818 - val_loss: 0.2567 - val_mean_squared_error: 0.0678 - val_acc: 0.9471\n",
      "Epoch 3/150\n",
      " - 30s - loss: 0.1946 - mean_squared_error: 0.0530 - acc: 0.9338 - val_loss: 0.1421 - val_mean_squared_error: 0.0338 - val_acc: 0.9647\n",
      "Epoch 4/150\n",
      " - 29s - loss: 0.1648 - mean_squared_error: 0.0451 - acc: 0.9438 - val_loss: 0.1932 - val_mean_squared_error: 0.0553 - val_acc: 0.9353\n",
      "Epoch 5/150\n",
      " - 29s - loss: 0.1413 - mean_squared_error: 0.0390 - acc: 0.9509 - val_loss: 0.2040 - val_mean_squared_error: 0.0669 - val_acc: 0.9059\n",
      "Epoch 6/150\n",
      " - 28s - loss: 0.1101 - mean_squared_error: 0.0308 - acc: 0.9597 - val_loss: 0.2539 - val_mean_squared_error: 0.0896 - val_acc: 0.8294\n",
      "Epoch 7/150\n",
      " - 28s - loss: 0.0997 - mean_squared_error: 0.0284 - acc: 0.9644 - val_loss: 0.0997 - val_mean_squared_error: 0.0270 - val_acc: 0.9706\n",
      "Epoch 8/150\n",
      " - 28s - loss: 0.0969 - mean_squared_error: 0.0282 - acc: 0.9612 - val_loss: 0.1787 - val_mean_squared_error: 0.0571 - val_acc: 0.9294\n",
      "Epoch 9/150\n",
      " - 30s - loss: 0.0867 - mean_squared_error: 0.0252 - acc: 0.9656 - val_loss: 0.2873 - val_mean_squared_error: 0.1025 - val_acc: 0.8529\n",
      "Epoch 10/150\n",
      " - 27s - loss: 0.0808 - mean_squared_error: 0.0231 - acc: 0.9694 - val_loss: 0.5550 - val_mean_squared_error: 0.1654 - val_acc: 0.8088\n",
      "Epoch 11/150\n",
      " - 29s - loss: 0.0756 - mean_squared_error: 0.0223 - acc: 0.9694 - val_loss: 0.2498 - val_mean_squared_error: 0.0895 - val_acc: 0.8706\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12/150\n",
      " - 24s - loss: 0.0772 - mean_squared_error: 0.0226 - acc: 0.9703 - val_loss: 0.4031 - val_mean_squared_error: 0.1227 - val_acc: 0.8529\n",
      "Epoch 13/150\n",
      " - 25s - loss: 0.0770 - mean_squared_error: 0.0228 - acc: 0.9691 - val_loss: 0.2621 - val_mean_squared_error: 0.0915 - val_acc: 0.8735\n",
      "Epoch 14/150\n",
      " - 28s - loss: 0.0747 - mean_squared_error: 0.0225 - acc: 0.9668 - val_loss: 0.2158 - val_mean_squared_error: 0.0767 - val_acc: 0.8853\n",
      "Epoch 15/150\n",
      " - 28s - loss: 0.0614 - mean_squared_error: 0.0182 - acc: 0.9735 - val_loss: 0.2493 - val_mean_squared_error: 0.0871 - val_acc: 0.8765\n",
      "Epoch 16/150\n",
      " - 28s - loss: 0.0568 - mean_squared_error: 0.0173 - acc: 0.9753 - val_loss: 0.3450 - val_mean_squared_error: 0.1091 - val_acc: 0.8647\n",
      "Epoch 17/150\n",
      " - 28s - loss: 0.0600 - mean_squared_error: 0.0183 - acc: 0.9724 - val_loss: 0.2653 - val_mean_squared_error: 0.0898 - val_acc: 0.8735\n",
      "Epoch 00017: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9947, 0.9676, 0.0237, 0.9332, 0.8922, 0.9123, 0.9542\n",
      "\ttest: 0.9786, 0.9382, 0.0501, 0.8621, 0.7937, 0.8264, 0.9080\n",
      "8 fold\n",
      "Train on 40 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 37s - loss: 0.6007 - mean_squared_error: 0.2072 - acc: 0.6618 - val_loss: 0.4830 - val_mean_squared_error: 0.1502 - val_acc: 0.8853\n",
      "Epoch 2/150\n",
      " - 28s - loss: 0.3129 - mean_squared_error: 0.0907 - acc: 0.8900 - val_loss: 0.2944 - val_mean_squared_error: 0.0768 - val_acc: 0.9412\n",
      "Epoch 3/150\n",
      " - 29s - loss: 0.1680 - mean_squared_error: 0.0445 - acc: 0.9456 - val_loss: 0.1790 - val_mean_squared_error: 0.0473 - val_acc: 0.9412\n",
      "Epoch 4/150\n",
      " - 28s - loss: 0.1217 - mean_squared_error: 0.0323 - acc: 0.9597 - val_loss: 0.1468 - val_mean_squared_error: 0.0381 - val_acc: 0.9588\n",
      "Epoch 5/150\n",
      " - 29s - loss: 0.1210 - mean_squared_error: 0.0339 - acc: 0.9562 - val_loss: 0.1076 - val_mean_squared_error: 0.0273 - val_acc: 0.9647\n",
      "Epoch 6/150\n",
      " - 28s - loss: 0.1055 - mean_squared_error: 0.0292 - acc: 0.9615 - val_loss: 0.0909 - val_mean_squared_error: 0.0234 - val_acc: 0.9735\n",
      "Epoch 7/150\n",
      " - 28s - loss: 0.0981 - mean_squared_error: 0.0263 - acc: 0.9662 - val_loss: 0.0661 - val_mean_squared_error: 0.0168 - val_acc: 0.9765\n",
      "Epoch 8/150\n",
      " - 28s - loss: 0.0900 - mean_squared_error: 0.0240 - acc: 0.9706 - val_loss: 0.0626 - val_mean_squared_error: 0.0172 - val_acc: 0.9765\n",
      "Epoch 9/150\n",
      " - 29s - loss: 0.0778 - mean_squared_error: 0.0220 - acc: 0.9715 - val_loss: 0.0554 - val_mean_squared_error: 0.0156 - val_acc: 0.9794\n",
      "Epoch 10/150\n",
      " - 28s - loss: 0.0716 - mean_squared_error: 0.0203 - acc: 0.9726 - val_loss: 0.0540 - val_mean_squared_error: 0.0151 - val_acc: 0.9794\n",
      "Epoch 11/150\n",
      " - 26s - loss: 0.0684 - mean_squared_error: 0.0192 - acc: 0.9744 - val_loss: 0.0455 - val_mean_squared_error: 0.0126 - val_acc: 0.9824\n",
      "Epoch 12/150\n",
      " - 26s - loss: 0.0678 - mean_squared_error: 0.0188 - acc: 0.9765 - val_loss: 0.0439 - val_mean_squared_error: 0.0124 - val_acc: 0.9824\n",
      "Epoch 13/150\n",
      " - 26s - loss: 0.0604 - mean_squared_error: 0.0174 - acc: 0.9768 - val_loss: 0.0506 - val_mean_squared_error: 0.0147 - val_acc: 0.9735\n",
      "Epoch 14/150\n",
      " - 27s - loss: 0.0567 - mean_squared_error: 0.0164 - acc: 0.9771 - val_loss: 0.0453 - val_mean_squared_error: 0.0145 - val_acc: 0.9765\n",
      "Epoch 15/150\n",
      " - 26s - loss: 0.0594 - mean_squared_error: 0.0174 - acc: 0.9753 - val_loss: 0.0437 - val_mean_squared_error: 0.0113 - val_acc: 0.9882\n",
      "Epoch 16/150\n",
      " - 28s - loss: 0.0579 - mean_squared_error: 0.0169 - acc: 0.9759 - val_loss: 0.0399 - val_mean_squared_error: 0.0118 - val_acc: 0.9824\n",
      "Epoch 17/150\n",
      " - 29s - loss: 0.0489 - mean_squared_error: 0.0139 - acc: 0.9806 - val_loss: 0.0374 - val_mean_squared_error: 0.0113 - val_acc: 0.9824\n",
      "Epoch 18/150\n",
      " - 28s - loss: 0.0447 - mean_squared_error: 0.0127 - acc: 0.9850 - val_loss: 0.0372 - val_mean_squared_error: 0.0107 - val_acc: 0.9853\n",
      "Epoch 19/150\n",
      " - 27s - loss: 0.0466 - mean_squared_error: 0.0136 - acc: 0.9838 - val_loss: 0.0938 - val_mean_squared_error: 0.0254 - val_acc: 0.9647\n",
      "Epoch 20/150\n",
      " - 28s - loss: 0.0606 - mean_squared_error: 0.0172 - acc: 0.9782 - val_loss: 0.0323 - val_mean_squared_error: 0.0069 - val_acc: 0.9941\n",
      "Epoch 21/150\n",
      " - 27s - loss: 0.0511 - mean_squared_error: 0.0148 - acc: 0.9806 - val_loss: 0.0525 - val_mean_squared_error: 0.0149 - val_acc: 0.9794\n",
      "Epoch 22/150\n",
      " - 25s - loss: 0.0677 - mean_squared_error: 0.0186 - acc: 0.9747 - val_loss: 0.0344 - val_mean_squared_error: 0.0081 - val_acc: 0.9941\n",
      "Epoch 23/150\n",
      " - 29s - loss: 0.0531 - mean_squared_error: 0.0153 - acc: 0.9794 - val_loss: 0.0286 - val_mean_squared_error: 0.0076 - val_acc: 0.9912\n",
      "Epoch 24/150\n",
      " - 29s - loss: 0.0436 - mean_squared_error: 0.0127 - acc: 0.9815 - val_loss: 0.0337 - val_mean_squared_error: 0.0086 - val_acc: 0.9882\n",
      "Epoch 25/150\n",
      " - 27s - loss: 0.0365 - mean_squared_error: 0.0103 - acc: 0.9876 - val_loss: 0.0379 - val_mean_squared_error: 0.0100 - val_acc: 0.9912\n",
      "Epoch 26/150\n",
      " - 26s - loss: 0.0341 - mean_squared_error: 0.0096 - acc: 0.9879 - val_loss: 0.0276 - val_mean_squared_error: 0.0075 - val_acc: 0.9912\n",
      "Epoch 27/150\n",
      " - 27s - loss: 0.0375 - mean_squared_error: 0.0110 - acc: 0.9841 - val_loss: 0.0506 - val_mean_squared_error: 0.0149 - val_acc: 0.9853\n",
      "Epoch 28/150\n",
      " - 28s - loss: 0.0480 - mean_squared_error: 0.0128 - acc: 0.9841 - val_loss: 0.0323 - val_mean_squared_error: 0.0091 - val_acc: 0.9882\n",
      "Epoch 29/150\n",
      " - 29s - loss: 0.0388 - mean_squared_error: 0.0111 - acc: 0.9856 - val_loss: 0.0373 - val_mean_squared_error: 0.0102 - val_acc: 0.9882\n",
      "Epoch 30/150\n",
      " - 28s - loss: 0.0356 - mean_squared_error: 0.0101 - acc: 0.9865 - val_loss: 0.0268 - val_mean_squared_error: 0.0080 - val_acc: 0.9882\n",
      "Epoch 00030: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9995, 0.9912, 0.0072, 0.9675, 0.9835, 0.9754, 0.9819\n",
      "\ttest: 0.8913, 0.8588, 0.1217, 0.4107, 0.6053, 0.4894, 0.6789\n",
      "9 fold\n",
      "Train on 41 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 37s - loss: 0.5301 - mean_squared_error: 0.1751 - acc: 0.7670 - val_loss: 0.3964 - val_mean_squared_error: 0.1142 - val_acc: 0.8765\n",
      "Epoch 2/150\n",
      " - 30s - loss: 0.3275 - mean_squared_error: 0.0975 - acc: 0.8743 - val_loss: 0.2610 - val_mean_squared_error: 0.0696 - val_acc: 0.9206\n",
      "Epoch 3/150\n",
      " - 30s - loss: 0.2126 - mean_squared_error: 0.0579 - acc: 0.9300 - val_loss: 0.1075 - val_mean_squared_error: 0.0261 - val_acc: 0.9676\n",
      "Epoch 4/150\n",
      " - 28s - loss: 0.1560 - mean_squared_error: 0.0432 - acc: 0.9469 - val_loss: 0.1582 - val_mean_squared_error: 0.0444 - val_acc: 0.9471\n",
      "Epoch 5/150\n",
      " - 30s - loss: 0.1422 - mean_squared_error: 0.0394 - acc: 0.9501 - val_loss: 0.0637 - val_mean_squared_error: 0.0148 - val_acc: 0.9824\n",
      "Epoch 6/150\n",
      " - 28s - loss: 0.1174 - mean_squared_error: 0.0324 - acc: 0.9593 - val_loss: 0.0786 - val_mean_squared_error: 0.0232 - val_acc: 0.9676\n",
      "Epoch 7/150\n",
      " - 28s - loss: 0.1107 - mean_squared_error: 0.0315 - acc: 0.9584 - val_loss: 0.0406 - val_mean_squared_error: 0.0105 - val_acc: 0.9824\n",
      "Epoch 8/150\n",
      " - 29s - loss: 0.0984 - mean_squared_error: 0.0288 - acc: 0.9618 - val_loss: 0.0377 - val_mean_squared_error: 0.0104 - val_acc: 0.9824\n",
      "Epoch 9/150\n",
      " - 27s - loss: 0.0910 - mean_squared_error: 0.0265 - acc: 0.9638 - val_loss: 0.0622 - val_mean_squared_error: 0.0190 - val_acc: 0.9735\n",
      "Epoch 10/150\n",
      " - 31s - loss: 0.0809 - mean_squared_error: 0.0242 - acc: 0.9661 - val_loss: 0.0359 - val_mean_squared_error: 0.0099 - val_acc: 0.9853\n",
      "Epoch 11/150\n",
      " - 29s - loss: 0.0863 - mean_squared_error: 0.0256 - acc: 0.9630 - val_loss: 0.0327 - val_mean_squared_error: 0.0101 - val_acc: 0.9853\n",
      "Epoch 12/150\n",
      " - 30s - loss: 0.0866 - mean_squared_error: 0.0259 - acc: 0.9644 - val_loss: 0.0262 - val_mean_squared_error: 0.0073 - val_acc: 0.9882\n",
      "Epoch 13/150\n",
      " - 28s - loss: 0.0683 - mean_squared_error: 0.0200 - acc: 0.9719 - val_loss: 0.0267 - val_mean_squared_error: 0.0082 - val_acc: 0.9882\n",
      "Epoch 14/150\n",
      " - 27s - loss: 0.0621 - mean_squared_error: 0.0181 - acc: 0.9776 - val_loss: 0.0253 - val_mean_squared_error: 0.0075 - val_acc: 0.9912\n",
      "Epoch 15/150\n",
      " - 27s - loss: 0.0641 - mean_squared_error: 0.0192 - acc: 0.9742 - val_loss: 0.0214 - val_mean_squared_error: 0.0064 - val_acc: 0.9882\n",
      "Epoch 16/150\n",
      " - 28s - loss: 0.0563 - mean_squared_error: 0.0166 - acc: 0.9770 - val_loss: 0.0394 - val_mean_squared_error: 0.0120 - val_acc: 0.9824\n",
      "Epoch 17/150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - 28s - loss: 0.0575 - mean_squared_error: 0.0171 - acc: 0.9776 - val_loss: 0.0228 - val_mean_squared_error: 0.0068 - val_acc: 0.9882\n",
      "Epoch 18/150\n",
      " - 28s - loss: 0.0542 - mean_squared_error: 0.0169 - acc: 0.9759 - val_loss: 0.0189 - val_mean_squared_error: 0.0051 - val_acc: 0.9941\n",
      "Epoch 19/150\n",
      " - 29s - loss: 0.0571 - mean_squared_error: 0.0164 - acc: 0.9770 - val_loss: 0.0195 - val_mean_squared_error: 0.0057 - val_acc: 0.9941\n",
      "Epoch 20/150\n",
      " - 29s - loss: 0.0609 - mean_squared_error: 0.0186 - acc: 0.9719 - val_loss: 0.0461 - val_mean_squared_error: 0.0145 - val_acc: 0.9794\n",
      "Epoch 21/150\n",
      " - 28s - loss: 0.0529 - mean_squared_error: 0.0147 - acc: 0.9819 - val_loss: 0.0370 - val_mean_squared_error: 0.0120 - val_acc: 0.9824\n",
      "Epoch 22/150\n",
      " - 29s - loss: 0.0501 - mean_squared_error: 0.0145 - acc: 0.9813 - val_loss: 0.0181 - val_mean_squared_error: 0.0052 - val_acc: 0.9912\n",
      "Epoch 23/150\n",
      " - 27s - loss: 0.0453 - mean_squared_error: 0.0128 - acc: 0.9842 - val_loss: 0.0187 - val_mean_squared_error: 0.0056 - val_acc: 0.9882\n",
      "Epoch 24/150\n",
      " - 28s - loss: 0.0427 - mean_squared_error: 0.0129 - acc: 0.9828 - val_loss: 0.0522 - val_mean_squared_error: 0.0146 - val_acc: 0.9794\n",
      "Epoch 25/150\n",
      " - 28s - loss: 0.0495 - mean_squared_error: 0.0142 - acc: 0.9805 - val_loss: 0.0195 - val_mean_squared_error: 0.0054 - val_acc: 0.9941\n",
      "Epoch 26/150\n",
      " - 27s - loss: 0.0813 - mean_squared_error: 0.0197 - acc: 0.9768 - val_loss: 0.0389 - val_mean_squared_error: 0.0104 - val_acc: 0.9882\n",
      "Epoch 27/150\n",
      " - 24s - loss: 0.0516 - mean_squared_error: 0.0149 - acc: 0.9799 - val_loss: 0.0249 - val_mean_squared_error: 0.0074 - val_acc: 0.9882\n",
      "Epoch 28/150\n",
      " - 22s - loss: 0.0481 - mean_squared_error: 0.0136 - acc: 0.9834 - val_loss: 0.0179 - val_mean_squared_error: 0.0052 - val_acc: 0.9912\n",
      "Epoch 00028: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9982, 0.9851, 0.0114, 0.9505, 0.9660, 0.9582, 0.9716\n",
      "\ttest: 0.9754, 0.7216, 0.2346, 0.9556, 0.3839, 0.5478, 0.8135\n",
      "10 fold\n",
      "Train on 41 samples, validate on 4 samples\n",
      "Epoch 1/150\n",
      " - 37s - loss: 0.6303 - mean_squared_error: 0.2207 - acc: 0.6310 - val_loss: 0.4398 - val_mean_squared_error: 0.1290 - val_acc: 0.9294\n",
      "Epoch 2/150\n",
      " - 28s - loss: 0.3568 - mean_squared_error: 0.1043 - acc: 0.8861 - val_loss: 0.2276 - val_mean_squared_error: 0.0526 - val_acc: 0.9735\n",
      "Epoch 3/150\n",
      " - 27s - loss: 0.2264 - mean_squared_error: 0.0620 - acc: 0.9228 - val_loss: 0.1938 - val_mean_squared_error: 0.0574 - val_acc: 0.9235\n",
      "Epoch 4/150\n",
      " - 29s - loss: 0.1618 - mean_squared_error: 0.0432 - acc: 0.9492 - val_loss: 0.2893 - val_mean_squared_error: 0.0979 - val_acc: 0.8029\n",
      "Epoch 5/150\n",
      " - 28s - loss: 0.1399 - mean_squared_error: 0.0382 - acc: 0.9524 - val_loss: 0.3613 - val_mean_squared_error: 0.1262 - val_acc: 0.7971\n",
      "Epoch 6/150\n",
      " - 28s - loss: 0.1267 - mean_squared_error: 0.0353 - acc: 0.9555 - val_loss: 0.1996 - val_mean_squared_error: 0.0631 - val_acc: 0.9206\n",
      "Epoch 7/150\n",
      " - 28s - loss: 0.1072 - mean_squared_error: 0.0292 - acc: 0.9650 - val_loss: 0.2242 - val_mean_squared_error: 0.0736 - val_acc: 0.9176\n",
      "Epoch 8/150\n",
      " - 26s - loss: 0.1047 - mean_squared_error: 0.0298 - acc: 0.9615 - val_loss: 0.4043 - val_mean_squared_error: 0.1374 - val_acc: 0.8000\n",
      "Epoch 9/150\n",
      " - 25s - loss: 0.0934 - mean_squared_error: 0.0268 - acc: 0.9653 - val_loss: 0.0644 - val_mean_squared_error: 0.0182 - val_acc: 0.9765\n",
      "Epoch 10/150\n",
      " - 27s - loss: 0.0890 - mean_squared_error: 0.0260 - acc: 0.9670 - val_loss: 0.6065 - val_mean_squared_error: 0.1802 - val_acc: 0.7824\n",
      "Epoch 11/150\n",
      " - 28s - loss: 0.0967 - mean_squared_error: 0.0277 - acc: 0.9664 - val_loss: 0.1148 - val_mean_squared_error: 0.0322 - val_acc: 0.9647\n",
      "Epoch 12/150\n",
      " - 28s - loss: 0.0932 - mean_squared_error: 0.0270 - acc: 0.9650 - val_loss: 0.0682 - val_mean_squared_error: 0.0190 - val_acc: 0.9735\n",
      "Epoch 13/150\n",
      " - 27s - loss: 0.0810 - mean_squared_error: 0.0238 - acc: 0.9676 - val_loss: 0.1493 - val_mean_squared_error: 0.0498 - val_acc: 0.9118\n",
      "Epoch 14/150\n",
      " - 31s - loss: 0.0818 - mean_squared_error: 0.0243 - acc: 0.9661 - val_loss: 0.0637 - val_mean_squared_error: 0.0160 - val_acc: 0.9882\n",
      "Epoch 15/150\n",
      " - 29s - loss: 0.0829 - mean_squared_error: 0.0250 - acc: 0.9667 - val_loss: 0.1865 - val_mean_squared_error: 0.0675 - val_acc: 0.8794\n",
      "Epoch 16/150\n",
      " - 28s - loss: 0.0701 - mean_squared_error: 0.0206 - acc: 0.9719 - val_loss: 0.1021 - val_mean_squared_error: 0.0326 - val_acc: 0.9500\n",
      "Epoch 17/150\n",
      " - 26s - loss: 0.0637 - mean_squared_error: 0.0191 - acc: 0.9730 - val_loss: 0.2100 - val_mean_squared_error: 0.0767 - val_acc: 0.8676\n",
      "Epoch 18/150\n",
      " - 30s - loss: 0.0708 - mean_squared_error: 0.0206 - acc: 0.9713 - val_loss: 0.3736 - val_mean_squared_error: 0.1191 - val_acc: 0.8382\n",
      "Epoch 19/150\n",
      " - 31s - loss: 0.0674 - mean_squared_error: 0.0183 - acc: 0.9770 - val_loss: 0.1209 - val_mean_squared_error: 0.0411 - val_acc: 0.9265\n",
      "Epoch 20/150\n",
      " - 26s - loss: 0.0561 - mean_squared_error: 0.0167 - acc: 0.9773 - val_loss: 0.1738 - val_mean_squared_error: 0.0630 - val_acc: 0.8941\n",
      "Epoch 21/150\n",
      " - 25s - loss: 0.0499 - mean_squared_error: 0.0146 - acc: 0.9816 - val_loss: 0.0861 - val_mean_squared_error: 0.0268 - val_acc: 0.9676\n",
      "Epoch 22/150\n",
      " - 27s - loss: 0.0526 - mean_squared_error: 0.0154 - acc: 0.9785 - val_loss: 0.1644 - val_mean_squared_error: 0.0597 - val_acc: 0.9000\n",
      "Epoch 23/150\n",
      " - 32s - loss: 0.0586 - mean_squared_error: 0.0179 - acc: 0.9742 - val_loss: 0.0990 - val_mean_squared_error: 0.0322 - val_acc: 0.9647\n",
      "Epoch 24/150\n",
      " - 27s - loss: 0.0656 - mean_squared_error: 0.0192 - acc: 0.9762 - val_loss: 0.3496 - val_mean_squared_error: 0.1082 - val_acc: 0.8588\n",
      "Epoch 00024: early stopping\n",
      "\n",
      "AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:\n",
      "\ttrain: 0.9950, 0.9707, 0.0234, 0.9169, 0.9169, 0.9169, 0.9496\n",
      "\ttest: 0.9982, 0.9725, 0.0192, 0.8793, 1.0000, 0.9358, 0.9397\n"
     ]
    }
   ],
   "source": [
    "# define the stratified 10-fold iterator\n",
    "skfold = skms.StratifiedKFold(n_splits=10, random_state=0)\n",
    "\n",
    "# initialize the scoring metrics \n",
    "# acc: accuracy, auc: area under roc curve, mse: mean squared error, \n",
    "# ba: balanced precision, recall, precison and f1.\n",
    "acc_cv_train=np.zeros(10)\n",
    "acc_cv_test=np.zeros_like(acc_cv_train)\n",
    "d_acc_cv={'train': acc_cv_train, 'test': acc_cv_test}\n",
    "\n",
    "auc_cv_train=np.zeros_like(acc_cv_train)\n",
    "auc_cv_test=np.zeros_like(acc_cv_train)\n",
    "d_auc_cv={'train': auc_cv_train, 'test': auc_cv_test}\n",
    "\n",
    "mse_cv_train=np.zeros_like(acc_cv_train)\n",
    "mse_cv_test=np.zeros_like(acc_cv_train)\n",
    "d_mse_cv={'train': mse_cv_train, 'test': mse_cv_test}\n",
    "\n",
    "recall_cv_train=np.zeros_like(acc_cv_train)\n",
    "recall_cv_test=np.zeros_like(recall_cv_train)\n",
    "d_recall_cv={'train': recall_cv_train, 'test': recall_cv_test}\n",
    "\n",
    "precision_cv_train=np.zeros_like(acc_cv_train)\n",
    "precision_cv_test=np.zeros_like(precision_cv_train)\n",
    "d_precision_cv={'train': precision_cv_train, 'test': precision_cv_test}\n",
    "\n",
    "f1_cv_train=np.zeros_like(acc_cv_train)\n",
    "f1_cv_test=np.zeros_like(f1_cv_train)\n",
    "d_f1_cv={'train': f1_cv_train, 'test': f1_cv_test}\n",
    "\n",
    "ba_cv_train=np.zeros_like(acc_cv_train)\n",
    "ba_cv_test=np.zeros_like(ba_cv_train)\n",
    "d_ba_cv={'train': ba_cv_train, 'test': ba_cv_test}\n",
    "\n",
    "# dictionaries to hold intermediate data and output data\n",
    "d_xtrain={}\n",
    "d_xtest={}\n",
    "d_models = {}\n",
    "\n",
    "# define early stopping criteria for training to avoid overfitting\n",
    "earlystopping = EarlyStopping(monitor='val_mean_squared_error', patience=10, min_delta=0.0001, verbose=1, mode='auto')\n",
    "\n",
    "# stratified 10-fold X-val, within each fold, a new model will be created and training using 9 folds of data\n",
    "for i, (train_index, test_index) in enumerate(skfold.split(X, group)):\n",
    "    print('%d fold' % (i+1))\n",
    "    x_train = X[train_index]\n",
    "    x_test = X[test_index]\n",
    "    \n",
    "    # because of the nature of time-series prediction, the target needs to be of shape (n_samples, n_timesteps, 1)\n",
    "    y_train = Y[train_index].reshape(-1, n_timesteps, 1)  \n",
    "    y_test = Y[test_index].reshape(-1, n_timesteps, 1)\n",
    "    \n",
    "    # preprocessing the data: z-score standardization across time for each feature\n",
    "    scaler = skp.StandardScaler()\n",
    "    x_train = scaler.fit_transform(x_train.reshape((len(train_index)*n_timesteps, n_features))) \\\n",
    "                                    .reshape((len(train_index), n_timesteps, n_features))\n",
    "    x_test = scaler.transform(x_test.reshape((len(test_index)*n_timesteps, n_features))) \\\n",
    "                                    .reshape((len(test_index), n_timesteps, n_features))\n",
    "    d_xtrain[i]=x_train\n",
    "    d_xtest[i]=x_test\n",
    "    \n",
    "    d_x = {'train': x_train, 'test': x_test}\n",
    "    d_y = {'train': y_train, 'test': y_test}\n",
    "    \n",
    "    # building the model:\n",
    "    model = Sequential()\n",
    "    model.add(LSTM(100, return_sequences=True, input_shape=(n_timesteps, n_features)))\n",
    "    model.add(Dropout(0.2))\n",
    "    model.add(Dense(80, activation='relu'))\n",
    "    model.add(Dropout(0.2))\n",
    "    model.add(TimeDistributed(Dense(1, activation='sigmoid')))\n",
    "    \n",
    "    # only print summary during the first fold\n",
    "    if i == 0:\n",
    "        print(model.summary())\n",
    "    \n",
    "    # logging into tensorboard\n",
    "    cv_dir = 'cv%d' % i\n",
    "    cv_dir = os.path.join(logdir, cv_dir)\n",
    "    tensorboard = TensorBoard(log_dir=cv_dir)\n",
    "    \n",
    "    # optimize using Adam with loss for binary prediction\n",
    "    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.0007), metrics=['mean_squared_error', 'accuracy'])  \n",
    "    \n",
    "    # training. use 8% of training samples as validation set.\n",
    "    model.fit(x_train, y_train, shuffle=True, validation_split = 0.08,\n",
    "               batch_size = 1, epochs=150, verbose=2, callbacks=[earlystopping, tensorboard]) #, , aucroc])\n",
    "\n",
    "    d_models[i]=model\n",
    "    \n",
    "    print('')\n",
    "    print('AUC, ACC, MSE, RECALL, PRECISION, F1, BALANCED_ACC:')\n",
    "    for key in d_acc_cv.keys():\n",
    "        pred=model.predict(d_x[key])\n",
    "        d_auc_cv[key][i] = skm.roc_auc_score(d_y[key].flatten(), pred.flatten())\n",
    "        d_acc_cv[key][i] = skm.accuracy_score(d_y[key].flatten(), pred.flatten()>0.5)\n",
    "        d_mse_cv[key][i] = skm.mean_squared_error(d_y[key].flatten(), pred.flatten())\n",
    "        d_recall_cv[key][i] = skm.recall_score(d_y[key].flatten(), pred.flatten()>0.5)\n",
    "        d_precision_cv[key][i] = skm.precision_score(d_y[key].flatten(), pred.flatten()>0.5)\n",
    "        d_f1_cv[key][i] = skm.f1_score(d_y[key].flatten(), pred.flatten()>0.5)\n",
    "        d_ba_cv[key][i] = skm.balanced_accuracy_score(d_y[key].flatten(), pred.flatten()>0.5)\n",
    "        print('\\t%s: %.4f, %.4f, %.4f, %.4f, %.4f, %.4f, %.4f' % \\\n",
    "              (key, d_auc_cv[key][i], d_acc_cv[key][i], d_mse_cv[key][i], d_recall_cv[key][i], \n",
    "                    d_precision_cv[key][i], d_f1_cv[key][i], d_ba_cv[key][i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Average performance over 10 folds. Metrics shown are AUC, Accuracy, MSE, Recall, Precision, F1, and Balanced Accuracy, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: 0.995 (0.007)\n",
      "test: 0.972 (0.032)\n",
      "\n",
      "train: 0.972 (0.021)\n",
      "test: 0.916 (0.078)\n",
      "\n",
      "train: 0.022 (0.018)\n",
      "test: 0.066 (0.064)\n",
      "\n",
      "train: 0.938 (0.027)\n",
      "test: 0.835 (0.145)\n",
      "\n",
      "train: 0.913 (0.075)\n",
      "test: 0.792 (0.201)\n",
      "\n",
      "train: 0.924 (0.052)\n",
      "test: 0.795 (0.157)\n",
      "\n",
      "train: 0.959 (0.023)\n",
      "test: 0.884 (0.079)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for score in [d_auc_cv, d_acc_cv, d_mse_cv, d_recall_cv, d_precision_cv, d_f1_cv, d_ba_cv]:\n",
    "    for j in ['train', 'test']:\n",
    "        print('%s: %.3f (%.3f)' % (j, score[j].mean(), score[j].std()))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the data and cross-validated prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "list_keys=list(d_df_resample_norm_cv.keys())\n",
    "sns.set_context('talk')\n",
    "for i in list_keys:\n",
    "#     print(i)\n",
    "    visualize_prediction_clustermap(d_df_resample_norm_cv[i], columns=feature_cols, \n",
    "                                    row_cluster=True, col_cluster=False, method='centroid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create figures where horizontal axis is time and vertical axis is features which are clustered to show similarity and correlation to the anomaly. Ground truth labels is shown up top as `Anomaly` while `Pred` is our recurrent network prediction.\n",
    "![prediction](https://s3.amazonaws.com/cisco-anomaly-detection/prediction_clustermap_example_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "## Appendix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_test_key=[i for _, i in skfold.split(X, group)]\n",
    "cv_train_key=[i for i, _ in skfold.split(X, group)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([ 0,  1, 12, 20, 21, 35, 36]),\n",
       " array([ 2,  3, 13, 22, 23, 37, 38]),\n",
       " array([ 4, 14, 24, 25, 39, 40]),\n",
       " array([ 5, 15, 26, 27, 41]),\n",
       " array([ 6, 16, 28, 29, 42]),\n",
       " array([ 7, 17, 30, 43]),\n",
       " array([ 8, 18, 31, 44]),\n",
       " array([ 9, 19, 32, 45]),\n",
       " array([10, 33, 46]),\n",
       " array([11, 34, 47])]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_test_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 13, 14, 15, 16, 17, 18, 19,\n",
       "        22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 39, 40,\n",
       "        41, 42, 43, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  4,  5,  6,  7,  8,  9, 10, 11, 12, 14, 15, 16, 17, 18, 19,\n",
       "        20, 21, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 39, 40,\n",
       "        41, 42, 43, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  5,  6,  7,  8,  9, 10, 11, 12, 13, 15, 16, 17, 18,\n",
       "        19, 20, 21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,\n",
       "        38, 41, 42, 43, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  6,  7,  8,  9, 10, 11, 12, 13, 14, 16, 17, 18,\n",
       "        19, 20, 21, 22, 23, 24, 25, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,\n",
       "        38, 39, 40, 42, 43, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  7,  8,  9, 10, 11, 12, 13, 14, 15, 17, 18,\n",
       "        19, 20, 21, 22, 23, 24, 25, 26, 27, 30, 31, 32, 33, 34, 35, 36, 37,\n",
       "        38, 39, 40, 41, 43, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  6,  8,  9, 10, 11, 12, 13, 14, 15, 16, 18,\n",
       "        19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36,\n",
       "        37, 38, 39, 40, 41, 42, 44, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  6,  7,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "        19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36,\n",
       "        37, 38, 39, 40, 41, 42, 43, 45, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  6,  7,  8, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "        18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 34, 35, 36,\n",
       "        37, 38, 39, 40, 41, 42, 43, 44, 46, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 11, 12, 13, 14, 15, 16, 17,\n",
       "        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 34, 35,\n",
       "        36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 47]),\n",
       " array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 12, 13, 14, 15, 16, 17,\n",
       "        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35,\n",
       "        36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46])]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_train_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "d_df_resample_norm_cv={}\n",
    "for i in range(10):\n",
    "    cv_pred=d_models[i].predict(d_xtest[i])\n",
    "    for j in range(cv_pred.shape[0]):\n",
    "        key = '%s_%s' % (i, j)\n",
    "        test_ind = cv_test_key[i][j]\n",
    "        d_df_resample_norm_cv[key] = d_df_resample['2_leaf1_0'].copy()\n",
    "        d_df_resample_norm_cv[key].index = d_df_resample_norm_cv[key]['time']\n",
    "        d_df_resample_norm_cv[key][feature_cols] = d_xtest[i][j]\n",
    "        d_df_resample_norm_cv[key]['Anomaly'] = Y[test_ind].astype(bool)\n",
    "        d_df_resample_norm_cv[key]['Anomaly_orig'] = Y_orig[test_ind].astype(bool)\n",
    "        d_df_resample_norm_cv[key]['Pred'] = cv_pred[j].flatten()>0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2017-06-29 16:34:30', '2017-06-29 16:34:40',\n",
       "               '2017-06-29 16:34:50', '2017-06-29 16:35:00',\n",
       "               '2017-06-29 16:35:10', '2017-06-29 16:35:20',\n",
       "               '2017-06-29 16:35:30', '2017-06-29 16:35:40',\n",
       "               '2017-06-29 16:35:50', '2017-06-29 16:36:00',\n",
       "               '2017-06-29 16:36:10', '2017-06-29 16:36:20',\n",
       "               '2017-06-29 16:36:30', '2017-06-29 16:36:40',\n",
       "               '2017-06-29 16:36:50', '2017-06-29 16:37:00',\n",
       "               '2017-06-29 16:37:10', '2017-06-29 16:37:20',\n",
       "               '2017-06-29 16:37:30', '2017-06-29 16:37:40',\n",
       "               '2017-06-29 16:37:50', '2017-06-29 16:38:00',\n",
       "               '2017-06-29 16:38:10', '2017-06-29 16:38:20',\n",
       "               '2017-06-29 16:38:30', '2017-06-29 16:38:40',\n",
       "               '2017-06-29 16:38:50', '2017-06-29 16:39:00',\n",
       "               '2017-06-29 16:39:10', '2017-06-29 16:39:20',\n",
       "               '2017-06-29 16:39:30', '2017-06-29 16:39:40',\n",
       "               '2017-06-29 16:39:50', '2017-06-29 16:40:00',\n",
       "               '2017-06-29 16:40:10', '2017-06-29 16:40:20',\n",
       "               '2017-06-29 16:40:30', '2017-06-29 16:40:40',\n",
       "               '2017-06-29 16:40:50', '2017-06-29 16:41:00',\n",
       "               '2017-06-29 16:41:10', '2017-06-29 16:41:20',\n",
       "               '2017-06-29 16:41:30', '2017-06-29 16:41:40',\n",
       "               '2017-06-29 16:41:50', '2017-06-29 16:42:00',\n",
       "               '2017-06-29 16:42:10', '2017-06-29 16:42:20',\n",
       "               '2017-06-29 16:42:30', '2017-06-29 16:42:40',\n",
       "               '2017-06-29 16:42:50', '2017-06-29 16:43:00',\n",
       "               '2017-06-29 16:43:10', '2017-06-29 16:43:20',\n",
       "               '2017-06-29 16:43:30', '2017-06-29 16:43:40',\n",
       "               '2017-06-29 16:43:50', '2017-06-29 16:44:00',\n",
       "               '2017-06-29 16:44:10', '2017-06-29 16:44:20',\n",
       "               '2017-06-29 16:44:30', '2017-06-29 16:44:40',\n",
       "               '2017-06-29 16:44:50', '2017-06-29 16:45:00',\n",
       "               '2017-06-29 16:45:10', '2017-06-29 16:45:20',\n",
       "               '2017-06-29 16:45:30', '2017-06-29 16:45:40',\n",
       "               '2017-06-29 16:45:50', '2017-06-29 16:46:00',\n",
       "               '2017-06-29 16:46:10', '2017-06-29 16:46:20',\n",
       "               '2017-06-29 16:46:30', '2017-06-29 16:46:40',\n",
       "               '2017-06-29 16:46:50', '2017-06-29 16:47:00',\n",
       "               '2017-06-29 16:47:10', '2017-06-29 16:47:20',\n",
       "               '2017-06-29 16:47:30', '2017-06-29 16:47:40',\n",
       "               '2017-06-29 16:47:50', '2017-06-29 16:48:00',\n",
       "               '2017-06-29 16:48:10', '2017-06-29 16:48:20',\n",
       "               '2017-06-29 16:48:30'],\n",
       "              dtype='datetime64[ns]', name='time', freq=None)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_df_resample_norm_cv[i].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}