src/Augmenting Feature Maps D&C.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev2 toc-item\"><a href=\"#Baseline-model\" data-toc-modified-id=\"Baseline-model-01\"><span class=\"toc-item-num\">0.1&nbsp;&nbsp;</span>Baseline model</a></div><div class=\"lev2 toc-item\"><a href=\"#Image-Augmentation\" data-toc-modified-id=\"Image-Augmentation-02\"><span class=\"toc-item-num\">0.2&nbsp;&nbsp;</span>Image Augmentation</a></div><div class=\"lev2 toc-item\"><a href=\"#Augmenting-feature-maps\" data-toc-modified-id=\"Augmenting-feature-maps-03\"><span class=\"toc-item-num\">0.3&nbsp;&nbsp;</span>Augmenting feature maps</a></div><div class=\"lev2 toc-item\"><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-04\"><span class=\"toc-item-num\">0.4&nbsp;&nbsp;</span>Conclusion</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "[Data augmentation of convolutional feature maps](http://mtdat.blogspot.com/2017/02/data-augmentation-of-convolutional.html) post explains when we could want to spatially transform features maps.  \n",
    "This notebook is to run experiment to see how data augmentation in feature maps space can improve performance of a classifier.\n",
    "\n",
    "We are going to do the following:  \n",
    "Use VGG with fixed weights as pre-trained network and develop three models simple on top of it\n",
    "\n",
    "1. The simple fully convolutional baseline model\n",
    "2. The model above trained on additional images produced by apply spatila transformations to the original images\n",
    "3. The same model but trained with transformations of feature maps\n",
    "\n",
    "The models will be trained on small subsets(10,20,40,80 samples) of Kaggle's [Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Francois Chollet in [his tutorial]() used Kaggle [Dogs vs Cats Redux](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data) competition data to show how to use image augmentation and pre-trained models to train a classifier small amount of data. Francois used only 2,000 (1000 cats and 1000 dogs) out of 25,000 images to achieve accuracy of 0.94. \n",
    "\n",
    "I will make the task more challenging by training a classifier on very small number of examples starting from 5 samples per class, then 10, 20, and 40 samples per class.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append('..')\n",
    "from util import *\n",
    "\n",
    "import math\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 2)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fpath = full_path('/train/*.jpg')\n",
    "img_df = pd.DataFrame({'path':glob.glob(fpath)})\n",
    "img_df['dog'] = (img_df.path.str.find('train/dog.') > 0).astype(int)\n",
    "img_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit, StratifiedShuffleSplit\n",
    "from sklearn.model_selection import train_test_split\n",
    "test_idx, tmp_idx = next(StratifiedShuffleSplit(train_size = 0.1, test_size = 0.1, random_state= 517).split(img_df, img_df.dog))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#n_samples = 10\n",
    "n_samples = 80"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500,)"
      ]
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(2500,)"
      ]
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80,)"
      ]
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_idx, val_idx = train_test_split(tmp_idx, train_size = n_samples, test_size=500, random_state=801, \n",
    "                                     stratify = img_df.iloc[tmp_idx].dog)\n",
    "val_idx.shape\n",
    "test_idx.shape\n",
    "train_idx.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2500, 2)"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(500, 2)"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 2)"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for idx in [test_idx, val_idx,train_idx]:\n",
    "    img_df.loc[idx].shape\n",
    "    np.mean(img_df.loc[idx].dog)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 2)"
      ]
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(2500, 2)"
      ]
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 2)"
      ]
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_df = img_df.iloc[val_idx]\n",
    "val_df.shape\n",
    "test_df = img_df.iloc[test_idx]\n",
    "test_df.shape\n",
    "train_df = img_df.iloc[train_idx]\n",
    "train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will have a fixed train set of 2500 images, validation set of 500 images, and the train of *n_samples* images.  \n",
    "All the sets have 50% cats and 50% dogs. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((500, 374), 294),\n",
       " ((499, 375), 269),\n",
       " ((374, 500), 28),\n",
       " ((499, 333), 26),\n",
       " ((375, 499), 24),\n",
       " ((320, 239), 20),\n",
       " ((500, 332), 17),\n",
       " ((319, 240), 15),\n",
       " ((300, 224), 12),\n",
       " ((399, 300), 12)]"
      ]
     },
     "execution_count": 278,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img_dims = [load_img(p).size for p in test_df.path]\n",
    "from collections import Counter\n",
    "c = Counter(img_dims)\n",
    "from operator import itemgetter\n",
    "c_sort = sorted(c.items(), key=itemgetter(1), reverse = True)\n",
    "c_sort[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the most common image sizes is (500,374) (and (499, 375) which is virtually the same.  \n",
    "So I'm going to resize all the image to (500,374). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "target_size = (374,500) #(height,width)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((80, 374, 500, 3), (80,))"
      ]
     },
     "execution_count": 343,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "((500, 374, 500, 3), (500,))"
      ]
     },
     "execution_count": 343,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, y_train = prep_x_y(train_df, target_size = target_size)\n",
    "X_train.shape, y_train.shape\n",
    "\n",
    "X_val, y_val = prep_x_y(val_df, target_size = target_size)\n",
    "X_val.shape, y_val.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2500, 374, 500, 3), (2500,))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "X_test, y_test = prep_x_y(test_df, target_size = target_size)\n",
    "X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#list of tuples (n_samples, model name, accuarcy, logloss)\n",
    "res_lst = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline model\n",
    "\n",
    "The first model is to set a baseline it will be a full convolution network attached to top of pre-trained VGG.  \n",
    "As I'm not going to use image augmentation for the baseline I will first run VGG for train, validation, and test, sets and keep output tensors as input to the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "conv_out = attach_top_to_vgg(lambda x: x, layer_name = 'block5_conv3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 345,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 23, 31, 512)"
      ]
     },
     "execution_count": 345,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 23, 31, 512)"
      ]
     },
     "execution_count": 345,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X2_val = conv_out.predict(X_val, batch_size=24)\n",
    "X2_val.shape\n",
    "\n",
    "X2_train = conv_out.predict(X_train, batch_size=24)\n",
    "X2_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 53.8 s, sys: 12.7 s, total: 1min 6s\n",
      "Wall time: 1min 2s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(2500, 23, 31, 512)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time X2_test = conv_out.predict(X_test, batch_size=24)\n",
    "X2_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X2_train, X2_val, X2_test are feature maps produced by *block5_conv3* layer of VGG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fc_bin_m1(x):\n",
    "    \"\"\"\n",
    "    Fully Convolutional binary model\n",
    "    \"\"\"\n",
    "    x = MaxPooling2D((2, 2), strides=(2, 2), name='b1_pool')(x)\n",
    "    x = Convolution2D(256, 1, 1, activation='relu', name='conv_1x1')(x)\n",
    "    x = BatchNormalization(axis=3)(x)\n",
    "    x = Convolution2D(128, 7, 7, activation='relu', border_mode='valid', name='conv_FC1')(x)\n",
    "    x = BatchNormalization(axis=3)(x)\n",
    "    x = Convolution2D(1, 1, 1, activation='sigmoid', border_mode='valid', name='conv_FC_last')(x)\n",
    "    x = GlobalAveragePooling2D(name='avg_pool')(x)\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 346,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "m_name = 'bs'\n",
    "input_shape = (None, None, 512)\n",
    "m_input = Input(shape=input_shape)\n",
    "model = Model(m_input, fc_bin_m1(m_input))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer= Adam(lr=0.001),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 348,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 16 if n_samples > 16 else n_samples\n",
    "batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 349,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 80 samples, validate on 500 samples\n",
      "Epoch 1/30\n",
      "2s - loss: 0.6481 - acc: 0.6500 - val_loss: 0.5314 - val_acc: 0.7640\n",
      "Epoch 2/30\n",
      "0s - loss: 0.5398 - acc: 0.7875 - val_loss: 0.5283 - val_acc: 0.6840\n",
      "Epoch 3/30\n",
      "0s - loss: 0.4509 - acc: 0.9000 - val_loss: 0.5270 - val_acc: 0.6600\n",
      "Epoch 4/30\n",
      "0s - loss: 0.3870 - acc: 0.9125 - val_loss: 0.5134 - val_acc: 0.6680\n",
      "Epoch 5/30\n",
      "0s - loss: 0.3588 - acc: 0.9125 - val_loss: 0.4771 - val_acc: 0.7080\n",
      "Epoch 6/30\n",
      "0s - loss: 0.3135 - acc: 0.9375 - val_loss: 0.4483 - val_acc: 0.7320\n",
      "Epoch 7/30\n",
      "0s - loss: 0.2772 - acc: 0.9500 - val_loss: 0.4133 - val_acc: 0.7640\n",
      "Epoch 8/30\n",
      "0s - loss: 0.2763 - acc: 0.9250 - val_loss: 0.3852 - val_acc: 0.7820\n",
      "Epoch 9/30\n",
      "0s - loss: 0.2360 - acc: 0.9625 - val_loss: 0.3513 - val_acc: 0.8220\n",
      "Epoch 10/30\n",
      "0s - loss: 0.2769 - acc: 0.9125 - val_loss: 0.3322 - val_acc: 0.8340\n",
      "Epoch 11/30\n",
      "0s - loss: 0.2074 - acc: 0.9750 - val_loss: 0.3046 - val_acc: 0.8600\n",
      "Epoch 12/30\n",
      "0s - loss: 0.2387 - acc: 0.9250 - val_loss: 0.2838 - val_acc: 0.8780\n",
      "Epoch 13/30\n",
      "0s - loss: 0.2158 - acc: 0.9750 - val_loss: 0.2749 - val_acc: 0.8880\n",
      "Epoch 14/30\n",
      "0s - loss: 0.1765 - acc: 0.9875 - val_loss: 0.2628 - val_acc: 0.9020\n",
      "Epoch 15/30\n",
      "0s - loss: 0.1803 - acc: 0.9750 - val_loss: 0.2542 - val_acc: 0.9100\n",
      "Epoch 16/30\n",
      "0s - loss: 0.1539 - acc: 0.9875 - val_loss: 0.2454 - val_acc: 0.9160\n",
      "Epoch 17/30\n",
      "0s - loss: 0.1623 - acc: 1.0000 - val_loss: 0.2393 - val_acc: 0.9240\n",
      "Epoch 18/30\n",
      "0s - loss: 0.2087 - acc: 0.9375 - val_loss: 0.2328 - val_acc: 0.9280\n",
      "Epoch 19/30\n",
      "0s - loss: 0.1786 - acc: 0.9875 - val_loss: 0.2297 - val_acc: 0.9340\n",
      "Epoch 20/30\n",
      "0s - loss: 0.1261 - acc: 1.0000 - val_loss: 0.2261 - val_acc: 0.9360\n",
      "Epoch 21/30\n",
      "0s - loss: 0.1281 - acc: 0.9875 - val_loss: 0.2214 - val_acc: 0.9400\n",
      "Epoch 22/30\n",
      "0s - loss: 0.1684 - acc: 0.9750 - val_loss: 0.2163 - val_acc: 0.9480\n",
      "Epoch 23/30\n",
      "0s - loss: 0.1354 - acc: 0.9750 - val_loss: 0.2113 - val_acc: 0.9500\n",
      "Epoch 24/30\n",
      "0s - loss: 0.1581 - acc: 1.0000 - val_loss: 0.2084 - val_acc: 0.9480\n",
      "Epoch 25/30\n",
      "0s - loss: 0.1076 - acc: 1.0000 - val_loss: 0.2059 - val_acc: 0.9520\n",
      "Epoch 26/30\n",
      "0s - loss: 0.0952 - acc: 1.0000 - val_loss: 0.2031 - val_acc: 0.9480\n",
      "Epoch 27/30\n",
      "0s - loss: 0.0919 - acc: 1.0000 - val_loss: 0.2006 - val_acc: 0.9460\n",
      "Epoch 28/30\n",
      "0s - loss: 0.0982 - acc: 1.0000 - val_loss: 0.1984 - val_acc: 0.9460\n",
      "Epoch 29/30\n",
      "0s - loss: 0.0939 - acc: 1.0000 - val_loss: 0.1966 - val_acc: 0.9480\n",
      "Epoch 30/30\n",
      "0s - loss: 0.0796 - acc: 1.0000 - val_loss: 0.1948 - val_acc: 0.9520\n"
     ]
    }
   ],
   "source": [
    "model.optimizer.lr = 1e-5\n",
    "history = model.fit(X2_train, y_train, validation_data=(X2_val, y_val),\n",
    "                    batch_size = batch_size,\n",
    "                    nb_epoch = 30, verbose = 2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 351,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 80 samples, validate on 500 samples\n",
      "Epoch 1/20\n",
      "0s - loss: 0.1336 - acc: 0.9625 - val_loss: 0.1936 - val_acc: 0.9520\n",
      "Epoch 2/20\n",
      "0s - loss: 0.0818 - acc: 1.0000 - val_loss: 0.1925 - val_acc: 0.9480\n",
      "Epoch 3/20\n",
      "0s - loss: 0.0804 - acc: 0.9875 - val_loss: 0.1914 - val_acc: 0.9480\n",
      "Epoch 4/20\n",
      "0s - loss: 0.0979 - acc: 1.0000 - val_loss: 0.1900 - val_acc: 0.9500\n",
      "Epoch 5/20\n",
      "0s - loss: 0.0967 - acc: 1.0000 - val_loss: 0.1884 - val_acc: 0.9500\n",
      "Epoch 6/20\n",
      "0s - loss: 0.0803 - acc: 1.0000 - val_loss: 0.1870 - val_acc: 0.9500\n",
      "Epoch 7/20\n",
      "0s - loss: 0.0635 - acc: 1.0000 - val_loss: 0.1861 - val_acc: 0.9520\n",
      "Epoch 8/20\n",
      "0s - loss: 0.1609 - acc: 0.9625 - val_loss: 0.1857 - val_acc: 0.9500\n",
      "Epoch 9/20\n",
      "0s - loss: 0.0765 - acc: 1.0000 - val_loss: 0.1848 - val_acc: 0.9500\n",
      "Epoch 10/20\n",
      "0s - loss: 0.0620 - acc: 1.0000 - val_loss: 0.1839 - val_acc: 0.9500\n",
      "Epoch 11/20\n",
      "0s - loss: 0.0614 - acc: 1.0000 - val_loss: 0.1831 - val_acc: 0.9460\n",
      "Epoch 12/20\n",
      "0s - loss: 0.0743 - acc: 1.0000 - val_loss: 0.1824 - val_acc: 0.9480\n",
      "Epoch 13/20\n",
      "0s - loss: 0.0725 - acc: 1.0000 - val_loss: 0.1812 - val_acc: 0.9500\n",
      "Epoch 14/20\n",
      "0s - loss: 0.0773 - acc: 1.0000 - val_loss: 0.1801 - val_acc: 0.9520\n",
      "Epoch 15/20\n",
      "0s - loss: 0.0580 - acc: 1.0000 - val_loss: 0.1793 - val_acc: 0.9520\n",
      "Epoch 16/20\n",
      "0s - loss: 0.0635 - acc: 1.0000 - val_loss: 0.1786 - val_acc: 0.9520\n",
      "Epoch 17/20\n",
      "0s - loss: 0.0453 - acc: 1.0000 - val_loss: 0.1779 - val_acc: 0.9500\n",
      "Epoch 18/20\n",
      "0s - loss: 0.0503 - acc: 1.0000 - val_loss: 0.1773 - val_acc: 0.9480\n",
      "Epoch 19/20\n",
      "0s - loss: 0.0384 - acc: 1.0000 - val_loss: 0.1764 - val_acc: 0.9480\n",
      "Epoch 20/20\n",
      "0s - loss: 0.0370 - acc: 1.0000 - val_loss: 0.1757 - val_acc: 0.9480\n",
      "CPU times: user 16.5 s, sys: 708 ms, total: 17.2 s\n",
      "Wall time: 13.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.optimizer.lr = 1e-6\n",
    "history = model.fit(X2_train, y_train, validation_data=(X2_val, y_val),\n",
    "                    batch_size = batch_size,\n",
    "                    nb_epoch = 20, verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 352,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2500, 1)"
      ]
     },
     "execution_count": 352,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 'bs', 0.94079999999999997, 0.19860990840010345)"
      ]
     },
     "execution_count": 352,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974)]"
      ]
     },
     "execution_count": 352,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = model.predict(X2_test)\n",
    "pred.shape\n",
    "pred = reduce_conf(pred)\n",
    "pred_class = (pred > 0.5).astype('float32')\n",
    "sc = (n_samples, m_name, metrics.accuracy_score(y_test, pred_class), metrics.log_loss(y_test, pred))\n",
    "sc\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 353,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974),\n",
       " (80, 'bs', 0.94079999999999997, 0.19860990840010345)]"
      ]
     },
     "execution_count": 353,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_lst.append(sc)\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.93      0.88      0.91      1250\n",
      "        1.0       0.89      0.93      0.91      1250\n",
      "\n",
      "avg / total       0.91      0.91      0.91      2500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test, (pred > 0.5).astype('int')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have got a very strong baseline with very simple and fast model. Thank to using pre-trained model.\n",
    "But let's see if we can improve that with image augmentation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image Augmentation\n",
    "\n",
    "We are going to use Keras ImageDataGenerator to do the following augmentations:\n",
    " random crops, and random horizontal flips.  \n",
    "\n",
    "Why those?\n",
    "Because they are common on ImageNet Challenge. And also because similar transformations could be applied to feature maps. So we are giving feature map transformation a fair chance.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_datagen = ImageDataGenerator(\n",
    "        #shear_range=0.2,\n",
    "        width_shift_range=0.2,\n",
    "        height_shift_range =0.1,\n",
    "        horizontal_flip=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because of image augmentation we cannot use prepared output of VGG convolution layers. So we need to attach our model to VGG. The model is the same model but the training starts from the image level, VGG weights are fixed. And for every batch we run the input through the all layer of VGG - it takes time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_name = 'img_aug'\n",
    "model = attach_top_to_vgg(fc_bin_m1, layer_name = 'block5_conv3')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer= Adam(lr=0.001),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "16s - loss: 0.7018 - acc: 0.5125 - val_loss: 0.7821 - val_acc: 0.4940\n",
      "Epoch 2/30\n",
      "14s - loss: 0.6080 - acc: 0.7500 - val_loss: 0.7070 - val_acc: 0.5180\n",
      "Epoch 3/30\n",
      "14s - loss: 0.5195 - acc: 0.8000 - val_loss: 0.6654 - val_acc: 0.5440\n",
      "Epoch 4/30\n",
      "14s - loss: 0.4838 - acc: 0.8375 - val_loss: 0.6292 - val_acc: 0.5700\n",
      "Epoch 5/30\n",
      "14s - loss: 0.4172 - acc: 0.8500 - val_loss: 0.5862 - val_acc: 0.6100\n",
      "Epoch 6/30\n",
      "14s - loss: 0.3925 - acc: 0.8875 - val_loss: 0.5511 - val_acc: 0.6400\n",
      "Epoch 7/30\n",
      "14s - loss: 0.3754 - acc: 0.8500 - val_loss: 0.5158 - val_acc: 0.6720\n",
      "Epoch 8/30\n",
      "14s - loss: 0.3545 - acc: 0.8875 - val_loss: 0.4804 - val_acc: 0.6940\n",
      "Epoch 9/30\n",
      "14s - loss: 0.3482 - acc: 0.9125 - val_loss: 0.4417 - val_acc: 0.7300\n",
      "Epoch 10/30\n",
      "14s - loss: 0.3137 - acc: 0.9250 - val_loss: 0.4054 - val_acc: 0.7780\n",
      "Epoch 11/30\n",
      "14s - loss: 0.3184 - acc: 0.9250 - val_loss: 0.3746 - val_acc: 0.8060\n",
      "Epoch 12/30\n",
      "14s - loss: 0.3016 - acc: 0.9500 - val_loss: 0.3499 - val_acc: 0.8200\n",
      "Epoch 13/30\n",
      "14s - loss: 0.2728 - acc: 0.9375 - val_loss: 0.3307 - val_acc: 0.8420\n",
      "Epoch 14/30\n",
      "14s - loss: 0.2563 - acc: 0.9500 - val_loss: 0.3157 - val_acc: 0.8680\n",
      "Epoch 15/30\n",
      "14s - loss: 0.2659 - acc: 0.9625 - val_loss: 0.3002 - val_acc: 0.8840\n",
      "Epoch 16/30\n",
      "14s - loss: 0.2460 - acc: 0.9500 - val_loss: 0.2880 - val_acc: 0.9000\n",
      "Epoch 17/30\n",
      "14s - loss: 0.2358 - acc: 0.9750 - val_loss: 0.2788 - val_acc: 0.9040\n",
      "Epoch 18/30\n",
      "14s - loss: 0.2293 - acc: 0.9750 - val_loss: 0.2711 - val_acc: 0.9100\n",
      "Epoch 19/30\n",
      "14s - loss: 0.2186 - acc: 0.9625 - val_loss: 0.2623 - val_acc: 0.9200\n",
      "Epoch 20/30\n",
      "14s - loss: 0.2241 - acc: 0.9625 - val_loss: 0.2533 - val_acc: 0.9240\n",
      "Epoch 21/30\n",
      "14s - loss: 0.2188 - acc: 0.9750 - val_loss: 0.2449 - val_acc: 0.9260\n",
      "Epoch 22/30\n",
      "14s - loss: 0.2208 - acc: 0.9750 - val_loss: 0.2383 - val_acc: 0.9280\n",
      "Epoch 23/30\n",
      "14s - loss: 0.1953 - acc: 0.9625 - val_loss: 0.2325 - val_acc: 0.9300\n",
      "Epoch 24/30\n",
      "14s - loss: 0.1969 - acc: 0.9750 - val_loss: 0.2270 - val_acc: 0.9340\n",
      "Epoch 25/30\n",
      "14s - loss: 0.1829 - acc: 0.9875 - val_loss: 0.2220 - val_acc: 0.9380\n",
      "Epoch 26/30\n",
      "14s - loss: 0.1792 - acc: 0.9625 - val_loss: 0.2178 - val_acc: 0.9380\n",
      "Epoch 27/30\n",
      "14s - loss: 0.1834 - acc: 1.0000 - val_loss: 0.2141 - val_acc: 0.9380\n",
      "Epoch 28/30\n",
      "14s - loss: 0.1774 - acc: 0.9750 - val_loss: 0.2112 - val_acc: 0.9340\n",
      "Epoch 29/30\n",
      "14s - loss: 0.1755 - acc: 0.9875 - val_loss: 0.2082 - val_acc: 0.9420\n",
      "Epoch 30/30\n",
      "14s - loss: 0.1730 - acc: 0.9875 - val_loss: 0.2059 - val_acc: 0.9420\n"
     ]
    }
   ],
   "source": [
    "model.optimizer.lr = 1e-5\n",
    "history = model.fit_generator(train_datagen.flow(X_train, y_train, batch_size = batch_size),\n",
    "                    samples_per_epoch=len(X_train), nb_epoch = 30, verbose = 2,\n",
    "                    validation_data=(X_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 368,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "14s - loss: 0.1656 - acc: 0.9750 - val_loss: 0.2039 - val_acc: 0.9420\n",
      "Epoch 2/20\n",
      "14s - loss: 0.1591 - acc: 0.9750 - val_loss: 0.2013 - val_acc: 0.9420\n",
      "Epoch 3/20\n",
      "14s - loss: 0.1485 - acc: 0.9875 - val_loss: 0.1986 - val_acc: 0.9480\n",
      "Epoch 4/20\n",
      "14s - loss: 0.1612 - acc: 0.9625 - val_loss: 0.1962 - val_acc: 0.9460\n",
      "Epoch 5/20\n",
      "14s - loss: 0.1497 - acc: 1.0000 - val_loss: 0.1942 - val_acc: 0.9460\n",
      "Epoch 6/20\n",
      "14s - loss: 0.1437 - acc: 0.9875 - val_loss: 0.1925 - val_acc: 0.9500\n",
      "Epoch 7/20\n",
      "14s - loss: 0.1336 - acc: 1.0000 - val_loss: 0.1909 - val_acc: 0.9520\n",
      "Epoch 8/20\n",
      "14s - loss: 0.1285 - acc: 1.0000 - val_loss: 0.1895 - val_acc: 0.9520\n",
      "Epoch 9/20\n",
      "14s - loss: 0.1244 - acc: 0.9875 - val_loss: 0.1879 - val_acc: 0.9520\n",
      "Epoch 10/20\n",
      "14s - loss: 0.1382 - acc: 0.9750 - val_loss: 0.1867 - val_acc: 0.9520\n",
      "Epoch 11/20\n",
      "14s - loss: 0.1254 - acc: 0.9875 - val_loss: 0.1856 - val_acc: 0.9480\n",
      "Epoch 12/20\n",
      "14s - loss: 0.1313 - acc: 1.0000 - val_loss: 0.1848 - val_acc: 0.9480\n",
      "Epoch 13/20\n",
      "14s - loss: 0.1260 - acc: 0.9875 - val_loss: 0.1840 - val_acc: 0.9480\n",
      "Epoch 14/20\n",
      "14s - loss: 0.1089 - acc: 1.0000 - val_loss: 0.1833 - val_acc: 0.9480\n",
      "Epoch 15/20\n",
      "14s - loss: 0.1207 - acc: 1.0000 - val_loss: 0.1827 - val_acc: 0.9480\n",
      "Epoch 16/20\n",
      "14s - loss: 0.1062 - acc: 1.0000 - val_loss: 0.1818 - val_acc: 0.9480\n",
      "Epoch 17/20\n",
      "14s - loss: 0.1067 - acc: 1.0000 - val_loss: 0.1809 - val_acc: 0.9480\n",
      "Epoch 18/20\n",
      "14s - loss: 0.1049 - acc: 1.0000 - val_loss: 0.1803 - val_acc: 0.9480\n",
      "Epoch 19/20\n",
      "14s - loss: 0.1108 - acc: 1.0000 - val_loss: 0.1797 - val_acc: 0.9480\n",
      "Epoch 20/20\n",
      "14s - loss: 0.1029 - acc: 1.0000 - val_loss: 0.1790 - val_acc: 0.9480\n",
      "CPU times: user 1min 28s, sys: 9.08 s, total: 1min 37s\n",
      "Wall time: 4min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.optimizer.lr = 1e-6\n",
    "history = model.fit_generator(train_datagen.flow(X_train, y_train, batch_size = batch_size),\n",
    "                    samples_per_epoch=len(X_train), nb_epoch = 20, verbose = 2,\n",
    "                    validation_data=(X_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 369,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2500, 1)"
      ]
     },
     "execution_count": 369,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 'img_aug', 0.93359999999999999, 0.20017809524908661)"
      ]
     },
     "execution_count": 369,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974),\n",
       " (80, 'bs', 0.94079999999999997, 0.19860990840010345),\n",
       " (80, 'fm_aug', 0.94079999999999997, 0.20648835626021028)]"
      ]
     },
     "execution_count": 369,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = model.predict(X_test,batch_size=24)\n",
    "pred.shape\n",
    "pred = reduce_conf(pred)\n",
    "pred_class = (pred > 0.5).astype('float32')\n",
    "sc = (n_samples, m_name, metrics.accuracy_score(y_test, pred_class), metrics.log_loss(y_test, pred))\n",
    "sc\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 370,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974),\n",
       " (80, 'bs', 0.94079999999999997, 0.19860990840010345),\n",
       " (80, 'fm_aug', 0.94079999999999997, 0.20648835626021028),\n",
       " (80, 'img_aug', 0.93359999999999999, 0.20017809524908661)]"
      ]
     },
     "execution_count": 370,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_lst.append(sc)\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have to be patient - training gets 100 times slower. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Augmenting feature maps\n",
    "\n",
    "Now we are going to apply similar spatial transformation to feature maps extracted by the pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def aug_fun(x, cr = (0.1,0.1)):\n",
    "    shape = (x.shape[1],x.shape[2]) #expects a batch of image tensors\n",
    "    crop_shape = ((int(shape[0] - cr[0]*shape[0])), (int(shape[1] - cr[1]*shape[1])))\n",
    "    #print(crop_shape)\n",
    "    stride = [0,0]\n",
    "    stride[0] = np.random.randint(shape[0] - crop_shape[0] + 1)\n",
    "    stride[1] = np.random.randint(shape[1] - crop_shape[1] + 1)\n",
    "    #print(\"sride:\", stride)\n",
    "    res = x[:,stride[0]:(crop_shape[0]+stride[0]),stride[1]:(crop_shape[1]+stride[1]),:]\n",
    "    #and random flip\n",
    "    if np.random.binomial(1,0.5):\n",
    "        res = np.flip(res, axis = 2)\n",
    "    return res\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_crop_gen(X, y, cr = (0.2,0.2)):\n",
    "    while True:\n",
    "        for i in np.random.permutation(X.shape[0]):\n",
    "            res = (aug_fun(X[i:i+1,::], cr), y[i:i+1])\n",
    "            yield res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def aug_fun_flex(x, cr = (0.1,0.1)):\n",
    "    shape = (x.shape[1],x.shape[2]) #expects a batch of image tensors\n",
    "    crop_shape = (shape[0] - np.random.randint(math.ceil(cr[0]*shape[0])+1), \n",
    "                  shape[1] - np.random.randint(math.ceil(cr[1]*shape[1])+1))\n",
    "    #print(crop_shape)\n",
    "    stride = [0,0]\n",
    "    stride[0] = np.random.randint(shape[0] - crop_shape[0] + 1)\n",
    "    stride[1] = np.random.randint(shape[1] - crop_shape[1] + 1)\n",
    "    #print(\"sride:\", stride)\n",
    "    res = x[:,stride[0]:(crop_shape[0]+stride[0]),stride[1]:(crop_shape[1]+stride[1]),:]\n",
    "    #and random flip\n",
    "    if np.random.binomial(1,0.5):\n",
    "        res = np.flip(res, axis = 2)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def crop_gen_flex(X, y, cr = (0.3,0.3), batch_size = 8):\n",
    "    while True:\n",
    "        idx = np.random.permutation(X.shape[0])\n",
    "        for j in range(0, X.shape[0], batch_size):\n",
    "            b_idx = idx[j:(j+batch_size)]\n",
    "            res = (aug_fun_flex(X[b_idx,::], cr), y[b_idx])\n",
    "            yield res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 354,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_gen = gen_batch_wrap(conv_crop_gen(X2_train, y_train, cr = (0.2,0.2)), batch_size = batch_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m_name = 'fm_aug'\n",
    "input_shape = (None, None, 512)\n",
    "m_input = Input(shape=input_shape)\n",
    "model = Model(m_input, fc_bin_m1(m_input))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 382,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer= Adam(lr=0.001),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 383,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "2s - loss: 0.7343 - acc: 0.5500 - val_loss: 0.6855 - val_acc: 0.5620\n",
      "Epoch 2/30\n",
      "0s - loss: 0.6479 - acc: 0.5875 - val_loss: 0.6158 - val_acc: 0.6340\n",
      "Epoch 3/30\n",
      "0s - loss: 0.5221 - acc: 0.7750 - val_loss: 0.6177 - val_acc: 0.5800\n",
      "Epoch 4/30\n",
      "0s - loss: 0.4715 - acc: 0.8500 - val_loss: 0.6221 - val_acc: 0.5760\n",
      "Epoch 5/30\n",
      "0s - loss: 0.3986 - acc: 0.9125 - val_loss: 0.5918 - val_acc: 0.6040\n",
      "Epoch 6/30\n",
      "0s - loss: 0.3876 - acc: 0.9125 - val_loss: 0.5585 - val_acc: 0.6240\n",
      "Epoch 7/30\n",
      "0s - loss: 0.3368 - acc: 0.9375 - val_loss: 0.5114 - val_acc: 0.6760\n",
      "Epoch 8/30\n",
      "0s - loss: 0.2901 - acc: 0.9500 - val_loss: 0.4685 - val_acc: 0.7180\n",
      "Epoch 9/30\n",
      "0s - loss: 0.3008 - acc: 0.9125 - val_loss: 0.4372 - val_acc: 0.7540\n",
      "Epoch 10/30\n",
      "0s - loss: 0.2615 - acc: 0.9875 - val_loss: 0.4075 - val_acc: 0.7820\n",
      "Epoch 11/30\n",
      "0s - loss: 0.2598 - acc: 0.9625 - val_loss: 0.3877 - val_acc: 0.8040\n",
      "Epoch 12/30\n",
      "0s - loss: 0.2812 - acc: 0.9125 - val_loss: 0.3659 - val_acc: 0.8240\n",
      "Epoch 13/30\n",
      "0s - loss: 0.1997 - acc: 0.9750 - val_loss: 0.3443 - val_acc: 0.8460\n",
      "Epoch 14/30\n",
      "0s - loss: 0.2120 - acc: 0.9500 - val_loss: 0.3276 - val_acc: 0.8600\n",
      "Epoch 15/30\n",
      "0s - loss: 0.2063 - acc: 0.9875 - val_loss: 0.3129 - val_acc: 0.8720\n",
      "Epoch 16/30\n",
      "0s - loss: 0.1912 - acc: 0.9750 - val_loss: 0.3000 - val_acc: 0.8840\n",
      "Epoch 17/30\n",
      "0s - loss: 0.2257 - acc: 0.9250 - val_loss: 0.2893 - val_acc: 0.8920\n",
      "Epoch 18/30\n",
      "0s - loss: 0.1804 - acc: 0.9500 - val_loss: 0.2811 - val_acc: 0.9080\n",
      "Epoch 19/30\n",
      "0s - loss: 0.2192 - acc: 0.9500 - val_loss: 0.2717 - val_acc: 0.9160\n",
      "Epoch 20/30\n",
      "0s - loss: 0.1482 - acc: 0.9750 - val_loss: 0.2650 - val_acc: 0.9200\n",
      "Epoch 21/30\n",
      "0s - loss: 0.1630 - acc: 0.9875 - val_loss: 0.2582 - val_acc: 0.9200\n",
      "Epoch 22/30\n",
      "0s - loss: 0.1430 - acc: 0.9875 - val_loss: 0.2523 - val_acc: 0.9280\n",
      "Epoch 23/30\n",
      "0s - loss: 0.1476 - acc: 0.9750 - val_loss: 0.2469 - val_acc: 0.9340\n",
      "Epoch 24/30\n",
      "0s - loss: 0.1459 - acc: 0.9750 - val_loss: 0.2425 - val_acc: 0.9340\n",
      "Epoch 25/30\n",
      "0s - loss: 0.1453 - acc: 0.9750 - val_loss: 0.2380 - val_acc: 0.9380\n",
      "Epoch 26/30\n",
      "0s - loss: 0.1356 - acc: 1.0000 - val_loss: 0.2347 - val_acc: 0.9400\n",
      "Epoch 27/30\n",
      "0s - loss: 0.1323 - acc: 0.9875 - val_loss: 0.2313 - val_acc: 0.9420\n",
      "Epoch 28/30\n",
      "0s - loss: 0.1223 - acc: 0.9875 - val_loss: 0.2276 - val_acc: 0.9460\n",
      "Epoch 29/30\n",
      "0s - loss: 0.1495 - acc: 1.0000 - val_loss: 0.2242 - val_acc: 0.9440\n",
      "Epoch 30/30\n",
      "0s - loss: 0.1108 - acc: 0.9875 - val_loss: 0.2213 - val_acc: 0.9440\n"
     ]
    }
   ],
   "source": [
    "model.optimizer.lr = 1e-5\n",
    "history = model.fit_generator(train_gen,\n",
    "                    samples_per_epoch=len(X2_train), nb_epoch = 30, verbose = 2,\n",
    "                    validation_data=(X2_val, y_val))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "0s - loss: 0.1087 - acc: 0.9875 - val_loss: 0.2179 - val_acc: 0.9440\n",
      "Epoch 2/20\n",
      "0s - loss: 0.1159 - acc: 0.9875 - val_loss: 0.2150 - val_acc: 0.9420\n",
      "Epoch 3/20\n",
      "0s - loss: 0.1145 - acc: 0.9875 - val_loss: 0.2125 - val_acc: 0.9420\n",
      "Epoch 4/20\n",
      "0s - loss: 0.1232 - acc: 0.9750 - val_loss: 0.2101 - val_acc: 0.9420\n",
      "Epoch 5/20\n",
      "0s - loss: 0.1088 - acc: 0.9875 - val_loss: 0.2082 - val_acc: 0.9420\n",
      "Epoch 6/20\n",
      "0s - loss: 0.1088 - acc: 1.0000 - val_loss: 0.2064 - val_acc: 0.9420\n",
      "Epoch 7/20\n",
      "0s - loss: 0.1036 - acc: 0.9875 - val_loss: 0.2047 - val_acc: 0.9420\n",
      "Epoch 8/20\n",
      "0s - loss: 0.0779 - acc: 1.0000 - val_loss: 0.2034 - val_acc: 0.9460\n",
      "Epoch 9/20\n",
      "0s - loss: 0.0845 - acc: 1.0000 - val_loss: 0.2016 - val_acc: 0.9440\n",
      "Epoch 10/20\n",
      "0s - loss: 0.0854 - acc: 1.0000 - val_loss: 0.2000 - val_acc: 0.9440\n",
      "Epoch 11/20\n",
      "0s - loss: 0.0957 - acc: 1.0000 - val_loss: 0.1982 - val_acc: 0.9440\n",
      "Epoch 12/20\n",
      "0s - loss: 0.0901 - acc: 1.0000 - val_loss: 0.1966 - val_acc: 0.9460\n",
      "Epoch 13/20\n",
      "0s - loss: 0.1016 - acc: 0.9875 - val_loss: 0.1952 - val_acc: 0.9480\n",
      "Epoch 14/20\n",
      "0s - loss: 0.0696 - acc: 1.0000 - val_loss: 0.1936 - val_acc: 0.9480\n",
      "Epoch 15/20\n",
      "0s - loss: 0.0862 - acc: 0.9875 - val_loss: 0.1921 - val_acc: 0.9480\n",
      "Epoch 16/20\n",
      "0s - loss: 0.0874 - acc: 1.0000 - val_loss: 0.1907 - val_acc: 0.9480\n",
      "Epoch 17/20\n",
      "0s - loss: 0.1092 - acc: 0.9875 - val_loss: 0.1897 - val_acc: 0.9480\n",
      "Epoch 18/20\n",
      "0s - loss: 0.1092 - acc: 0.9875 - val_loss: 0.1888 - val_acc: 0.9480\n",
      "Epoch 19/20\n",
      "0s - loss: 0.0580 - acc: 1.0000 - val_loss: 0.1882 - val_acc: 0.9480\n",
      "Epoch 20/20\n",
      "0s - loss: 0.0799 - acc: 1.0000 - val_loss: 0.1876 - val_acc: 0.9460\n",
      "CPU times: user 16 s, sys: 532 ms, total: 16.5 s\n",
      "Wall time: 12.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.optimizer.lr = 1e-6\n",
    "history = model.fit_generator(train_gen,\n",
    "                    samples_per_epoch=len(X2_train), nb_epoch = 20, verbose = 2,\n",
    "                    validation_data=(X2_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 360,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2500, 1)"
      ]
     },
     "execution_count": 360,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(80, 'fm_aug', 0.94079999999999997, 0.20648835626021028)"
      ]
     },
     "execution_count": 360,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974),\n",
       " (80, 'bs', 0.94079999999999997, 0.19860990840010345)]"
      ]
     },
     "execution_count": 360,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = model.predict(X2_test)\n",
    "pred.shape\n",
    "pred = reduce_conf(pred)\n",
    "pred_class = (pred > 0.5).astype('float32')\n",
    "sc = (n_samples, m_name, metrics.accuracy_score(y_test, pred_class), metrics.log_loss(y_test, pred))\n",
    "sc\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 362,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(10, 'fm_aug', 0.85319999999999996, 0.36306085548214612),\n",
       " (10, 'bs', 0.73480000000000001, 0.47574473093301056),\n",
       " (10, 'img_aug', 0.80600000000000005, 0.39488133698888123),\n",
       " (20, 'bs', 0.90800000000000003, 0.29638660823404789),\n",
       " (20, 'fm_aug', 0.91479999999999995, 0.26443358783498405),\n",
       " (20, 'img_aug', 0.93200000000000005, 0.25656785827279088),\n",
       " (40, 'bs', 0.92879999999999996, 0.2364862256679684),\n",
       " (40, 'fm_aug', 0.93320000000000003, 0.22113356392942368),\n",
       " (40, 'img_aug', 0.93240000000000001, 0.21645076183713974),\n",
       " (80, 'bs', 0.94079999999999997, 0.19860990840010345),\n",
       " (80, 'fm_aug', 0.94079999999999997, 0.20648835626021028)]"
      ]
     },
     "execution_count": 362,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_lst.append(sc)\n",
    "res_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 388,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pred_lst = [model.predict(aug_fun(X2_test, cr = (0.1,0.3))) for i in range(10)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 389,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.17868495296649634,\n",
       " 0.18569375315867365,\n",
       " 0.20369601135216653,\n",
       " 0.18570210080705582,\n",
       " 0.18602345951274038,\n",
       " 0.19475754460133612,\n",
       " 0.20732106727436184,\n",
       " 0.18169053114876152,\n",
       " 0.18219134603515266,\n",
       " 0.17960556001923977]"
      ]
     },
     "execution_count": 389,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.1826728599641472"
      ]
     },
     "execution_count": 389,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0.94040000000000001"
      ]
     },
     "execution_count": 389,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc = [metrics.log_loss(y_test, reduce_conf(pr)) for pr in pred_lst] \n",
    "sc\n",
    "pred = reduce_conf(np.mean(pred_lst, axis = 0))\n",
    "metrics.log_loss(y_test, pred)\n",
    "pred_class = (pred > 0.5).astype('float32')\n",
    "metrics.accuracy_score(y_test, pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 391,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(res_lst, columns=['n_samples','model','accuracy','logloss'])\n",
    "df = df.sort_values(by=['n_samples','model'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['sec/epoch'] = 0\n",
    "df = df.replace({'model':{'bs':'baseline','img_aug':'image augmentation','fm_aug':'feature maps augmentation'}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 390,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "n_samples      int64\n",
       "model         object\n",
       "accuracy     float64\n",
       "logloss      float64\n",
       "sec/epoch      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 390,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_samples</th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>logloss</th>\n",
       "      <th>sec/epoch</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "      <td>baseline</td>\n",
       "      <td>0.7348</td>\n",
       "      <td>0.475745</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>feature maps augmentation</td>\n",
       "      <td>0.8532</td>\n",
       "      <td>0.363061</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>image augmentation</td>\n",
       "      <td>0.8060</td>\n",
       "      <td>0.394881</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20</td>\n",
       "      <td>baseline</td>\n",
       "      <td>0.9080</td>\n",
       "      <td>0.296387</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>feature maps augmentation</td>\n",
       "      <td>0.9148</td>\n",
       "      <td>0.264434</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20</td>\n",
       "      <td>image augmentation</td>\n",
       "      <td>0.9320</td>\n",
       "      <td>0.256568</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>40</td>\n",
       "      <td>baseline</td>\n",
       "      <td>0.9288</td>\n",
       "      <td>0.236486</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>40</td>\n",
       "      <td>feature maps augmentation</td>\n",
       "      <td>0.9332</td>\n",
       "      <td>0.221134</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>40</td>\n",
       "      <td>image augmentation</td>\n",
       "      <td>0.9324</td>\n",
       "      <td>0.216451</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>80</td>\n",
       "      <td>baseline</td>\n",
       "      <td>0.9408</td>\n",
       "      <td>0.198610</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>80</td>\n",
       "      <td>feature maps augmentation</td>\n",
       "      <td>0.9408</td>\n",
       "      <td>0.206488</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>80</td>\n",
       "      <td>image augmentation</td>\n",
       "      <td>0.9336</td>\n",
       "      <td>0.200178</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    n_samples                      model  accuracy   logloss  sec/epoch\n",
       "1          10                   baseline    0.7348  0.475745          0\n",
       "0          10  feature maps augmentation    0.8532  0.363061          0\n",
       "2          10         image augmentation    0.8060  0.394881          0\n",
       "3          20                   baseline    0.9080  0.296387          0\n",
       "4          20  feature maps augmentation    0.9148  0.264434          0\n",
       "5          20         image augmentation    0.9320  0.256568          0\n",
       "6          40                   baseline    0.9288  0.236486          0\n",
       "7          40  feature maps augmentation    0.9332  0.221134          0\n",
       "8          40         image augmentation    0.9324  0.216451          0\n",
       "9          80                   baseline    0.9408  0.198610          0\n",
       "10         80  feature maps augmentation    0.9408  0.206488          0\n",
       "11         80         image augmentation    0.9336  0.200178          0"
      ]
     },
     "execution_count": 390,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#df\n",
    "df.dtypes\n",
    "#print(df.to_html(index = False, float_format='%.3f'))\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "139px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": false,
   "threshold": 4,
   "toc_cell": true,
   "toc_position": {
    "height": "192px",
    "left": "925.952px",
    "right": "20px",
    "top": "-0.0426136px",
    "width": "173px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}