tutorials/5-steps-with-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Keras and Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we show how a Keras model for image recognition can be incorporated into Steps pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import log_loss\n",
    "from sklearn.externals import joblib\n",
    "\n",
    "from keras.models import Sequential, Model, load_model\n",
    "from keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n",
    "from keras import optimizers, regularizers\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from steppy.base import Step, BaseTransformer\n",
    "from steppy.adapter import Adapter, E\n",
    "from toolkit.keras_transformers.models import ClassifierGenerator\n",
    "\n",
    "EXPERIMENT_DIR = './ex5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "# By default pipelines will try to load previously trained models so we delete the cache to ba sure we're starting from scratch\n",
    "shutil.rmtree(EXPERIMENT_DIR, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start off by loading our favourite dataset for digits recognition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()\n",
    "X_digits, y_digits = digits.data, digits.target\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_digits, y_digits, test_size=0.2, stratify=y_digits, random_state=643793)\n",
    "\n",
    "print('{} samples for training'.format(len(y_train)))\n",
    "print('{} samples for test'.format(len(y_valid)))\n",
    "\n",
    "data = {\n",
    "    'input': {\n",
    "        'images': X_train,\n",
    "        'labels': y_train,\n",
    "    },\n",
    "    'input_valid': {\n",
    "        'images': X_valid,\n",
    "        'labels': y_valid\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience let's define a few constants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TARGET_SHAPE = (8, 8, 1)  # Shape of images. In this dataset we have 8x8 pictures.\n",
    "                          # Third dimension stands for the number of channels. We uses grayscale images, so 1 channel only.\n",
    "N_CLASSES = 10  # Number of categories in this classification problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we train a neural net we have to prepare the data properly.\n",
    "\n",
    "Sklearn keeps the digit images as one-dimensional vectors. It's fine for models like XGBoost or RandomForest, because they ignore the two-dimensional nature of images anyway. However, CNNs don't. That's why the first transformer that we define recovers this structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReshapeData(BaseTransformer):        \n",
    "    def transform(self, X, y, **kwargs):\n",
    "        X = X.reshape((X.shape[0], ) + TARGET_SHAPE)\n",
    "        return {\n",
    "            'X': X,\n",
    "            'y': y\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further we use Keras' tool, ImageDataGenerator, for preparation of image stream. It takes care of mundane tasks like standarization, shuffling, augmenting or portioning the stream into batches. Let's create a generator with quite a few online augmentations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrepareDatagen(BaseTransformer):\n",
    "    def fit(self, X, **kwargs): \n",
    "        self.datagen = ImageDataGenerator(\n",
    "            featurewise_center=True,\n",
    "            featurewise_std_normalization=True,\n",
    "            rotation_range=10,\n",
    "            width_shift_range=0.1,\n",
    "            height_shift_range=0.1)\n",
    "        self.datagen.fit(X)\n",
    "        \n",
    "    def transform(self, X, **kwargs):        \n",
    "        return {\n",
    "            'datagen': self.datagen,\n",
    "        }\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.datagen, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.datagen = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put together the first steps of the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reshape_step = Step(\n",
    "    name=\"reshape\",\n",
    "    transformer=ReshapeData(),\n",
    "    input_data=['input'],\n",
    "    adapter=Adapter({\n",
    "        'X': E('input', 'images'),\n",
    "        'y': E('input', 'labels')\n",
    "    }),\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    is_fittable=True\n",
    ")\n",
    "\n",
    "reshape_valid_step = Step(\n",
    "    name=\"reshape_valid\",\n",
    "    transformer=ReshapeData(),\n",
    "    input_data=['input_valid'],\n",
    "    adapter=Adapter({\n",
    "        'X': E('input_valid', 'images'),\n",
    "        'y': E('input_valid', 'labels')\n",
    "    }),\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    is_fittable=True\n",
    ")\n",
    "\n",
    "datagen_step = Step(\n",
    "    name=\"loader\",\n",
    "    transformer=PrepareDatagen(),\n",
    "    input_steps=[reshape_step],\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    is_fittable=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we created a step that reshapes vector representations of train images into two-dimensional arrays. Later we will need validation images in the same form, so we also created an analogous step for them. The third step creates an instance of ImageDataGenerator. It takes as input reshaped train images, so that it can calculate means and variances for standarization.\n",
    "\n",
    "To check that what we did actually works, let's define an auxilliary step that displays the generated image data stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataDisplay(BaseTransformer):\n",
    "    def transform(self, datagen, X, y, **kwargs):\n",
    "        img_batch, lbl_batch = datagen.flow(X, y, batch_size=32).next()\n",
    "        n_row = 4\n",
    "        fix, axs = plt.subplots(n_row, 8, figsize=(8, 2 * n_row))\n",
    "        for i, ax in enumerate(axs.ravel()):\n",
    "            ax.imshow(img_batch[i].reshape(8, 8), cmap='gray')\n",
    "            ax.axis('off')\n",
    "            ax.set_title('lbl = {}'.format(lbl_batch[i]))\n",
    "            \n",
    "display_step = Step(\n",
    "    name=\"display\",\n",
    "    transformer=DataDisplay(),\n",
    "    input_steps=[reshape_step, datagen_step],\n",
    "    adapter=Adapter({\n",
    "        'datagen': E(datagen_step.name, 'datagen'),\n",
    "        'X': E(reshape_step.name, 'X'),\n",
    "        'y': E(reshape_step.name, 'y')\n",
    "    }),\n",
    "    experiment_directory=EXPERIMENT_DIR\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display_step.fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps for CNN training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We proceed to the crux of this notebook: step/transformer wrapping a Keras model. Steps library contains classes that facilitate this task. We will use `ClassifierGenerator` which extends `KerasModelTransformer`. Their design follows a _template method pattern_ which means that the main part of the code is defined in abstract classes and the user has to derive from them and implement some auxiliary methods, in this case: `_build_optimizer`, `_build_loss`, `_build_model`, `_create_callbacks`. That's what we do below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "class KerasCnn(ClassifierGenerator):\n",
    "    def _build_optimizer(self, **kwargs):\n",
    "        return Adam(lr=kwargs['learning_rate'])\n",
    "\n",
    "    def _build_loss(self, **kwargs):\n",
    "        return 'sparse_categorical_crossentropy'\n",
    "    \n",
    "    def _build_model(self, **kwargs):\n",
    "        dropout_ratio = kwargs['dropout_ratio']\n",
    "        regularization = kwargs['regularization']\n",
    "        \n",
    "        input_img = Input(shape=TARGET_SHAPE)\n",
    "\n",
    "        layer = Conv2D(8, kernel_size=(3, 3), padding='same', activation='relu')(input_img)\n",
    "        layer = Conv2D(8, kernel_size=(3, 3), padding='same', activation='relu')(layer)\n",
    "        layer = MaxPooling2D((2, 2), padding='same')(layer)\n",
    "\n",
    "        layer = Conv2D(16, kernel_size=(3, 3), padding='same', activation='relu')(layer)\n",
    "        layer = Conv2D(16, kernel_size=(3, 3), padding='same', activation='relu')(layer)\n",
    "        layer = MaxPooling2D((2, 2), padding='same')(layer)\n",
    "\n",
    "        layer = Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu')(layer)\n",
    "        layer = Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu')(layer)\n",
    "        layer = MaxPooling2D((2, 2), padding='same')(layer)\n",
    "\n",
    "        layer = Flatten()(layer)\n",
    "        layer = Dense(64, activation='relu', kernel_regularizer=regularizers.l2(regularization))(layer)\n",
    "        if dropout_ratio > 0:\n",
    "            layer = Dropout(dropout_ratio)(layer)\n",
    "        predictions = Dense(N_CLASSES, activation='softmax')(layer)\n",
    "\n",
    "        model = Model(input_img, predictions)\n",
    "        return model\n",
    "\n",
    "    def _create_callbacks(self, **kwargs):\n",
    "        checkpoint_filepath = kwargs['model_checkpoint']['filepath']\n",
    "        Path(checkpoint_filepath).parents[0].mkdir(parents=True, exist_ok=True)\n",
    "        model_checkpoint = ModelCheckpoint(**kwargs['model_checkpoint'])\n",
    "        return [model_checkpoint]\n",
    "    \n",
    "    def transform(self, datagen, X, datagen_valid=None, X_valid=None, *args, **kwargs):\n",
    "        flow_args = self.training_config['flow_args']\n",
    "        batch_size = flow_args['batch_size']\n",
    "        train_steps = math.ceil(len(X) / batch_size)\n",
    "        y_proba_train = self.model.predict_generator(\n",
    "            datagen.flow(X, shuffle=False, **flow_args), steps=train_steps)\n",
    "        result = dict(output=y_proba_train)\n",
    "        if X_valid is not None:\n",
    "            if datagen_valid is None:\n",
    "                datagen_valid = datagen\n",
    "            valid_steps = math.ceil(len(X_valid) / batch_size)\n",
    "            y_proba_valid = self.model.predict_generator(\n",
    "                datagen_valid.flow(X_valid, shuffle=False, **flow_args), steps=valid_steps)\n",
    "            result.update(dict(output_valid=y_proba_valid))\n",
    "        return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`KerasModelTransformer`'s initializer takes 3 arguments.\n",
    "1. `architecture_config` - contains model and optimizer parameters.\n",
    "2. `training_config` - contains parameters for model's `fit_generator` and generator's `flow` methods.\n",
    "3. `callbacks_config` - contains parameters for callbacks instantiated in `_create_callbacks` methods.\n",
    "\n",
    "The exact structure of these arguments is best explained on an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "architecture_config = {\n",
    "    'model_params': {\n",
    "        'dropout_ratio': 0.5,\n",
    "        'regularization': 0.01\n",
    "    },\n",
    "    'optimizer_params': {\n",
    "        'learning_rate': 1e-3\n",
    "    }\n",
    "}\n",
    "\n",
    "training_config = {\n",
    "    'fit_args': {\n",
    "        'epochs': 100,\n",
    "        'verbose': True\n",
    "    },\n",
    "    'flow_args': {\n",
    "        'batch_size': 64,\n",
    "    }\n",
    "}\n",
    "\n",
    "callbacks_config = {\n",
    "    'model_checkpoint': {\n",
    "        'filepath': str(Path(EXPERIMENT_DIR) / 'checkpoints' / 'best_model.hdf5'),\n",
    "        'save_best_only': True\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all dependencies necessary to add the crucial step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_step = Step(\n",
    "    name=\"CNN\",\n",
    "    transformer=KerasCnn(architecture_config, training_config, callbacks_config),\n",
    "    input_steps=[datagen_step, reshape_step, reshape_valid_step],\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    adapter=Adapter({\n",
    "        'datagen': E(datagen_step.name, 'datagen'),\n",
    "        'X': E(reshape_step.name, 'X'),\n",
    "        'y': E(reshape_step.name, 'y'),\n",
    "        'X_valid': E(reshape_valid_step.name, 'X'),\n",
    "        'y_valid': E(reshape_valid_step.name, 'y')\n",
    "    }),\n",
    "    is_fittable=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we didn't specify `datagen_valid` the same generator will be used for train and validation data. In particular it means that validation images are augmented as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "result = cnn_step.fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A short function below summarizes the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval_pred(title, y_true, y_pred):\n",
    "    print(title)\n",
    "    print(\"  Log-loss: \", log_loss(y_true=y_true, y_pred=y_pred))\n",
    "    choices = np.argmax(y_pred, axis=1)\n",
    "    print(\"  Accuracy: {:.2%}\".format(np.sum(choices == y_true) / len(y_true)))\n",
    "    \n",
    "eval_pred(\"Results on training\", y_true=y_train, y_pred=result['output'])\n",
    "eval_pred(\"Results on validation\", y_true=y_valid, y_pred=result['output_valid'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we do test time augmentation, it makes sense to run prediction phase a few times and average the results.\n",
    "As we can see below it improves the overall score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "results_valid = []\n",
    "for i in range(10):\n",
    "    print(\"Iteration {}/10\".format(i+1))\n",
    "    results_valid.append(cnn_step.transform(data)['output_valid'])\n",
    "y_avg_pred = np.mean(np.array(results_valid), axis=0)\n",
    "eval_pred(\"Results on averaged predictions\", y_true=y_valid, y_pred=y_avg_pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}