ch04-using-keras-with-tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Using Keras on top of TensorFlow for fast prototyping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Objective**: Build a deep learning model that can learn the alphabet using Keras and compare with TensorFlow implementation\n",
    "\n",
    "**Agenda**\n",
    "1. Recap of the previous chapter\n",
    "2. What is Keras and why should I care?\n",
    "3. Implementing the same model with Keras\n",
    "   - Building the computational graph\n",
    "   - Setting up TensorBoard visualization\n",
    "   - Training and evaluating the model\n",
    "4. Conclusions on the difference between TensorFlow and Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Recap of the previous chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In [Chapter 2](ch02-working-with-tensorflow.ipynb) of the workshop we built a deep learning model capable of predicting the next letter of the alphabet based on an input sequence of consecutive letters.\n",
    "\n",
    "We did so by transforming the objective into a classification problem where each letter of the alphabet becomes an output class and the model needed to learn to predict what is the probability of each letter being the next letter in the alphabet given the input sequence.\n",
    "\n",
    "Afterwards we created a model like in the image below using LSTM cells and a fully connected layer.\n",
    "\n",
    "![Model architecture](./img/model.png)\n",
    "\n",
    "In this chapter we will tackle the same task but this time we will be using a powerfull library which enables us to quickly build and validate models - **Keras**.\n",
    "\n",
    "But first, a few words about Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## What is Keras and why should I care?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### What is Keras?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "As the [documentation](https://keras.io/) specifies,\n",
    "\n",
    "> Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a **focus on enabling fast experimentation**. Being able to go from idea to result with the least possible delay is key to doing good research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Installation and configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "To install Keras in your Python environment just run\n",
    "```\n",
    "pip install keras\n",
    "```\n",
    "\n",
    "After installation you can [configure Keras to run on a specific backend](https://keras.io/backend/#switching-from-one-backend-to-another) by changing (or creating) the configuration file located at\n",
    "- `$HOME/.keras/keras.json` on **Linux** or\n",
    "- `%USERPROFILE%/.keras/keras.json` on **Windows**.\n",
    "\n",
    "The default configuration file looks like this:\n",
    "```\n",
    "{\n",
    "\t\"image_data_format\": \"channels_last\",\n",
    "\t\"epsilon\": 1e-07,\n",
    "\t\"floatx\": \"float32\",\n",
    "\t\"backend\": \"tensorflow\"\n",
    "}\n",
    "```\n",
    "\n",
    "As TensorFlow is the default backend for Keras there's nothing for us to configure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### The two flavors of Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Keras comes in two flavors:\n",
    "- The `Sequential` model which is **a linear stack of layers**\n",
    "- The `Functional API` which allows building **more complicated models** such as models with multiple inputs, non-linear graphs etc.\n",
    "\n",
    "In our exercise we will be using `Sequential` model. The answer to why should I care? will hopefully be self evident after the implementation. Otherwise you'll find it within the conclusions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Implementing the same model with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's build the same model as in previous chapter in Keras to see the difference between those two.\n",
    "\n",
    "As usual, let's start with the imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "As stated earlier we will be using the `Sequential` API and our model contains a `LSTM` layer and a `Dense` layer; let's import those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We will want to see our model graph and statistics in `TensorBoard` but before setting that up we need some more imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from keras.callbacks import TensorBoard\n",
    "import os.path as path\n",
    "import datetime\n",
    "import tempfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Again, for reproducibility, let's set the random seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(2018)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We'll also need the utility constructs from the previous chapter so let's put them here. First the `alphabet` and the `Encoding` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class Encoding():\n",
    "    def __init__(self):\n",
    "        self._char_to_int = dict((c, i) for i, c in enumerate(alphabet))\n",
    "        self._int_to_char = dict((i, c) for i, c in enumerate(alphabet))\n",
    "\n",
    "    def encode_sequence(self, sequence):\n",
    "        return [self._char_to_int[char] for char in sequence]\n",
    "\n",
    "    def encode_letter(self, letter):\n",
    "        return self._char_to_int[letter]\n",
    "\n",
    "    def decode_letter(self, value):\n",
    "        return self._int_to_char[value]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Then the `Dataset` but with a small change in the `_normalize_labels` method: we will no longer need to reshape our outputs to `(1000, 1, 26)`; the Keras model will accept the shape `(1000, 26)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class Dataset():\n",
    "    def __init__(self, size=1000, seq_length=5, print_data=True):\n",
    "        self._size = size\n",
    "        self._sequence_length = seq_length\n",
    "        self._inputs = []\n",
    "        self._labels = []\n",
    "        self._print_data = print_data\n",
    "        self._encoding = Encoding()\n",
    "\n",
    "    @property\n",
    "    def inputs(self):\n",
    "        return self._inputs\n",
    "\n",
    "    @property\n",
    "    def labels(self):\n",
    "        return self._labels\n",
    "\n",
    "    @property\n",
    "    def num_classes(self):\n",
    "        return len(alphabet)\n",
    "\n",
    "    @property\n",
    "    def sequence_length(self):\n",
    "        return self._sequence_length\n",
    "\n",
    "    def initialize(self):\n",
    "        self._generate_random_data()\n",
    "        self._reshape_inputs()\n",
    "        self._normalize_inputs()\n",
    "        self._normalize_labels()\n",
    "\n",
    "    def shuffle(self):\n",
    "        perm = np.arange(self._size)\n",
    "        np.random.shuffle(perm)\n",
    "        self._inputs = self._inputs[perm]\n",
    "        self._labels = self._labels[perm]\n",
    "\n",
    "\n",
    "    def _normalize_labels(self):\n",
    "        self._labels = to_categorical(self._labels, num_classes=self.num_classes)\n",
    "\n",
    "    def _reshape_inputs(self):\n",
    "        self._inputs = pad_sequences(self._inputs,\n",
    "                                     maxlen=self._sequence_length,\n",
    "                                     dtype='float32')\n",
    "        self._inputs = np.reshape(self._inputs, (self._size, self._sequence_length, 1))\n",
    "\n",
    "    def _normalize_inputs(self):\n",
    "        self._inputs = self._inputs / float(self.num_classes)\n",
    "\n",
    "    def _generate_random_data(self):\n",
    "        for i in range(self._size):\n",
    "            start = np.random.randint(self.num_classes - 2)\n",
    "            end = np.random.randint(start, min(start + self._sequence_length, self.num_classes - 1))\n",
    "            input_seq = alphabet[start:end + 1]\n",
    "            output_seq = alphabet[end + 1]\n",
    "\n",
    "            if(self._print_data):\n",
    "                print(\"{}->{}\".format(input_seq, output_seq))\n",
    "\n",
    "            sample = self._encoding.encode_sequence(input_seq)\n",
    "            label = self._encoding.encode_letter(output_seq)\n",
    "\n",
    "            self._inputs.append(sample)\n",
    "            self._labels.append(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Now that the dependencies are imported let's create the required instances and initialize them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "e = Encoding()\n",
    "dataset = Dataset(print_data=False)\n",
    "dataset.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Building the computational graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The first thing we need to do is to declare that this is a `Sequential` model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Now let's add the layers:\n",
    "- A `LSTM` layer with 32 units and input shape `(5, 1)`\n",
    "- A `Dense` layer with 26 units and `softmax` activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model.add(LSTM(32, input_shape=(dataset.sequence_length, 1)))\n",
    "model.add(Dense(dataset.num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "And that is it. We now have built our model.\n",
    "\n",
    "Let's see a summary of it using [`keras.utils.print_summary`](https://keras.io/utils/#print_summary) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Setting up TensorBoard visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Although the summary above does provide some quick info about our model it can't compete with what TensorBoard can offer regarding model visualization so let's plug TensorBoard visualization into the model.\n",
    "\n",
    "To do so, we need to provide a [`Callback`](https://keras.io/callbacks/) to the `fit` method of our model which will write the data to the summary file. Luckily for us, Keras already provides such a callback called [`TensorBoard`](https://keras.io/callbacks/#tensorboard). Let's set it up.\n",
    "\n",
    "Good practices encourage us to write summary files in separate folders per run/training and start TensorBoard in the parent directory so we'll do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "temp_dir = tempfile.gettempdir()\n",
    "run_dir = datetime.datetime.now().strftime('%Y-%m-%d-%H%M')\n",
    "log_dir = path.join(temp_dir, 'tensorflow-workshop', run_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Now that we have set up the output directory for the run let's instantiate the callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "tensorboardDisplay = TensorBoard(log_dir=log_dir,\n",
    "                                 histogram_freq=0,\n",
    "                                 write_graph=True,\n",
    "                                 write_images=True,\n",
    "                                 write_grads=True,\n",
    "                                 batch_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Training and evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Although our model is defined we cannot train it yet because it lacks the optimizer and the loss. To prepare the model for training we need to call the [`compile`](https://keras.io/models/sequential/#compile) method and pass to it the optimizer, loss and metrics we want to track:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Now, let's do the actual training and plug-in the TensorBoard callback to see how loss and accuracy evolve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model.fit(dataset.inputs, dataset.labels, epochs=500, batch_size=1,\n",
    "          callbacks=[tensorboardDisplay])\n",
    "\n",
    "root_dir = path.join(temp_dir, 'tensorflow-workshop')\n",
    "print(\"Training finished.\\nStart TensorBoard in '{}' to visualize the model.\".format(root_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's put the model to test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "seq = input('Enter a sequence of max 5 consecutive letters:')\n",
    "seq = seq.upper()\n",
    "print(\"You entered {}\".format(seq))\n",
    "seq = e.encode_sequence(seq)\n",
    "seq = pad_sequences([seq], dataset.sequence_length)\n",
    "seq = np.reshape(seq, (1, dataset.sequence_length, 1))\n",
    "seq = seq / float(dataset.num_classes)\n",
    "prediction = model.predict(seq)\n",
    "letter = np.argmax(prediction)\n",
    "print(\"The next letter is: {}\".format(e.decode_letter(letter)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Conclusions on the difference between TensorFlow and Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Keras API is shorter and more expressive thus allowing fast prototyping and shorter development time for building and trying a model.\n",
    "- The high-level of abstraction provided by Keras comes with a cost - a loss in flexibility and details\n",
    "- TensorFlow on the other hand, provides a lot of flexibility allowing the developers to control every detail of the model\n",
    "- As a downside the flexibility and level of detail in TensorFlow take a toll on maintenace and complexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Take the best of both worlds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "There are a lot of things to take into consideration when trying to solve problem with a deep learning model. Amongst them is finding the best computational graph to tackle the problem and when it comes to choosing an architecture from a plethora of possibilities use Keras to quickly try a few of them. Once the overall architecture is settled you may want to switch to TensorFlow in order to get fine-grained control of the model and tunning possibilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "* [Keras documentation](https://keras.io)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "name": "ch03-using-keras-with-tensorflow.ipynb"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}