Attention Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention Tutorial\n",
    "\n",
    "One of the most influential and interesting new neural networks types is the attention network. It's been used succesfully in translation services, [medical diagnosis](https://arxiv.org/pdf/1710.08312.pdf), and other tasks.\n",
    "\n",
    "Below we'll be walking through how to write your very own attention network. Our goal is to make a network that can translate human written times ('quarter after 3 pm') to military time ('15:15').\n",
    "\n",
    "The attention mechamism is defined in section **Model**.\n",
    "\n",
    "For a tutorial on how Attention Networks work, please visit [MuffinTech](http://muffintech.org/blog/id/12).\n",
    "\n",
    "Credit to Andrew Ng for reference model and inspiration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Imports\n",
    "from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply, Reshape\n",
    "from keras.layers import RepeatVector, Dense, Activation, Lambda\n",
    "from keras.optimizers import Adam\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import load_model, Model\n",
    "from keras.callbacks import LearningRateScheduler\n",
    "import keras.backend as K\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import random\n",
    "import math\n",
    "import json\n",
    "\n",
    "# Pinkie Pie was here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "The dataset was created using some simple rules. It is not exhaustive, but provides some very nice challenges.\n",
    "\n",
    "The dataset is included in the Github repo.\n",
    "\n",
    "Some example data pairs are listed below:\n",
    "\n",
    "['48 min before 10 a.m', '09:12']  \n",
    "['t11:36', '11:36']  \n",
    "[\"nine o'clock forty six p.m\", '21:46']  \n",
    "['2:59p.m.', '14:59']  \n",
    "['23 min after 20 p.m.', '20:23']  \n",
    "['46 min after seven p.m.', '19:46']  \n",
    "['10 before nine pm', '20:50']  \n",
    "['3.20', '03:20']  \n",
    "['7.57', '07:57']  \n",
    "['six hours and fifty five am', '06:55']  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/Time Dataset.json','r') as f:\n",
    "    dataset = json.loads(f.read())\n",
    "with open('data/Time Vocabs.json','r') as f:\n",
    "    human_vocab, machine_vocab = json.loads(f.read())\n",
    "    \n",
    "human_vocab_size = len(human_vocab)\n",
    "machine_vocab_size = len(machine_vocab)\n",
    "\n",
    "# Number of training examples\n",
    "m = len(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's define some general helper methods. They are used to help tokenize data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty):\n",
    "    \"\"\"\n",
    "    A method for tokenizing data.\n",
    "    \n",
    "    Inputs:\n",
    "    dataset - A list of sentence data pairs.\n",
    "    human_vocab - A dictionary of tokens (char) to id's.\n",
    "    machine_vocab - A dictionary of tokens (char) to id's.\n",
    "    Tx - X data size\n",
    "    Ty - Y data size\n",
    "    \n",
    "    Outputs:\n",
    "    X - Sparse tokens for X data\n",
    "    Y - Sparse tokens for Y data\n",
    "    Xoh - One hot tokens for X data\n",
    "    Yoh - One hot tokens for Y data\n",
    "    \"\"\"\n",
    "    \n",
    "    # Metadata\n",
    "    m = len(dataset)\n",
    "    \n",
    "    # Initialize\n",
    "    X = np.zeros([m, Tx], dtype='int32')\n",
    "    Y = np.zeros([m, Ty], dtype='int32')\n",
    "    \n",
    "    # Process data\n",
    "    for i in range(m):\n",
    "        data = dataset[i]\n",
    "        X[i] = np.array(tokenize(data[0], human_vocab, Tx))\n",
    "        Y[i] = np.array(tokenize(data[1], machine_vocab, Ty))\n",
    "    \n",
    "    # Expand one hots\n",
    "    Xoh = oh_2d(X, len(human_vocab))\n",
    "    Yoh = oh_2d(Y, len(machine_vocab))\n",
    "    \n",
    "    return (X, Y, Xoh, Yoh)\n",
    "    \n",
    "def tokenize(sentence, vocab, length):\n",
    "    \"\"\"\n",
    "    Returns a series of id's for a given input token sequence.\n",
    "    \n",
    "    It is advised that the vocab supports <pad> and <unk>.\n",
    "    \n",
    "    Inputs:\n",
    "    sentence - Series of tokens\n",
    "    vocab - A dictionary from token to id\n",
    "    length - Max number of tokens to consider\n",
    "    \n",
    "    Outputs:\n",
    "    tokens - \n",
    "    \"\"\"\n",
    "    tokens = [0]*length\n",
    "    for i in range(length):\n",
    "        char = sentence[i] if i < len(sentence) else \"<pad>\"\n",
    "        char = char if (char in vocab) else \"<unk>\"\n",
    "        tokens[i] = vocab[char]\n",
    "        \n",
    "    return tokens\n",
    "\n",
    "def ids_to_keys(sentence, vocab):\n",
    "    \"\"\"\n",
    "    Converts a series of id's into the keys of a dictionary.\n",
    "    \"\"\"\n",
    "    return [list(vocab.keys())[id] for id in sentence]\n",
    "\n",
    "def oh_2d(dense, max_value):\n",
    "    \"\"\"\n",
    "    Create a one hot array for the 2D input dense array.\n",
    "    \"\"\"\n",
    "    # Initialize\n",
    "    oh = np.zeros(np.append(dense.shape, [max_value]))\n",
    "    \n",
    "    # Set correct indices\n",
    "    ids1, ids2 = np.meshgrid(np.arange(dense.shape[0]), np.arange(dense.shape[1]))\n",
    "    \n",
    "    oh[ids1.flatten(), ids2.flatten(), dense.flatten('F').astype(int)] = 1\n",
    "    \n",
    "    return oh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our next goal is to tokenize the data using our vocabularies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Tx = 41 # Max x sequence length\n",
    "Ty = 5 # y sequence length\n",
    "X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)\n",
    "\n",
    "# Split data 80-20 between training and test\n",
    "train_size = int(0.8*m)\n",
    "Xoh_train = Xoh[:train_size]\n",
    "Yoh_train = Yoh[:train_size]\n",
    "Xoh_test = Xoh[train_size:]\n",
    "Yoh_test = Yoh[train_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be careful, let's check that the code works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data point 4.\n",
      "\n",
      "The data input is: 8:25\n",
      "The data output is: 08:25\n",
      "\n",
      "The tokenized input is:[11 13  5  8 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40\n",
      " 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40]\n",
      "The tokenized output is: [ 0  8 10  2  5]\n",
      "\n",
      "The one-hot input is: [[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]]\n",
      "The one-hot output is: [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n",
      " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "i = 4\n",
    "print(\"Input data point \" + str(i) + \".\")\n",
    "print(\"\")\n",
    "print(\"The data input is: \" + str(dataset[i][0]))\n",
    "print(\"The data output is: \" + str(dataset[i][1]))\n",
    "print(\"\")\n",
    "print(\"The tokenized input is:\" + str(X[i]))\n",
    "print(\"The tokenized output is: \" + str(Y[i]))\n",
    "print(\"\")\n",
    "print(\"The one-hot input is:\", Xoh[i])\n",
    "print(\"The one-hot output is:\", Yoh[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "Our next goal is to define our model. The important part will be defining the attention mechanism and then making sure to apply that correctly.\n",
    "\n",
    "Define some model metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer1_size = 32\n",
    "layer2_size = 64 # Attention layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next two code snippets defined the attention mechanism. This is split into two arcs:\n",
    "\n",
    "* Calculating context\n",
    "* Creating an attention layer\n",
    "\n",
    "As a refresher, an attention network pays attention to certain parts of the input at each output time step. _attention_ denotes which inputs are most relevant to the current output step. An input step will have attention weight ~1 if it is relevant, and ~0 otherwise. The _context_ is the \"summary of the input\".\n",
    "\n",
    "The requirements are thus. The attention matrix should have shape $(T_x)$ and sum to 1. Additionally, the context should be calculated in the same manner for each time step. Beyond that, there is some flexibility. This notebook calculates both this way:\n",
    "\n",
    "$$\n",
    "attention = Softmax(Dense(Dense(x, y_{t-1})))\n",
    "$$\n",
    "<br/>\n",
    "$$\n",
    "context = \\sum_{i=1}^{m} ( attention_i * x_i )\n",
    "$$\n",
    "\n",
    "For safety, $y_0$ is defined as $\\vec{0}$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define part of the attention layer gloablly so as to\n",
    "# share the same layers for each attention step.\n",
    "def softmax(x):\n",
    "    return K.softmax(x, axis=1)\n",
    "\n",
    "at_repeat = RepeatVector(Tx)\n",
    "at_concatenate = Concatenate(axis=-1)\n",
    "at_dense1 = Dense(8, activation=\"tanh\")\n",
    "at_dense2 = Dense(1, activation=\"relu\")\n",
    "at_softmax = Activation(softmax, name='attention_weights')\n",
    "at_dot = Dot(axes=1)\n",
    "\n",
    "def one_step_of_attention(h_prev, a):\n",
    "    \"\"\"\n",
    "    Get the context.\n",
    "    \n",
    "    Input:\n",
    "    h_prev - Previous hidden state of a RNN layer (m, n_h)\n",
    "    a - Input data, possibly processed (m, Tx, n_a)\n",
    "    \n",
    "    Output:\n",
    "    context - Current context (m, Tx, n_a)\n",
    "    \"\"\"\n",
    "    # Repeat vector to match a's dimensions\n",
    "    h_repeat = at_repeat(h_prev)\n",
    "    # Calculate attention weights\n",
    "    i = at_concatenate([a, h_repeat])\n",
    "    i = at_dense1(i)\n",
    "    i = at_dense2(i)\n",
    "    attention = at_softmax(i)\n",
    "    # Calculate the context\n",
    "    context = at_dot([attention, a])\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def attention_layer(X, n_h, Ty):\n",
    "    \"\"\"\n",
    "    Creates an attention layer.\n",
    "    \n",
    "    Input:\n",
    "    X - Layer input (m, Tx, x_vocab_size)\n",
    "    n_h - Size of LSTM hidden layer\n",
    "    Ty - Timesteps in output sequence\n",
    "    \n",
    "    Output:\n",
    "    output - The output of the attention layer (m, Tx, n_h)\n",
    "    \"\"\"    \n",
    "    # Define the default state for the LSTM layer\n",
    "    h = Lambda(lambda X: K.zeros(shape=(K.shape(X)[0], n_h)))(X)\n",
    "    c = Lambda(lambda X: K.zeros(shape=(K.shape(X)[0], n_h)))(X)\n",
    "    # Messy, but the alternative is using more Input()\n",
    "    \n",
    "    at_LSTM = LSTM(n_h, return_state=True)\n",
    "    \n",
    "    output = []\n",
    "              \n",
    "    # Run attention step and RNN for each output time step\n",
    "    for _ in range(Ty):\n",
    "        context = one_step_of_attention(h, X)\n",
    "        \n",
    "        h, _, c = at_LSTM(context, initial_state=[h, c])\n",
    "        \n",
    "        output.append(h)\n",
    "        \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample model is organized as follows:\n",
    "\n",
    "1. BiLSTM\n",
    "2. Attention Layer\n",
    "    * Outputs Ty lists of activations.\n",
    "3. Dense\n",
    "    * Necessary to convert attention layer's output to the correct y dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer3 = Dense(machine_vocab_size, activation=softmax)\n",
    "\n",
    "def get_model(Tx, Ty, layer1_size, layer2_size, x_vocab_size, y_vocab_size):\n",
    "    \"\"\"\n",
    "    Creates a model.\n",
    "    \n",
    "    input:\n",
    "    Tx - Number of x timesteps\n",
    "    Ty - Number of y timesteps\n",
    "    size_layer1 - Number of neurons in BiLSTM\n",
    "    size_layer2 - Number of neurons in attention LSTM hidden layer\n",
    "    x_vocab_size - Number of possible token types for x\n",
    "    y_vocab_size - Number of possible token types for y\n",
    "    \n",
    "    Output:\n",
    "    model - A Keras Model.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create layers one by one\n",
    "    X = Input(shape=(Tx, x_vocab_size))\n",
    "    \n",
    "    a1 = Bidirectional(LSTM(layer1_size, return_sequences=True), merge_mode='concat')(X)\n",
    "\n",
    "    a2 = attention_layer(a1, layer2_size, Ty)\n",
    "    \n",
    "    a3 = [layer3(timestep) for timestep in a2]\n",
    "        \n",
    "    # Create Keras model\n",
    "    model = Model(inputs=[X], outputs=a3)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The steps from here on out are for creating the model and training it. Simple as that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Obtain a model instance\n",
    "model = get_model(Tx, Ty, layer1_size, layer2_size, human_vocab_size, machine_vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create optimizer\n",
    "opt = Adam(lr=0.05, decay=0.04, clipnorm=1.0)\n",
    "model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group the output by timestep, not example\n",
    "outputs_train = list(Yoh_train.swapaxes(0,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\n",
      "Epoch 1/30\n",
      "8000/8000 [==============================] - 13s 2ms/step - loss: 7.1344 - dense_3_loss: 2.1335 - dense_3_acc: 0.5379 - dense_3_acc_1: 0.2478 - dense_3_acc_2: 0.9748 - dense_3_acc_3: 0.2354 - dense_3_acc_4: 0.2124\n",
      "Epoch 2/30\n",
      "8000/8000 [==============================] - 5s 606us/step - loss: 3.3504 - dense_3_loss: 0.7929 - dense_3_acc: 0.7854 - dense_3_acc_1: 0.6706 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.5629 - dense_3_acc_4: 0.7645\n",
      "Epoch 3/30\n",
      "8000/8000 [==============================] - 5s 645us/step - loss: 1.1928 - dense_3_loss: 0.1806 - dense_3_acc: 0.9091 - dense_3_acc_1: 0.9138 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.8390 - dense_3_acc_4: 0.9433\n",
      "Epoch 4/30\n",
      "8000/8000 [==============================] - 5s 617us/step - loss: 0.5314 - dense_3_loss: 0.0494 - dense_3_acc: 0.9733 - dense_3_acc_1: 0.9754 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9169 - dense_3_acc_4: 0.9936\n",
      "Epoch 5/30\n",
      "8000/8000 [==============================] - 5s 614us/step - loss: 0.2888 - dense_3_loss: 0.0249 - dense_3_acc: 0.9894 - dense_3_acc_1: 0.9863 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9693 - dense_3_acc_4: 0.9978\n",
      "Epoch 6/30\n",
      "8000/8000 [==============================] - 5s 602us/step - loss: 0.1833 - dense_3_loss: 0.0142 - dense_3_acc: 0.9914 - dense_3_acc_1: 0.9913 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9864 - dense_3_acc_4: 0.9994\n",
      "Epoch 7/30\n",
      "8000/8000 [==============================] - 5s 599us/step - loss: 0.1370 - dense_3_loss: 0.0100 - dense_3_acc: 0.9921 - dense_3_acc_1: 0.9928 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9890 - dense_3_acc_4: 0.9996\n",
      "Epoch 8/30\n",
      "8000/8000 [==============================] - 5s 602us/step - loss: 0.1070 - dense_3_loss: 0.0070 - dense_3_acc: 0.9938 - dense_3_acc_1: 0.9935 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9901 - dense_3_acc_4: 0.9999\n",
      "Epoch 9/30\n",
      "8000/8000 [==============================] - 5s 596us/step - loss: 0.0927 - dense_3_loss: 0.0062 - dense_3_acc: 0.9944 - dense_3_acc_1: 0.9938 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9908 - dense_3_acc_4: 0.9998\n",
      "Epoch 10/30\n",
      "8000/8000 [==============================] - 5s 599us/step - loss: 0.0786 - dense_3_loss: 0.0046 - dense_3_acc: 0.9950 - dense_3_acc_1: 0.9946 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9919 - dense_3_acc_4: 1.0000\n",
      "Epoch 11/30\n",
      "8000/8000 [==============================] - 5s 605us/step - loss: 0.0706 - dense_3_loss: 0.0040 - dense_3_acc: 0.9946 - dense_3_acc_1: 0.9949 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9921 - dense_3_acc_4: 1.0000\n",
      "Epoch 12/30\n",
      "8000/8000 [==============================] - 5s 602us/step - loss: 0.0640 - dense_3_loss: 0.0034 - dense_3_acc: 0.9944 - dense_3_acc_1: 0.9941 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9936 - dense_3_acc_4: 1.0000\n",
      "Epoch 13/30\n",
      "8000/8000 [==============================] - 5s 653us/step - loss: 0.0560 - dense_3_loss: 0.0032 - dense_3_acc: 0.9954 - dense_3_acc_1: 0.9951 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9943 - dense_3_acc_4: 1.0000\n",
      "Epoch 14/30\n",
      "8000/8000 [==============================] - 5s 682us/step - loss: 0.0539 - dense_3_loss: 0.0031 - dense_3_acc: 0.9953 - dense_3_acc_1: 0.9951 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9951 - dense_3_acc_4: 1.0000\n",
      "Epoch 15/30\n",
      "8000/8000 [==============================] - 5s 654us/step - loss: 0.0475 - dense_3_loss: 0.0027 - dense_3_acc: 0.9951 - dense_3_acc_1: 0.9956 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9959 - dense_3_acc_4: 1.0000\n",
      "Epoch 16/30\n",
      "8000/8000 [==============================] - 5s 635us/step - loss: 0.0444 - dense_3_loss: 0.0024 - dense_3_acc: 0.9948 - dense_3_acc_1: 0.9946 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9960 - dense_3_acc_4: 1.0000\n",
      "Epoch 17/30\n",
      "8000/8000 [==============================] - 5s 662us/step - loss: 0.0407 - dense_3_loss: 0.0023 - dense_3_acc: 0.9956 - dense_3_acc_1: 0.9955 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9970 - dense_3_acc_4: 1.0000\n",
      "Epoch 18/30\n",
      "8000/8000 [==============================] - 5s 658us/step - loss: 0.0389 - dense_3_loss: 0.0021 - dense_3_acc: 0.9956 - dense_3_acc_1: 0.9956 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9971 - dense_3_acc_4: 1.0000\n",
      "Epoch 19/30\n",
      "8000/8000 [==============================] - 5s 643us/step - loss: 0.0361 - dense_3_loss: 0.0020 - dense_3_acc: 0.9961 - dense_3_acc_1: 0.9956 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9978 - dense_3_acc_4: 1.0000\n",
      "Epoch 20/30\n",
      "8000/8000 [==============================] - 5s 654us/step - loss: 0.0337 - dense_3_loss: 0.0018 - dense_3_acc: 0.9955 - dense_3_acc_1: 0.9961 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9979 - dense_3_acc_4: 1.0000\n",
      "Epoch 21/30\n",
      "8000/8000 [==============================] - 5s 641us/step - loss: 0.0320 - dense_3_loss: 0.0017 - dense_3_acc: 0.9964 - dense_3_acc_1: 0.9966 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9980 - dense_3_acc_4: 1.0000\n",
      "Epoch 22/30\n",
      "8000/8000 [==============================] - 5s 649us/step - loss: 0.0321 - dense_3_loss: 0.0017 - dense_3_acc: 0.9961 - dense_3_acc_1: 0.9963 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9981 - dense_3_acc_4: 1.0000\n",
      "Epoch 23/30\n",
      "8000/8000 [==============================] - 5s 649us/step - loss: 0.0296 - dense_3_loss: 0.0016 - dense_3_acc: 0.9966 - dense_3_acc_1: 0.9964 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9981 - dense_3_acc_4: 1.0000\n",
      "Epoch 24/30\n",
      "8000/8000 [==============================] - 5s 636us/step - loss: 0.0290 - dense_3_loss: 0.0015 - dense_3_acc: 0.9960 - dense_3_acc_1: 0.9958 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9983 - dense_3_acc_4: 1.0000\n",
      "Epoch 25/30\n",
      "8000/8000 [==============================] - 5s 657us/step - loss: 0.0270 - dense_3_loss: 0.0015 - dense_3_acc: 0.9970 - dense_3_acc_1: 0.9966 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9989 - dense_3_acc_4: 1.0000\n",
      "Epoch 26/30\n",
      "8000/8000 [==============================] - 5s 653us/step - loss: 0.0267 - dense_3_loss: 0.0014 - dense_3_acc: 0.9963 - dense_3_acc_1: 0.9965 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9989 - dense_3_acc_4: 1.0000\n",
      "Epoch 27/30\n",
      "8000/8000 [==============================] - 5s 637us/step - loss: 0.0248 - dense_3_loss: 0.0013 - dense_3_acc: 0.9964 - dense_3_acc_1: 0.9969 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9991 - dense_3_acc_4: 1.0000\n",
      "Epoch 28/30\n",
      "8000/8000 [==============================] - 5s 655us/step - loss: 0.0248 - dense_3_loss: 0.0013 - dense_3_acc: 0.9968 - dense_3_acc_1: 0.9961 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9994 - dense_3_acc_4: 1.0000\n",
      "Epoch 29/30\n",
      "8000/8000 [==============================] - 5s 647us/step - loss: 0.0234 - dense_3_loss: 0.0012 - dense_3_acc: 0.9971 - dense_3_acc_1: 0.9973 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9994 - dense_3_acc_4: 1.0000\n",
      "Epoch 30/30\n",
      "8000/8000 [==============================] - 5s 656us/step - loss: 0.0237 - dense_3_loss: 0.0012 - dense_3_acc: 0.9971 - dense_3_acc_1: 0.9968 - dense_3_acc_2: 1.0000 - dense_3_acc_3: 0.9994 - dense_3_acc_4: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1224272b0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Time to train\n",
    "# It takes a few minutes on an quad-core CPU\n",
    "model.fit([Xoh_train], outputs_train, epochs=30, batch_size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "The final training loss should be in the range of 0.02 to 0.5\n",
    "\n",
    "The test loss should be at a similar level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 1s 669us/step\n",
      "Test loss:  0.0888983543291688\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the test performance\n",
    "outputs_test = list(Yoh_test.swapaxes(0,1))\n",
    "score = model.evaluate(Xoh_test, outputs_test) \n",
    "print('Test loss: ', score[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've created this beautiful model, let's see how it does in action.\n",
    "\n",
    "The below code finds a random example and runs it through our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: t6:27 a.m.\n",
      "Tokenized: [32  9 13  5 10  0 14  2 25  2 40 40 40 40 40 40 40 40 40 40 40 40 40 40\n",
      " 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40]\n",
      "Prediction: [0, 6, 10, 2, 7]\n",
      "Prediction text: 06:27\n"
     ]
    }
   ],
   "source": [
    "# Let's visually check model output.\n",
    "import random as random\n",
    "\n",
    "i = random.randint(0, m)\n",
    "\n",
    "def get_prediction(model, x):\n",
    "    prediction = model.predict(x)\n",
    "    max_prediction = [y.argmax() for y in prediction]\n",
    "    str_prediction = \"\".join(ids_to_keys(max_prediction, machine_vocab))\n",
    "    return (max_prediction, str_prediction)\n",
    "\n",
    "max_prediction, str_prediction = get_prediction(model, Xoh[i:i+1])\n",
    "\n",
    "print(\"Input: \" + str(dataset[i][0]))\n",
    "print(\"Tokenized: \" + str(X[i]))\n",
    "print(\"Prediction: \" + str(max_prediction))\n",
    "print(\"Prediction text: \" + str(str_prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last but not least, all introductions to Attention networks require a little tour.\n",
    "\n",
    "The below graph shows what inputs the model was focusing on when writing each individual letter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA58AAACfCAYAAAB++W3hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAG55JREFUeJzt3Xu8ZXVd//HXe2aAAcHrWMl1MMkkKi8TFy9Jikmo0O+RFiBeiuRnSRdT+WkampdfeemiSdmoiKKCRlZTToJ5wxCRwQsKqE2IMGAhyGVEgTlzPv2x1rF9juecvffMXrP3HF7Px2M9zlrf9V2f9dn7nAf6me93fVeqCkmSJEmSurRs3AlIkiRJkpY+i09JkiRJUucsPiVJkiRJnbP4lCRJkiR1zuJTkiRJktQ5i09JkiRJUucsPiXpHibJM5NcMO485kryqiTvHXceO0qStyX5o3HnIUnSjmLxKUk7QJJPJrklyW5z2s9K8to5bdckOWpE912dpJKsmGmrqvdV1S+OIn7PffZJMpXkx+c59w9J3jTK+026JPsn+W7PVknu6Dl+XFU9v6pes4Pz+mSS39yR95QkaYbFpyR1LMlq4HFAAceONZmOVNX1wMeAZ/W2J7k/cAzw7nHktaP0FvcAVXVtVe05s7XNP9vT9ukxpClJ0lhZfEpS954NfBY4C3jOTGOSU4BnAqe1o2H/nORsYH/gn9u209q+hyf5TJJbk3wpyZE9cT6Z5DVJLkqyOckFSVa1py9sf97axjsiyXOT/HvP9Y9OcmmS29qfjx4w9lzvZk7xCRwPXFlVX27jvTnJdUluT3JZksfNFyjJkUk2zWn7wYhwkmVJXprkP5PcnOSDbaFLkpVJ3tu239p+ph9d4D7XJHlZkivbkel3JVnZc/6pSb7YxvlMkp+Zc+3/S3I5cMfcArSf3lHvmc+b5LQkNyb5VpJfTnJMkq8n+U6SP+y5dujPn+R1NP8I8tb2b+Gtbf+fTPLR9h5fS/Krc3J8W3t+c5JPJTlgmM8pSdIMi09J6t6zgfe125NnCqGqWtu2vaEdDXtaVT0LuBZ4Wtv2hiT7AB8GXgvcH3gx8PdJHthzjxOBXwd+BNi17QPw8+3P+7bxLu5NrC1YPgy8BXgA8OfAh5M8YIDYc/0DsCrJY3vansXsUc9LgYe3n+P9wN/1FntD+B3gl4HHA3sDtwBntOeeA9wH2K/9TM8Hvr9IrGcCTwZ+HPgJ4BUASR4BnAn83zbO3wLrMnvq9AnAU2i+36lt+By9fgxYCewDnA68HTgJeBRN0fhHSQ5s+w79+avq5cCngVPbv4VTk9wL+CjN7+JHaP6x4K+THDzn+3kNsAr4Is3frCRJQ7P4lKQOtYXYAcAHq+oy4D9pirlhnASsr6r1VTVdVR8FNtBMZ53xrqr6elV9H/ggTYE3iKcA/1FVZ1fVVFWdA3wVeNqwsdvzf0dTbJPkIJrC6f09fd5bVTe39/ozYDfgoQPm2uv5wMuralNV3QW8Cnh6O/q4haboekhVba2qy6rq9kVivbWqrquq7wCvoykoAU4B/raqLmnjvBu4Czi859q3tNcuVtwOagvwuqraApxLU+y9uao2V9UVwJXAz4748z8VuKaq3tX+Tr4A/D3wjJ4+H66qC9v7vBw4Isl+I/i8kqR7GItPSerWc4ALquqm9vj99Ey9HdABwDPaKZS3JrkVeCzwoJ4+/9Wz/z1gTwazN/DNOW3fpBl925bY725zXUkz6nl+Vd04czLJi5Nc1U7xvZVmhG6habyLOQD4h57v4ypgK/CjwNnA+cC5SW5I8oYkuywS67qe/W/SfCcz93jRnO99v57zc6/dXjdX1dZ2f6aY/e+e89/nf7/7UX3+A4DD5nzGZ9KMws74wWesqu8C32H2dyBJ0kCGej5FkjS4JLsDvwosTzJTwO0G3DfJz1bVl2gWIZprbtt1wNlV9bxtSGO++L1uoClAeu0PfGQb7gXw7zTFyXE0I7anzZxon+88DXgicEVVTSe5Bcg8ce4A9ui5djnQO834OuA3quqiBfL4Y+CP0yz2tB74GvDOBfr2juLtT/OdzNzjdVX1ugWug/7fb1e29fPP97f1qap60iL3+sH3k2RPminTNyzcXZKk+TnyKUnd+WWa0aiDaaaqPhx4GM1zd89u+/w38OA5181tey/wtCRPTrK8XVDmyCT7DpDDt4Hpee4xYz3wE0lOTLIiya+1+f7LALF/SFUV8B7g9cB9gX/uOb0XMNXmtCLJ6cC9Fwj1dWBlkqe0o3avoCncZ7wNeN3M4jdJHpjkuHb/F5L8dFuw3k4zDXV6kbRfkGTf9vnXlwMfaNvfDjw/yWFp3KvNZ68Bv44ubevnn/u39S80v/9nJdml3X4uycN6+hyT5LFJdqV59vOzVTXKEV9J0j2Exackdec5NM9LXltV/zWzAW8Fntk+n/dO4OB2yuM/ttf9CfCKtu3F7f/RPw74Q5rC7TrgJQzw3/Cq+h7Nc4wXtfEOn3P+Zprn/l4E3EwzMvnUnmnC2+I9NCOIH2ifE5xxPs2I6tdpprfeyQLTVqvqNuC3gXcA19OMhPaufvtmYB1wQZLNNKsJH9ae+zHgPJrC6yrgUzRTURfyfuAC4GqaZ3Jf2+awAXgeze/rFmAj8Nw+n31H2dbP/2aaZ0NvSfKWqtoM/CLNQkM30Eyxfj2zC/33A6+kGdF+FM2ItiRJQ0vzj9SSJN3zJLkG+M2q+rdx5zKJkpwFbKqqV4w7F0nSzs+RT0mSJElS5yw+JUmSJEmzJDkzyY1JvrLA+SR5S5KNSS5P8si+MZ12K0mSJEnqleTnge8C76mqQ+Y5fwzwOzTvHT+M5t3Uh83t18uRT0mSJEnSLFV1Ic1icws5jqYwrar6LM2r5B60SH+LT0mSJEnS0PZh9qr1m9q2Ba3oNJ0h7bpsZe2+fHSvT1vxkJGFYstXt44u2KTbc/fRxfru90cXS5IkSerAndzB3XVXxp1HV578C/eqm78zu5657PK7rqB57dmMtVW1tss8Jqr43H35Xhxxv18ZWbz7v2t0BeO3H7t5ZLEAmJ7cYnb6kY8YWaxln/7CyGJJkiRJXbikPjbuFDp103em+MxHZg9Krtz7G3dW1ZrtCHs9sF/P8b5t24KcditJkiRJS9g0xV01NWsbgXXAs9tVbw8Hbquqby12wUSNfEqSJEmSRquALUwPdU2Sc4AjgVVJNgGvBHYBqKq3AetpVrrdCHwP+PV+MS0+JUmSJGkJK+CuGq74rKoT+pwv4AXDxLT4lCRJkqQlbLqKO6vGnUZ3z3wmOTPJjUm+0tU9JEmSJEmLK8KWmr2NQ5cLDp0FHN1hfEmSJElSHwXcWctnbePQ2bTbqrowyequ4kuSJEmS+psm3Fnjf+Jy/BlIkiRJkjpThLvHNNrZa+zFZ5JTgFMAVi7bc8zZSJIkSdLSMl3hztpl3GmMv/isqrXAWoD77PLA8S/BJEmSJElLSPOeT0c+JUmSJEkdKsKd0+Mf+ezyVSvnABcDD02yKcnJXd1LkiRJkjS/ZsGhXWdt49DlarcndBVbkiRJkjSY5j2fTruVJEmSJHVoupb4tFtJkiRJ0vjNjHz2bv0kOTrJ15JsTPLSec7vn+QTSb6Q5PIkx/SL6cinJEmSJC1hxXCvWkmyHDgDeBKwCbg0ybqqurKn2yuAD1bV3yQ5GFgPrF4srsWnJEmSJC1h0xXuGm7a7aHAxqq6GiDJucBxQG/xWcC92/37ADf0C2rxKUmSJElLWMGwCw7tA1zXc7wJOGxOn1cBFyT5HeBewFH9gk5W8fnjy1i2dnTL/n7rj/YeWaxddvnKyGIBZMXKkcWavuOOkcUCWPbpL4w0niRJkqTxWeA9n6uSbOg5XltVa4cIewJwVlX9WZIjgLOTHFJV0wtdMFnFpyRJkiRppJpptz9U+t1UVWsWuOR6YL+e433btl4nA0cDVNXFSVYCq4AbF8rD1W4lSZIkaQkrwtT08llbH5cCByU5MMmuwPHAujl9rgWeCJDkYcBK4NuLBXXkU5IkSZKWsGLekc+F+1dNJTkVOB9YDpxZVVckeTWwoarWAS8C3p7khTSPlT63qmqxuBafkiRJkrSEVcGWGm7Sa1Wtp3l9Sm/b6T37VwKPGSamxackSZIkLWFFuHuIkc+udJ5B+4LSDcD1VfXUru8nSZIkSfpfVeHurfeA4hP4PeAq/vcFpJIkSZKkHaSAqSGn3Xah0wyS7As8BXhHl/eRJEmSJM1vmnD31uWztnHoeuTzL4HTgL0W6pDkFOAUgJU/umA3SZIkSdI2qIK7+79epXOdjXwmeSpwY1Vdtli/qlpbVWuqas2u9929q3QkSZIk6R6pec/nslnbOHQ58vkY4Ngkx9C8cPTeSd5bVSd1eE9JkiRJUo8q2LKURz6r6mVVtW9VrQaOBz5u4SlJkiRJO1rYOr1s1jYOA901yQ+9PHS+NkmSJEnSZKmCqa3LZm3jMOhd/2rAtnlV1Sd9x6ckSZIk7XhF2DK9fNY2Dos+85nkCODRwAOT/EHPqXsD4580LEmSJEnqa3o6Q/VPcjTwZpq67x1V9afz9PlV4FU0rxL9UlWduFjMfgsO7Qrs2fbrfQ/K7cDTB85ckiRJkjQWM9NuB5VkOXAG8CRgE3BpknVVdWVPn4OAlwGPqapbkvxIv7iLFp9V9SngU0nOqqpvDpytJEmSJGkiFGF6uEWGDgU2VtXVAEnOBY4Druzp8zzgjKq6BaCqbuwXdNBXrZyVpOY2VtUTBrxekiRJkjQONfS0232A63qONwGHzenzEwBJLqKZmvuqqvrIYkEHLT5f3LO/EvgVYGrAawf3zWVM/9aeIwu32/RtI4tVP/ngkcUatRW33THagHfeNbJQUzfeNLJYkiRJUie2jjuBbhWw9Yen3a5KsqHneG1VrR0i7ArgIOBIYF/gwiQ/XVW3LnZB/2SrLpvTdFGSzw2RmCRJkiRpHArqh0c+b6qqNQtccT2wX8/xvm1br03AJVW1BfhGkq/TFKOXLpTGoO/5vH/PtirJk4H7DHKtJEmSJGmcQm2dvfVxKXBQkgOT7AocD6yb0+cfaUY9SbKKZhru1YsFHXTa7WU0o7WhmW77DeDkAa+VJEmSJI1LQQ2x4FBVTSU5FTif5nnOM6vqiiSvBjZU1br23C8muZJm4vJLqurmxeIOOu32wIEzlSRJkiRNlunhulfVemD9nLbTe/YL+IN2G8hAxWeSlcBvA4+lGQH9NPC2qrpz0BtJkiRJksagGGSqbecGnXb7HmAz8Fft8YnA2cAzukhKkiRJkjQ6Ge5VK50YtPg8pKoO7jn+RDu3V5IkSZI0ySowASOfgz51+vkkh88cJDkM2LBIf0mSJEnSJCia4rN3G4NBi89HAZ9Jck2Sa4CLgZ9L8uUkl/e7OMn6JHtvR56SJEmSpG2U6dnbOAw67fbo7blJVR2z0LkkpwCnAKzc5d7bcxtJkiRJ0jx2pmc+X1tVz+ptSHL23LZtUVVrgbUA99n9QbW98SRJkiRJPYqhX7XShUGLz5/qPUiygmYqriRJkiRpwmXruDPo88xnkpcl2Qz8TJLbk2xuj/8b+KdBb+Izn5IkSZI0HinI1szaxmHR4rOq/qSq9gLeWFX3rqq92u0BVfWyQW9SVcdU1Q3bna0kSZIkaWg704JD/5rk5+c2VtWFI85HkiRJkjRKNRnTbgctPl/Ss78SOBS4DHjCyDOSJEmSJI3UsKOdSY4G3gwsB95RVX+6QL9fAc4Dfq6qNiwWc6Dis6qeNucG+wF/Oci1kiRJkqQxquGKzyTLgTOAJwGbgEuTrKuqK+f02wv4PeCSQeIu+sznIjYBD9vGayVJkiRJO0hopt32bn0cCmysqqur6m7gXOC4efq9Bng9cOcgeQw08pnkr2jeDgNNwfoI4PODXCtJkiRJGqMhRz6BfYDreo43AYf1dkjySGC/qvpwkt7HNBc06DOfV9LM9QW4FTinqi4a8NqB1V13Mf0f3xhdvOnq32lAb7x6tB/3tIc+fmSxprZMjSwWANMjfBo5I17GuUb3O5UkSZLuKeYZ7VyVpPcZzbVVtXagWMky4M+B5w6Tw6LFZ5IVwP8HfgO4tm3eHzgzyeeqasswN5MkSZIk7WDzr3Z7U1WtWeCK64H9eo73bdtm7AUcAnwyzWDTjwHrkhy72KJD/Z75fCNwf+DAqnpkVT0SeDBwX+BNfa6VJEmSJE2AId/zeSlwUJIDk+wKHA+smzlZVbdV1aqqWl1Vq4HPAosWntC/+Hwq8Lyq2txzo9uB3wKO6ZuyJEmSJGmsUsMVn1U1BZwKnA9cBXywqq5I8uokx25rHv2e+ayqH37Irqq2JvHhO0mSJEnaCQz7ns+qWg+sn9N2+gJ9jxwkZr+RzyuTPHtuY5KTgK8OcgNJkiRJ0hjV0K9a6US/kc8XAB9K8hvAZW3bGmB34P8sdmGSM2mm7d5YVYdsb6KSJEmSpG2zbEwFZ69Fi8+quh44LMkTgJ9qm9dX1ccGiH0W8FbgPduVoSRJkiRp2xUw5LTbLgz0ns+q+jjw8WECV9WFSVZvQ06SJEmSpBEJsGzr+JfsGaj47FKSU4BTAFayx5izkSRJkqQlZv73fO5w/RYc6lxVra2qNVW1ZpfsNu50JEmSJGnJGfI9n50Y+8inJEmSJKk7qZ1gwSFJkiRJ0s4v0+N/5rOzabdJzgEuBh6aZFOSk7u6lyRJkiRpAQWZmr2NQ2cjn1V1QlexJUmSJEkDqslY7XbsCw5JkiRJkroThl9wKMnRSb6WZGOSl85z/g+SXJnk8iQfS3JAv5gWn5IkSZK0lFWRrbO3xSRZDpwB/BJwMHBCkoPndPsCsKaqfgY4D3hDvzQsPiVJkiRpKSuGKj6BQ4GNVXV1Vd0NnAscNytk1Seq6nvt4WeBffsFtfiUJEmSpCVuyOJzH+C6nuNNbdtCTgb+tV/QyXrVSkFNjWnppT5esvrwkcY7/4ZLRhbryXs/fGSxRq7G/2CzJEmSdE+W+RccWpVkQ8/x2qpaO3Ts5CRgDfD4fn0nq/iUJEmSJI3cPKOdN1XVmgW6Xw/s13O8b9s2O2ZyFPBy4PFVdVe/HJx2K0mSJElL2ZALDgGXAgclOTDJrsDxwLreDkkeAfwtcGxV3ThIGo58SpIkSdJSVpCpAd6vMtO9airJqcD5wHLgzKq6IsmrgQ1VtQ54I7An8HdJAK6tqmMXi2vxKUmSJElLXKaHW4ulqtYD6+e0nd6zf9SwOVh8SpIkSdISlqqhRj670ukzn0lemOSKJF9Jck6SlV3eT5IkSZI0j+np2dsYdFZ8JtkH+F1gTVUdQjNX+Piu7idJkiRJmkf7zGfvNg5dT7tdAeyeZAuwB3BDx/eTJEmSJPWqgqU87baqrgfeBFwLfAu4raou6Op+kiRJkqT5ZXp61jYOXU67vR9wHHAgsDdwryQnzdPvlCQbkmzYQt/3kkqSJEmShlEFU1tnb2PQ5YJDRwHfqKpvV9UW4EPAo+d2qqq1VbWmqtbswm4dpiNJkiRJ90AFbJ2evY1Bl898XgscnmQP4PvAE4ENHd5PkiRJkjRXFUxNjTuL7orPqrokyXnA54Ep4AvA2q7uJ0mSJEmaT8HW8Uy17dXpardV9UrglV3eQ5IkSZK0iGLpF5+SJEmSpDGroiZg2m2XCw5JkiRJksatCrZMzd76SHJ0kq8l2ZjkpfOc3y3JB9rzlyRZ3S+mxackSZIkLXG1deusbTFJlgNnAL8EHAyckOTgOd1OBm6pqocAfwG8vl8OFp+SJEmStJTNrHbbuy3uUGBjVV1dVXcD5wLHzelzHPDudv884IlJslhQi09JkiRJWsKqaqiRT2Af4Lqe401t27x9qmoKuA14wGJBJ2rBoc3cctO/1Xnf7NNtFXDTCG87yngDx1r+oFHG2zhQsMHj7fBYo45nbpMRz9zGH2vS45nb+GNNejxzG3+sSY9nbuOPNenxBo11wIjuN5E2c8v5H536wKo5zSuTbOg5XltVnb4ac6KKz6p6YL8+STZU1ZpR3XOU8SY5t1HHM7fxx5r0eOY2/liTHs/cxh9r0uOZ2/hjTXo8cxt/rEmPN+rcdlZVdfSQl1wP7NdzvG/bNl+fTUlWAPcBbl4sqNNuJUmSJEm9LgUOSnJgkl2B44F1c/qsA57T7j8d+HhV1WJBJ2rkU5IkSZI0XlU1leRU4HxgOXBmVV2R5NXAhqpaB7wTODvJRuA7NAXqonbG4nPU85BHGW+Scxt1PHMbf6xJj2du44816fHMbfyxJj2euY0/1qTHM7fxx5r0eJ0+w7iUVdV6YP2cttN79u8EnjFMzPQZGZUkSZIkabv5zKckSZIkqXM7TfGZZL8kn0hyZZIrkvzeuHOakeTMJDcm+coIY76w/ZxfSXJOkpWjii1JkiRJO9pOU3wCU8CLqupg4HDgBUkOHnNOM84Chl2+eEFJ9gF+F1hTVYfQPOTb9wFeSZIkSZpUO03xWVXfqqrPt/ubgauAfcabVaOqLqRZ4WmUVgC7t+/M2QO4YcTxt0mS1Um+muR9Sa5Kcl6SPSYkp7OSfL3N7agkFyX5jySHjjM/SZIkSTtR8dkryWrgEcAl482kG1V1PfAm4FrgW8BtVXXBeLOa5aHAX1fVw4Dbgd8ecz4ADwH+DPjJdjsReCzwYuAPx5iXJEmSJHbC4jPJnsDfA79fVbePO58uJLkfcBxwILA3cK8kJ403q1muq6qL2v330hR54/aNqvpyVU0DVwAfa19y+2Vg9VgzkyRJkrRzFZ9JdqEpPN9XVR8adz4dOoqmmPp2VW0BPgQ8esw59Zr7fp5JeF/PXT370z3H0+yc77OVJEmSlpSdpvhMEuCdwFVV9efjzqdj1wKHJ9mj/dxPpHnGdVLsn+SIdv9E4N/HmYwkSZKkybfTFJ/AY4BnAU9I8sV2O2Z7AiZZn2Tv7U0syTnAxcBDk2xKcvL2xKuqS4DzgM/TTBtdBqzd3jxH6Gs0qw1fBdwP+JvtCTaq30MXJjk3SZIkaWeS5rE4aTDtYk//0r4CRpIkSZIGsjONfEqSJEmSdlKOfEqSJEmSOufIpyRJkiSpcxafkiRJkqTOWXxKkiRJkjpn8SlJmkhJvttBzNVJThx1XEmS1J/FpyTpnmQ1YPEpSdIYWHxKkiZakiOTfDLJeUm+muR9SdKeuybJG5J8OcnnkjykbT8rydN7YsyMov4p8LgkX0zywh3/aSRJuuey+JQk7QweAfw+cDDwYOAxPeduq6qfBt4K/GWfOC8FPl1VD6+qv+gkU0mSNC+LT0nSzuBzVbWpqqaBL9JMn51xTs/PI3Z0YpIkaTAWn5KkncFdPftbgRU9xzXP/hTt/8YlWQbs2ml2kiSpL4tPSdLO7td6fl7c7l8DPKrdPxbYpd3fDOy1wzKTJEk/sKJ/F0mSJtr9klxOMzp6Qtv2duCfknwJ+AhwR9t+ObC1bT/L5z4lSdpxUlX9e0mSNIGSXAOsqaqbxp2LJElanNNuJUmSJEmdc+RTkiRJktQ5Rz4lSZIkSZ2z+JQkSZIkdc7iU5IkSZLUOYtPSZIkSVLnLD4lSZIkSZ2z+JQkSZIkde5/AO+TGxMoUnl3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1375e3240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "i = random.randint(0, m)\n",
    "\n",
    "def plot_attention_graph(model, x, Tx, Ty, human_vocab, layer=7):\n",
    "    # Process input\n",
    "    tokens = np.array([tokenize(x, human_vocab, Tx)])\n",
    "    tokens_oh = oh_2d(tokens, len(human_vocab))\n",
    "    \n",
    "    # Monitor model layer\n",
    "    layer = model.layers[layer]\n",
    "    \n",
    "    layer_over_time = K.function(model.inputs, [layer.get_output_at(t) for t in range(Ty)])\n",
    "    layer_output = layer_over_time([tokens_oh])\n",
    "    layer_output = [row.flatten().tolist() for row in layer_output]\n",
    "    \n",
    "    # Get model output\n",
    "    prediction = get_prediction(model, tokens_oh)[1]\n",
    "    \n",
    "    # Graph the data\n",
    "    fig = plt.figure()\n",
    "    fig.set_figwidth(20)\n",
    "    fig.set_figheight(1.8)\n",
    "    ax = fig.add_subplot(111)\n",
    "    \n",
    "    plt.title(\"Attention Values per Timestep\")\n",
    "    \n",
    "    plt.rc('figure')\n",
    "    cax = plt.imshow(layer_output, vmin=0, vmax=1)\n",
    "    fig.colorbar(cax)\n",
    "    \n",
    "    plt.xlabel(\"Input\")\n",
    "    ax.set_xticks(range(Tx))\n",
    "    ax.set_xticklabels(x)\n",
    "    \n",
    "    plt.ylabel(\"Output\")\n",
    "    ax.set_yticks(range(Ty))\n",
    "    ax.set_yticklabels(prediction)\n",
    "    \n",
    "    plt.show()\n",
    "    \n",
    "plot_attention_graph(model, dataset[i][0], Tx, Ty, human_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}