Building and serialising Keras models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras exercise\n",
    "\n",
    "In this exercise you will be creating a Keras model by loading a data set, preprocessing input data, building a Sequential Keras model and compiling the model with a training configuration. Afterwards, you train your model on the training data and evaluate it on the test set. To finish this exercise, you will past the accuracy of your model to the Coursera grader.\n",
    "\n",
    "This notebook is tested in IBM Watson Studio under python 3.6\n",
    "\n",
    "## Data\n",
    "\n",
    "For this exercise we will use the Reuters newswire dataset. This dataset consists of 11,228 newswires from the Reuters news agency. Each wire is encoded as a sequence of word indexes, just as in the IMDB data we encountered in lecture 5 of this series. Moreover, each wire is categorised into one of 46 topics, which will serve as our label. This dataset is available through the Keras API.\n",
    "\n",
    "## Goal\n",
    "\n",
    "We want to create a Multi-layer perceptron (MLP) using Keras which we can train to classify news items into the specified 46 topics.\n",
    "\n",
    "## Instructions\n",
    "\n",
    "We start by installing and importing everything we need for this exercise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow==2.2.0rc0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (2.2.0rc0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (0.7.0)\n",
      "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.4.1)\n",
      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (0.32.3)\n",
      "Requirement already satisfied: google-pasta>=0.1.8 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (0.2.0)\n",
      "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (2.10.0)\n",
      "Requirement already satisfied: gast==0.3.3 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (0.3.3)\n",
      "Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (3.11.2)\n",
      "Requirement already satisfied: six>=1.12.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.12.0)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.11.1)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.1.2)\n",
      "Requirement already satisfied: astunparse==1.6.3 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.6.3)\n",
      "Requirement already satisfied: numpy<2.0,>=1.16.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.19.4)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (3.3.0)\n",
      "Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (2.1.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.1.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (1.16.1)\n",
      "Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow==2.2.0rc0) (2.1.1)\n",
      "Requirement already satisfied: setuptools in /opt/conda/envs/Python36/lib/python3.6/site-packages (from protobuf>=3.8.0->tensorflow==2.2.0rc0) (40.8.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (0.4.2)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (0.14.1)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (1.23.0)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (2.21.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (3.0.1)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (1.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.5\" in /opt/conda/envs/Python36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (4.6)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (4.1.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (0.2.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (2020.6.20)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (2.8)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (1.24.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (3.1.0)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from rsa<5,>=3.1.4; python_version >= \"3.5\"->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.2.0rc0) (0.4.8)\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow==2.2.0rc0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "if not tf.__version__ == '2.2.0-rc0':\n",
    "    print(tf.__version__)\n",
    "    raise ValueError('please upgrade to TensorFlow 2.2.0-rc0, or restart your Kernel (Kernel->Restart & Clear Output)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT! => Please restart the kernel by clicking on \"Kernel\"->\"Restart and Clear Outout\" and wait until all output disapears. Then your changes are beeing picked up\n",
    "\n",
    "As you can see, we use Keras' Sequential model with only two types of layers: Dense and Dropout. We also specify a random seed to make our results reproducible. Next, we load the Reuters data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz\n",
      "2113536/2110848 [==============================] - 0s 0us/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/Python36/lib/python3.6/site-packages/tensorflow/python/keras/datasets/reuters.py:148: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "/opt/conda/envs/Python36/lib/python3.6/site-packages/tensorflow/python/keras/datasets/reuters.py:149: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "seed = 1337\n",
    "np.random.seed(seed)\n",
    "from tensorflow.keras.datasets import reuters\n",
    "\n",
    "max_words = 1000\n",
    "(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words,\n",
    "                                                         test_split=0.2,\n",
    "                                                         seed=seed)\n",
    "num_classes = np.max(y_train) + 1  # 46 topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we cap the maximum number of words in a news item to 1000 by specifying the *num_words* key word. Also, 20% of the data will be test data and we ensure reproducibility by setting our random seed.\n",
    "\n",
    "Our training features are still simply sequences of indexes and we need to further preprocess them, so that we can plug them into a *Dense* layer. For this we use a *Tokenizer* from Keras' text preprocessing module. This tokenizer will take an index sequence and map it to a vector of length *max_words=1000*. Each of the 1000 vector positions corresponds to one of the words in our newswire corpus. The output of the tokenizer has a 1 at the i-th position of the vector, if the word corresponding to i is in the description of the newswire, and 0 otherwise. Even if this word appears multiple times, we still just put a 1 into our vector, i.e. our tokenizer is binary. We use this tokenizer to transform both train and test features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')\n",
    "x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exercise part: label encoding\n",
    "\n",
    "Use to_categorical, as we did in the lectures, to transform both *y_train* and *y_test* into one-hot encoded vectors of length *num_classes*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(y_train, num_classes)\n",
    "y_test = to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exercise part: model definition\n",
    "\n",
    "Next, initialise a Keras *Sequential* model and add three layers to it:\n",
    "\n",
    "    Layer: Add a *Dense* layer with in input_shape=(max_words,), 512 output units and \"relu\" activation.\n",
    "    Layer: Add a *Dropout* layer with dropout rate of 50%.\n",
    "    Layer: Add a *Dense* layer with num_classes output units and \"softmax\" activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-7-ddf092c331c8>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-7-ddf092c331c8>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    Layer: Add a *Dense* layer with in input_shape=(max_words,), 512 output units and \"relu\" activation.\u001b[0m\n\u001b[0m               ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "Layer: Add a *Dense* layer with in input_shape=(max_words,), 512 output units and \"relu\" activation.\n",
    "Layer: Add a *Dropout* layer with dropout rate of 50%.\n",
    "Layer: Add a *Dense* layer with num_classes output units and \"softmax\" activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()  # Instantiate sequential model\n",
    "model.add(Dense(512, activation='relu', input_shape=(max_words,))) # Add first layer\n",
    "model.add(Dropout(0.5)) # Add second layer\n",
    "model.add(Dense(num_classes, activation='softmax')) # Add third layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Exercise part: model compilation\n",
    "\n",
    "As the next step, we need to compile our Keras model with a training configuration. Compile your model with \"categorical_crossentropy\" as loss function, \"adam\" as optimizer and specify \"accuracy\" as evaluation metric. NOTE: In case you get an error regarding h5py, just restart the kernel and start from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Exercise part: model training and evaluation\n",
    "\n",
    "Next, define the batch_size for training as 32 and train the model for 5 epochs on *x_train* and *y_train* by using the *fit* method of your model. Then calculate the score for your trained model by running *evaluate* on *x_test* and *y_test* with the same batch size as used in *fit*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "281/281 [==============================] - 4s 13ms/step - loss: 1.3926 - accuracy: 0.6843\n",
      "71/71 [==============================] - 0s 2ms/step - loss: 0.9746 - accuracy: 0.7747\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "model.fit(x_train, y_train, batch_size=batch_size)\n",
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have done everything as specified, in particular set the random seed as we did above, your test accuracy should be around 80% "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7747105956077576"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, now it's time to submit your result to the Coursera grader by executing the following cells (Programming Assingment, Week2). \n",
    "\n",
    "We have to install a little library in order to submit to coursera\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-11-20 15:38:36--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\r\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\r\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\r\n",
      "HTTP request sent, awaiting response... 200 OK\r\n",
      "Length: 2540 (2.5K) [text/plain]\r\n",
      "Saving to: ‘rklib.py’\r\n",
      "\r\n",
      "100%[======================================>] 2,540       --.-K/s   in 0s      \r\n",
      "\r\n",
      "2020-11-20 15:38:36 (20.5 MB/s) - ‘rklib.py’ saved [2540/2540]\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please provide your email address and obtain a submission token (secret) on the grader’s submission page in coursera, then execute the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"ozVf2\",\"id\":\"tE4j0qhMEeecqgpT6QjMdA~ozVf2~jOHfiytGEeuCNxIlakc3BQ\",\"courseId\":\"tE4j0qhMEeecqgpT6QjMdA\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "import json\n",
    "\n",
    "key = \"XbAMqtjdEeepUgo7OOVwng\"\n",
    "part = \"HCvcp\"\n",
    "email = \"turki32235@gmail.com\"\n",
    "token = \"eyKkd78f9udOCuVd\" #you can obtain it from the grader page on Coursera (have a look here if you need more information on how to obtain the token https://youtu.be/GcDo0Rwe06U?t=276)\n",
    "\n",
    "\n",
    "submit(email, token, 'XbAMqtjdEeepUgo7OOVwng', part, [part], json.dumps(score[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}