module-6/day-6-nn-2/intro-to-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style=\"color:brown;\">  Hands on practice with neural network</h1> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lesson plan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. How to evaluate a model? <a href=\"https://colah.github.io/posts/2015-09-Visual-Information/\">cross-entropy</a>\n",
    "2. Types of Neural Nets\n",
    "3. Keras & Tensorflow - specialized framework (python library) for machine learning\n",
    "4. Hands on image reconstruction with Keras (approximately PCA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some types of neural networks  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Classic - what we did on Friday\n",
    "- Auto-encoder (like PCA finding compressed representation) - now\n",
    "\n",
    "the hidden layer is an encoding of the generalization\n",
    "- Recurrent neural network - coming soon\n",
    "- Convlutional neural networks - coming soon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./img/RNN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./img/CNN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./img/autorec.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3. Tensorflow and Keras\n",
    "##### An Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Image Classification\n",
    "[CIFAR-10](http://www.cs.toronto.edu/~kriz/cifar.html) is a common benchmark in machine learning for image recognition. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. \n",
    "\n",
    "<img align =left src=\"./img/c10imgcat.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img align =left src=\"./img/tf-logo3.png\"><br>\n",
    "\n",
    "\n",
    "This task can be accomplished in TensorFlow and the code can be found [here](https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10/). Let's look at it together. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img align =left src=\"./img/keras.png\"><br>\n",
    "\n",
    "Now, the same task in [in Keras](https://keras.io/examples/cifar10_cnn/). After looking at it and comparing it to the tensorflow code, what do you notice? Any formatting look familiar?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Similar to another scenario we've seen:\n",
    "\n",
    "![statsmodels](./img/statsmodels_hybi_banner.png)\n",
    "\n",
    "vs\n",
    "\n",
    "![sklearn](./img/sklearn.png )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Okay, but should I use Keras or Tensorflow?\n",
    "![wrong question](https://www.pyimagesearch.com/wp-content/uploads/2018/10/keras_vs_tensorflow_wrong_question.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Keras is an API\n",
    "\n",
    "Coded in Python, that can be layered on top of many different back-end processing systems.\n",
    "\n",
    "![kerasback](./img/keras_2.png)\n",
    "\n",
    "While each of these systems has their own coding methods, Keras abstracts from that in streamlined pythonic manner we are used to seeing in other python modeling libraries.\n",
    "\n",
    "Keras development is backed primarily by Google, and the Keras API comes packaged in TensorFlow as tf.keras. Additionally, Microsoft maintains the CNTK Keras backend. Amazon AWS is maintaining the Keras fork with MXNet support. Other contributing companies include NVIDIA, Uber, and Apple (with CoreML)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Wait, what's TensorFlow?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Let's start with tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tensors are multidimensional matricies\n",
    "\n",
    "![tensor](./img/tensors.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow manages the flow of matrix math\n",
    "\n",
    "That makes neural network processing possible.\n",
    "\n",
    "![cat](./img/cat-tensors.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow at its start\n",
    "\n",
    "An open-source library\n",
    "\n",
    "![more-arch](./img/layers.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 2007 Keras was fully integrated into TensorFlow\n",
    "\n",
    "It \"comes with\" Tensorflow and provides all the medium to high end API services to integrate with tensorflow processing.\n",
    "\n",
    "![tensorflow-prog](./img/tensorflow_programming_environment.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras, an API with an intentional UX\n",
    "\n",
    "- Deliberately design end-to-end user workflows\n",
    "- Reduce cognitive load for your users\n",
    "- Provide helpful feedback to your users\n",
    "\n",
    "[full article here](https://blog.keras.io/user-experience-design-for-apis.html)<br>\n",
    "[full list of why to use Keras](https://keras.io/why-use-keras/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A few comparisons\n",
    "\n",
    "While you **can leverage both**, here are a few comparisons.\n",
    "\n",
    "| Comparison | Keras | Tensorflow|\n",
    "|------------|-------|-----------|\n",
    "| **Level of API** | high-level API | High and low-level APIs |\n",
    "| **Speed** |  can *seem* slower |  is a bit faster |\n",
    "| **Language architecture** | simple architecture, more readable and concise | straight tensorflow is a bit mroe complex |\n",
    "| **Debugging** | less frequent need to debug | difficult to debug |\n",
    "| **Datasets** | usually used for small datasets | high performance models and large datasets that require fast execution|\n",
    "\n",
    "This is also a _**non-issue**_ - as you can leverage tensorflow commands within keras and vice versa. If Keras ever seems slower, it's because the developer's time is more expensive than the GPUs. Keras is designed with the developer in mind. \n",
    "\n",
    "\n",
    "[reference link](https://www.edureka.co/blog/keras-vs-tensorflow-vs-pytorch/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More levers and buttons\n",
    "\n",
    "Coding directly in **Tensorflow** allows you to tweak more parameters to optimize performance. The **Keras** wrapper makes the code more accessible for developers prototyping models.\n",
    "\n",
    "![levers](./img/levers.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building our first Neural Network - Auto-encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install Keras & tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install tensorflow\n",
    "#!pip install keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras basic syntax "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://keras.io\">Documentation</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## initialize the model\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Input, Dense\n",
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "## prepare a placeholder for input\n",
    "# this is our input placeholder\n",
    "inp = Input(shape=(784,)) # here shape (784, ) means we specify only the number of rows but we can input as many columns as we would like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Adding layers --> simple as .add()\n",
    "## Dense means linear layer + activation\n",
    "model.add(Dense(units=64, activation='relu'))\n",
    "model.add(Dense(units=10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option I - gradient descent\n",
    "model.compile(loss='mean_squared_error',\n",
    "              optimizer='Adam',\n",
    "              metrics=['mae', 'acc'])\n",
    "\n",
    "# Option II - adadelta\n",
    "# model.compile(optimizer='adadelta', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://keras.io/optimizers/\">more optimizers</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples, validate on 100 samples\n",
      "Epoch 1/20\n",
      "1000/1000 [==============================] - 0s 140us/step - loss: 0.0907 - mae: 0.1799 - acc: 0.1060 - val_loss: 0.0912 - val_mae: 0.1806 - val_acc: 0.0600\n",
      "Epoch 2/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0902 - mae: 0.1798 - acc: 0.1070 - val_loss: 0.0909 - val_mae: 0.1805 - val_acc: 0.0500\n",
      "Epoch 3/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0901 - mae: 0.1797 - acc: 0.1180 - val_loss: 0.0909 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 4/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0900 - mae: 0.1797 - acc: 0.1200 - val_loss: 0.0909 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 5/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0899 - mae: 0.1796 - acc: 0.1250 - val_loss: 0.0909 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 6/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0899 - mae: 0.1796 - acc: 0.1310 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 7/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0898 - mae: 0.1795 - acc: 0.1320 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 8/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0898 - mae: 0.1795 - acc: 0.1350 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 9/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0897 - mae: 0.1794 - acc: 0.1420 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 10/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0897 - mae: 0.1794 - acc: 0.1400 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 11/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0897 - mae: 0.1794 - acc: 0.1440 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 12/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0896 - mae: 0.1793 - acc: 0.1460 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 13/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0896 - mae: 0.1793 - acc: 0.1430 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 14/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0895 - mae: 0.1792 - acc: 0.1470 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 15/20\n",
      "1000/1000 [==============================] - 0s 10us/step - loss: 0.0895 - mae: 0.1792 - acc: 0.1540 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 16/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0895 - mae: 0.1791 - acc: 0.1550 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0500\n",
      "Epoch 17/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0894 - mae: 0.1791 - acc: 0.1560 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0400\n",
      "Epoch 18/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0894 - mae: 0.1791 - acc: 0.1580 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0400\n",
      "Epoch 19/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0894 - mae: 0.1790 - acc: 0.1570 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0600\n",
      "Epoch 20/20\n",
      "1000/1000 [==============================] - 0s 9us/step - loss: 0.0893 - mae: 0.1790 - acc: 0.1610 - val_loss: 0.0908 - val_mae: 0.1805 - val_acc: 0.0500\n"
     ]
    }
   ],
   "source": [
    "# Training the model!\n",
    "import numpy as np\n",
    "X_train = np.random.random((1000, 20))\n",
    "y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)\n",
    "X_test = np.random.random((100, 20))\n",
    "y_test = keras.utils.to_categorical( np.random.randint(10, size=(100, 1)), num_classes=10)\n",
    "\n",
    "\n",
    "history = model.fit(X_train, y_train,\n",
    "                    batch_size=128, epochs=20,\n",
    "                    validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.09120684862136841,\n",
       " 0.09091106057167053,\n",
       " 0.09087534248828888,\n",
       " 0.09088060259819031,\n",
       " 0.0908711701631546,\n",
       " 0.09081519395112991,\n",
       " 0.09080556780099869,\n",
       " 0.0908145159482956,\n",
       " 0.09079357236623764,\n",
       " 0.09077334403991699,\n",
       " 0.09078788757324219,\n",
       " 0.09079352021217346,\n",
       " 0.09078691154718399,\n",
       " 0.09083890914916992,\n",
       " 0.09082396328449249,\n",
       " 0.09079183638095856,\n",
       " 0.09081190079450607,\n",
       " 0.09080591797828674,\n",
       " 0.09083160758018494,\n",
       " 0.0907876119017601]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Results of the model over epochs\n",
    "history.history['val_loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.09069199109077454,\n",
       " 0.09022906410694122,\n",
       " 0.09007719796895981,\n",
       " 0.09001332926750183,\n",
       " 0.08993827641010284,\n",
       " 0.08987856960296631,\n",
       " 0.08983123177289963,\n",
       " 0.0897862892150879,\n",
       " 0.08974233615398407,\n",
       " 0.08970101284980773,\n",
       " 0.089660799741745,\n",
       " 0.08962610131502151,\n",
       " 0.08958551847934723,\n",
       " 0.0895472851395607,\n",
       " 0.08951961076259612,\n",
       " 0.08946665000915527,\n",
       " 0.08944427073001862,\n",
       " 0.0893919706940651,\n",
       " 0.08937228941917419,\n",
       " 0.08931952005624771]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history['loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 64)                1344      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 1,994\n",
      "Trainable params: 1,994\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100/100 [==============================] - 0s 35us/step\n"
     ]
    }
   ],
   "source": [
    "## Evaluate\n",
    "loss_and_metrics = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.09078760385513306, 0.18045659363269806, 0.05000000074505806]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_and_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.engine.sequential.Sequential at 0x1432cc850>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Useful\n",
    "model.save('./model.h5') # Save your weights and computational graph\n",
    "from keras.models import load_model\n",
    "load_model('./model.h5') # loading the saved model from above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 160\r\n",
      "drwxr-xr-x   7 enkeboll  staff   224B Jan  6 15:46 \u001b[1m\u001b[34m.\u001b[m\u001b[m\r\n",
      "drwxr-xr-x   9 enkeboll  staff   288B Jan  6 15:01 \u001b[1m\u001b[34m..\u001b[m\u001b[m\r\n",
      "-rw-r--r--@  1 enkeboll  staff   6.0K Jan  6 15:01 .DS_Store\r\n",
      "drwxr-xr-x   3 enkeboll  staff    96B Jan  6 15:26 \u001b[1m\u001b[34m.ipynb_checkpoints\u001b[m\u001b[m\r\n",
      "drwxr-xr-x  16 enkeboll  staff   512B Oct  7 14:39 \u001b[1m\u001b[34mimg\u001b[m\u001b[m\r\n",
      "-rwxr-xr-x   1 enkeboll  staff    25K Jan  6 15:46 \u001b[35mintro-to-keras.ipynb\u001b[m\u001b[m\r\n",
      "-rw-r--r--   1 enkeboll  staff    44K Jan  6 15:45 model.h5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lah"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.applications.vgg16 import VGG16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5\n",
      "58892288/58889256 [==============================] - 8s 0us/step\n"
     ]
    }
   ],
   "source": [
    "model = VGG16(weights='imagenet', include_top=False)\n",
    "model.save('./vgg16.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"vgg16\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, None, None, 3)     0         \n",
      "_________________________________________________________________\n",
      "block1_conv1 (Conv2D)        (None, None, None, 64)    1792      \n",
      "_________________________________________________________________\n",
      "block1_conv2 (Conv2D)        (None, None, None, 64)    36928     \n",
      "_________________________________________________________________\n",
      "block1_pool (MaxPooling2D)   (None, None, None, 64)    0         \n",
      "_________________________________________________________________\n",
      "block2_conv1 (Conv2D)        (None, None, None, 128)   73856     \n",
      "_________________________________________________________________\n",
      "block2_conv2 (Conv2D)        (None, None, None, 128)   147584    \n",
      "_________________________________________________________________\n",
      "block2_pool (MaxPooling2D)   (None, None, None, 128)   0         \n",
      "_________________________________________________________________\n",
      "block3_conv1 (Conv2D)        (None, None, None, 256)   295168    \n",
      "_________________________________________________________________\n",
      "block3_conv2 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_conv3 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_pool (MaxPooling2D)   (None, None, None, 256)   0         \n",
      "_________________________________________________________________\n",
      "block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   \n",
      "_________________________________________________________________\n",
      "block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "_________________________________________________________________\n",
      "block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "=================================================================\n",
      "Total params: 14,714,688\n",
      "Trainable params: 14,714,688\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 115248\r\n",
      "drwxr-xr-x   8 enkeboll  staff   256B Jan  6 15:50 \u001b[1m\u001b[34m.\u001b[m\u001b[m/\r\n",
      "-rwxr-xr-x   1 enkeboll  staff    26K Jan  6 15:50 \u001b[35mintro-to-keras.ipynb\u001b[m\u001b[m*\r\n",
      "-rw-r--r--   1 enkeboll  staff    56M Jan  6 15:50 vgg16.h5\r\n",
      "-rw-r--r--   1 enkeboll  staff    44K Jan  6 15:45 model.h5\r\n",
      "drwxr-xr-x   3 enkeboll  staff    96B Jan  6 15:26 \u001b[1m\u001b[34m.ipynb_checkpoints\u001b[m\u001b[m/\r\n",
      "-rw-r--r--@  1 enkeboll  staff   6.0K Jan  6 15:01 .DS_Store\r\n",
      "drwxr-xr-x   9 enkeboll  staff   288B Jan  6 15:01 \u001b[1m\u001b[34m..\u001b[m\u001b[m/\r\n",
      "drwxr-xr-x  16 enkeboll  staff   512B Oct  7 14:39 \u001b[1m\u001b[34mimg\u001b[m\u001b[m/\r\n"
     ]
    }
   ],
   "source": [
    "ls -lath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your exercise:  Image reconstruction using auto-encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Import dataset - Mnist (handwritten digits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keras imports for the dataset and building our neural network\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential, load_model, Input\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.layers import Conv2D, BatchNormalization\n",
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### a. Load the Mnist dataset and split to train/test (use mnist.load_data())\n",
    "### b. Plot few images with their label\n",
    "### c. reshape images to 784 dim vector and normalize (divide by 255)\n",
    "### d. Create a new model with 1 Dense layers sized 32 (encoder) with relu layer. \n",
    "### Then add a dense layer sized 784 with sigmoid activation.\n",
    "### Run it for 50 epochs with batches of 256. Use binary cross-entropy. \n",
    "### e. Plot the model loss\n",
    "### f. Use predict to encode and decode x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keras imports for the dataset and building our neural network\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import (\n",
    "    Dense,\n",
    "    Dropout,\n",
    "    Flatten,\n",
    "    Conv2D,\n",
    "    MaxPooling2D\n",
    ")\n",
    "from keras import backend as K\n",
    "\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL']='3'\n",
    "## This is to avoid getting hundreds of log messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 256\n",
    "num_classes = 10\n",
    "epochs = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   3,\n",
       "         18,  18,  18, 126, 136, 175,  26, 166, 255, 247, 127,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,  30,  36,  94, 154, 170,\n",
       "        253, 253, 253, 253, 253, 225, 172, 253, 242, 195,  64,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,  49, 238, 253, 253, 253, 253,\n",
       "        253, 253, 253, 253, 251,  93,  82,  82,  56,  39,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,  18, 219, 253, 253, 253, 253,\n",
       "        253, 198, 182, 247, 241,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,  80, 156, 107, 253, 253,\n",
       "        205,  11,   0,  43, 154,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,  14,   1, 154, 253,\n",
       "         90,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 139, 253,\n",
       "        190,   2,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  11, 190,\n",
       "        253,  70,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  35,\n",
       "        241, 225, 160, 108,   1,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         81, 240, 253, 253, 119,  25,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,  45, 186, 253, 253, 150,  27,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,  16,  93, 252, 253, 187,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0, 249, 253, 249,  64,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,  46, 130, 183, 253, 253, 207,   2,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  39,\n",
       "        148, 229, 253, 253, 253, 250, 182,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  24, 114, 221,\n",
       "        253, 253, 253, 253, 201,  78,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,  23,  66, 213, 253, 253,\n",
       "        253, 253, 198,  81,   2,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,  18, 171, 219, 253, 253, 253, 253,\n",
       "        195,  80,   9,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,  55, 172, 226, 253, 253, 253, 253, 244, 133,\n",
       "         11,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0, 136, 253, 253, 253, 212, 135, 132,  16,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0]], dtype=uint8)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(64, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer='Adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_5 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 64)                589888    \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 609,354\n",
      "Trainable params: 609,354\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 47s 783us/step - loss: 0.3887 - accuracy: 0.8803 - val_loss: 0.0667 - val_accuracy: 0.9805\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 47s 789us/step - loss: 0.1414 - accuracy: 0.9591 - val_loss: 0.0544 - val_accuracy: 0.9817\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 47s 778us/step - loss: 0.1079 - accuracy: 0.9688 - val_loss: 0.0413 - val_accuracy: 0.9860\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 46s 773us/step - loss: 0.0924 - accuracy: 0.9732 - val_loss: 0.0375 - val_accuracy: 0.9872\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 47s 776us/step - loss: 0.0793 - accuracy: 0.9766 - val_loss: 0.0356 - val_accuracy: 0.9885\n",
      "Test loss: 0.035628745761197934\n",
      "Test accuracy: 0.9884999990463257\n"
     ]
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}