Eager_Execution_Gradient_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Eager Execution - Gradient .ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/Eager_Execution_Gradient_.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "KSrDBc-DNg8x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Enable Eager Execution "
      ]
    },
    {
      "metadata": {
        "id": "dHHHtRg0NdGx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "960741ef-ef3e-499e-98a1-9c0928f34743"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "tf.enable_eager_execution()\n",
        "\n",
        "print('Is it enabled ? ', tf.executing_eagerly())"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Is it enabled ?  True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y7CmID5qNzAY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we know that TenosrFlow works with static graphs. So, first you have to create the graph then execute it later. This makes debugging a bit complicated. With Eager Execution you can now evalute operations directly without creating a session."
      ]
    },
    {
      "metadata": {
        "id": "zuVssr1XNt0z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "70d0eb8c-70c8-4c42-96f5-b0625a5d122a"
      },
      "cell_type": "code",
      "source": [
        "x = 2\n",
        "m = tf.square(x)\n",
        "print(\"x^2 = {}\".format(m))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "x^2 = 4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QhV6mfuJOdgb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Cool, isn't it ? Now, let us look at a very important function called [tf.GradientTape ](https://https://www.tensorflow.org/api_docs/python/tf/GradientTape). This function allows you to record automatic differentiation operations. So If you evaluted a loss function $L$ included inside the scope of that function, you can evaluate the gradient with respect to the input. Seems complicated ? Let us see an example.  Suppose, we want to find the derivative of a simple function $f(x) = x^2$ at $x = 2$. Then do the following "
      ]
    },
    {
      "metadata": {
        "id": "0CPedGTyPcKy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cbaf0557-00e3-45c5-8327-27dc2f8c8720"
      },
      "cell_type": "code",
      "source": [
        "x = tf.Variable([[2.0]])\n",
        "with tf.GradientTape() as tape:\n",
        "  loss = x * x\n",
        "\n",
        "grad = tape.gradient(loss, x)\n",
        "print(grad)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor([[4.]], shape=(1, 1), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lOrG97j3Pos8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As you can see the derivative is evaluted correctly as $f'(x = 2) = 2 \\times 2 = 4$. This function is very important as it allows you to closely watch the gradient of the graph you are working on. Moreoever, you can calcuate the gradient of an arbitrary loss function and update the parameters accordingly. Let us take a look at a complicated example. "
      ]
    },
    {
      "metadata": {
        "id": "SiQOSWTWQHaG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluate the Gradient of a CNN\n",
        "In this example we will create a loss function on a CNN model. Then we will evaluate the gradient and update the parameters of the model. We will train the model on mnist dataset."
      ]
    },
    {
      "metadata": {
        "id": "djhdKtR4QfKY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load Data"
      ]
    },
    {
      "metadata": {
        "id": "qtNUKwbGQjUG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "9efd36a8-d2f4-4173-cd32-98d7d13f82f0"
      },
      "cell_type": "code",
      "source": [
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 1s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Oq-tfTzPQslc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Preprocess the Dataset"
      ]
    },
    {
      "metadata": {
        "id": "cQDTxumRQxVW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "\n",
        "x_train = tf.expand_dims(np.float32(x_train)/ 255., 3)\n",
        "x_test  = tf.expand_dims(np.float32(x_test )/ 255., 3)\n",
        "\n",
        "y_train = tf.one_hot(y_train, 10)\n",
        "y_test  = tf.one_hot(y_test , 10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "exd7YnKBQ4-r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Look at the Data\n",
        "\n",
        "Now since we use Eager Execution we can easily look at the data by calling `.numpy()` on the tenosr. This will return the numpy array "
      ]
    },
    {
      "metadata": {
        "id": "Nk7O5A2WRLST",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "e339c356-4b57-4116-ab05-22e105b33d55"
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "img = x_train[0].numpy().squeeze()\n",
        "lbl = np.argmax(y_train[0])\n",
        "print('Label :', lbl)\n",
        "plt.imshow(img)\n",
        "plt.show()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Label : 5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAFKCAYAAACU6307AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEyJJREFUeJzt3X1MlfX/x/HXiRPCGTgEOWxu3c2p\nsdQ5GxaaJjezdGt5UxkMXcstrUneZI5R0o2bKGFLpE2htCZrnUW2anOD7GYzhzhZo0ErzC1HZohF\n5g0anPj98dv3TBTlzeEcrgM9H391PufN57yvrnrtc53rXNfl6unp6REA4KZucboBABgOCEsAMCAs\nAcCAsAQAA8ISAAwISwAwICwBwICwBAADd7B/uGXLFjU2NsrlcqmwsFBTp04NZV8AEFGCCsujR4/q\n5MmT8vl8OnHihAoLC+Xz+ULdGwBEjKAOw+vq6pSdnS1JGj9+vM6dO6cLFy6EtDEAiCRBheXZs2c1\nZsyYwOvExES1t7eHrCkAiDQhOcHDvTgAjHRBhaXX69XZs2cDr8+cOaPk5OSQNQUAkSaosJw1a5Zq\namokSc3NzfJ6vYqLiwtpYwAQSYI6Gz59+nTdc889evLJJ+VyufTKK6+Eui8AiCgubv4LAP3jCh4A\nMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCA\nsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8IS\nAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAw\nICwBwMDtdAMY+f79919z7ZUrV8LYSW+xsbHq7OzsNfb++++b/vbixYvmz/nhhx/MtW+99Za5trCw\n8LqxnTt3Kj8/v9dYeXm5ec7Y2Fhz7fbt2011zz77rHnOSMbKEgAMglpZ1tfXa82aNZowYYIkaeLE\nidq0aVNIGwOASBL0YfiMGTNUVlYWyl4AIGJxGA4ABkGH5c8//6xVq1YpJydHhw8fDmVPABBxXD09\nPT0D/aO2tjY1NDRo/vz5am1t1fLly1VbW6vo6Ohw9AgAjgvqO8uUlBQtWLBAknT77bdr7Nixamtr\n02233RbS5jAy8NMhfjo0EgR1GP7ZZ5/p3XfflSS1t7frjz/+UEpKSkgbA4BIEtTKMjMzUxs2bNCX\nX36prq4uvfrqqxyCAxjRggrLuLg47dq1K9S9AEDECuoED5x37tw5c63f7zfXNjY29jmekZGhr7/+\nOvC6trbWPOdff/1lrq2oqDDXDpbf71dUVFTYP+fOO+8012ZlZZlr//dV2NX62qb4+HjznLNnzzbX\nlpaWmuomTZpknjOS8TtLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwIDL\nHSPMr7/+aqqbNm2aec6Ojo5g2wkYqksDh9JgtumWW+zrjC+++MJcO5BbpPXlvvvuU319fa8xr9dr\n/vu4uDhzbXJysrl2JGBlCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABkE93RHh\nk5SUZKobyHPaQ3EFT6SZN2+eufZm/05zcnJ6vd6/f79pzlGjRpk/f+7cuebaULjvvvuG9PP+K1hZ\nAoABYQkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAZc7hhhrA+seu+998xzVldX\nm2vT09Nv+N7HH38c+OclS5aY5xyIBx54wFT36aefmueMjo6+4XtVVVW9Xv/++++mOXfs2GH+fIwM\nrCwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA1dPT0+P000gvK5cuWKu\nvdGlgS6XS1f/p1JYWGies6SkxFz79ddfm+rmzJljnhMIBdPKsqWlRdnZ2YHraE+fPq1ly5YpNzdX\na9as0T///BPWJgHAaf2G5aVLl7R58+ZeN1goKytTbm6uPvjgA91xxx0DulEDAAxH/YZldHS0Kisr\n5fV6A2P19fXKysqSJGVkZKiuri58HQJABOj3Fm1ut1tud++yzs7OwHdbSUlJam9vD093ABAhBn0/\nS84PRb5Ro0aFZB6XyxX45+LiYvPfDaQWiFRBhaXH49Hly5cVExOjtra2XofoiDycDQcGL6jfWc6c\nOVM1NTWSpNraWs2ePTukTQFApOl3ZdnU1KRt27bp1KlTcrvdqqmpUWlpqQoKCuTz+TRu3DgtXLhw\nKHoFAMf0G5aTJ0/Wvn37rhvfu3dvWBoCgEjEA8v+A8JxgmfMmDEhmfNaZWVlprqBfPVzdd9AsLg2\nHAAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADDggWUIykCeu5Sbm2uu/eST\nT0x1jY2N5jknT55srgVuhJUlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYA\nYMDljgi7P//801w7fvx4U11iYqJ5zhs913779u164YUXeo3NmjXLNOeiRYvMn8/TJUcGVpYAYEBY\nAoABYQkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGDAFTyIKEePHjXVPfzww+Y5z5071+e43+9X\nVFSUeZ6r7dmzx1y7ZMkSc21cXFww7WAIsLIEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwB\nwICwBAADwhIADNxONwBcbcaMGaa65uZm85zr1q274XuPP/54r9cfffSRac6nn37a/PknTpww1774\n4ovm2vj4eHMtBo+VJQAYmMKypaVF2dnZqqqqkiQVFBTokUce0bJly7Rs2TJ988034ewRABzX72H4\npUuXtHnzZqWnp/caX79+vTIyMsLWGABEkn5XltHR0aqsrJTX6x2KfgAgIpnvZ7lz506NGTNGeXl5\nKigoUHt7u7q6upSUlKRNmzYpMTEx3L0CgGOCOhv+6KOPKiEhQampqaqoqFB5ebmKiopC3RtwQ6dP\nnzbX3uhs+Icffqgnn3yy15j1bPhAvPTSS+ZazoZHrqDOhqenpys1NVWSlJmZqZaWlpA2BQCRJqiw\nzM/PV2trqySpvr5eEyZMCGlTABBp+j0Mb2pq0rZt23Tq1Cm53W7V1NQoLy9Pa9euVWxsrDwej4qL\ni4eiVwBwTL9hOXnyZO3bt++68YceeigsDQFAJOLpjhjxLl++3Od4TEzMde8dOXLENGd2drb58wfy\nv9hjjz1mrvX5fOZaDB6XOwKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQlgBgQFgCgAFhCQAG\nXO4IBGHUqFHm2u7ubnOt222/xez3339/3dikSZP0008/XTeGwWNlCQAGhCUAGBCWAGBAWAKAAWEJ\nAAaEJQAYEJYAYEBYAoABYQkABvbLBYAI8ttvv5lr9+/f3+f46tWrVV5e3musrq7ONOdArsoZiLS0\nNHPtxIkTBzSOwWFlCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABjwwDKE\nXXt7u7n27bffNtXt3bvXPOevv/7a57jf71dUVJR5nmAN5DOeeOIJc21VVVUw7SBIrCwBwICwBAAD\nwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA57uiF4uXLjQ53hcXFyv9z7//HPznK+/\n/rq5tqWlxVzrpMzMTHPt1q1bzbX33ntvMO1gCJjCsqSkRA0NDeru7tbKlSs1ZcoUbdy4UX6/X8nJ\nyXrjjTcUHR0d7l4BwDH9huWRI0d0/Phx+Xw+dXR0aNGiRUpPT1dubq7mz5+vN998U9XV1crNzR2K\nfgHAEf1+Z5mWlqYdO3ZIkkaPHq3Ozk7V19crKytLkpSRkWF+MD0ADFf9hmVUVJQ8Ho8kqbq6WnPm\nzFFnZ2fgsDspKWlAt+ACgOHIfILn4MGDqq6u1p49ezRv3rzAOLfDHFni4uJM7+Xk5JjnHEjtUPP7\n/U63gGHCFJaHDh3Srl279M477yg+Pl4ej0eXL19WTEyM2tra5PV6w90nhsh/6Wz4YG7+y9nw/55+\nD8PPnz+vkpIS7d69WwkJCZKkmTNnqqamRpJUW1ur2bNnh7dLAHBYvyvLAwcOqKOjQ2vXrg2Mbd26\nVS+//LJ8Pp/GjRunhQsXhrVJAHBav2G5dOlSLV269LrxgTwDBQCGO67gGaYuXrxorm1tbTXX5uXl\n9Tl+7NgxzZ07N/D6u+++M8/ptKtPSPb33muvvWaaMy0tzfz5LpfLXIvIxbXhAGBAWAKAAWEJAAaE\nJQAYEJYAYEBYAoABYQkABoQlABgQlgBgQFgCgIGrhxtShl1nZ6e59uobltzMt99+a57zxx9/NNfe\nyGBuZzYQCxYsMNUVFRWZ55w2bVqf47feequ6urquGwP6wsoSAAwISwAwICwBwICwBAADwhIADAhL\nADAgLAHAgLAEAAPCEgAMCEsAMODpjtf45ZdfTHVbtmzpc7yiokLPPPNMr7GDBw+aP//kyZPmWid5\nPB5z7ebNm821zz33nKkuOjraPOfNcHkjrFhZAoABYQkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCW\nAGBAWAKAAQ8su8b27dtNdRs3buxzfKge7DV9+nRzbU5OjrnW7e77oq7nn39eZWVlgdfXXqV0MzEx\nMeZaIFKxsgQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMuNwRAAxMT3cs\nKSlRQ0ODuru7tXLlSn311Vdqbm5WQkKCJGnFihWaO3duOPsEAEf1G5ZHjhzR8ePH5fP51NHRoUWL\nFun+++/X+vXrlZGRMRQ9AoDj+g3LtLQ0TZ06VZI0evRodXZ2yu/3h70xAIgkA/rO0ufz6dixY4qK\nilJ7e7u6urqUlJSkTZs2KTExMZx9AoCjzGF58OBB7d69W3v27FFTU5MSEhKUmpqqiooK/f777yoq\nKgp3rwDgGNNPhw4dOqRdu3apsrJS8fHxSk9PV2pqqiQpMzNTLS0tYW0SAJzWb1ieP39eJSUl2r17\nd+Dsd35+vlpbWyVJ9fX1mjBhQni7BACH9XuC58CBA+ro6NDatWsDY4sXL9batWsVGxsrj8ej4uLi\nsDYJAE7jR+kAYMDljgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkA\nBoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQ\nlgBgQFgCgAFhCQAGhCUAGBCWAGDgduJDt2zZosbGRrlcLhUWFmrq1KlOtBFS9fX1WrNmjSZMmCBJ\nmjhxojZt2uRwV8FraWnRc889p6eeekp5eXk6ffq0Nm7cKL/fr+TkZL3xxhuKjo52us0BuXabCgoK\n1NzcrISEBEnSihUrNHfuXGebHKCSkhI1NDSou7tbK1eu1JQpU4b9fpKu366vvvrK8X015GF59OhR\nnTx5Uj6fTydOnFBhYaF8Pt9QtxEWM2bMUFlZmdNtDNqlS5e0efNmpaenB8bKysqUm5ur+fPn6803\n31R1dbVyc3Md7HJg+tomSVq/fr0yMjIc6mpwjhw5ouPHj8vn86mjo0OLFi1Senr6sN5PUt/bdf/9\n9zu+r4b8MLyurk7Z2dmSpPHjx+vcuXO6cOHCULeBm4iOjlZlZaW8Xm9grL6+XllZWZKkjIwM1dXV\nOdVeUPrapuEuLS1NO3bskCSNHj1anZ2dw34/SX1vl9/vd7grB8Ly7NmzGjNmTOB1YmKi2tvbh7qN\nsPj555+1atUq5eTk6PDhw063EzS3262YmJheY52dnYHDuaSkpGG3z/raJkmqqqrS8uXLtW7dOv35\n558OdBa8qKgoeTweSVJ1dbXmzJkz7PeT1Pd2RUVFOb6vHPnO8mo9PT1OtxASd955p1avXq358+er\ntbVVy5cvV21t7bD8vqg/I2WfPfroo0pISFBqaqoqKipUXl6uoqIip9sasIMHD6q6ulp79uzRvHnz\nAuPDfT9dvV1NTU2O76shX1l6vV6dPXs28PrMmTNKTk4e6jZCLiUlRQsWLJDL5dLtt9+usWPHqq2t\nzem2Qsbj8ejy5cuSpLa2thFxOJuenq7U1FRJUmZmplpaWhzuaOAOHTqkXbt2qbKyUvHx8SNmP127\nXZGwr4Y8LGfNmqWamhpJUnNzs7xer+Li4oa6jZD77LPP9O6770qS2tvb9ccffyglJcXhrkJn5syZ\ngf1WW1ur2bNnO9zR4OXn56u1tVXS/38n+79fMgwX58+fV0lJiXbv3h04SzwS9lNf2xUJ+8rV48Ba\nvbS0VMeOHZPL5dIrr7yiu+++e6hbCLkLFy5ow4YN+vvvv9XV1aXVq1frwQcfdLqtoDQ1NWnbtm06\ndeqU3G63UlJSVFpaqoKCAl25ckXjxo1TcXGxbr31VqdbNetrm/Ly8lRRUaHY2Fh5PB4VFxcrKSnJ\n6VbNfD6fdu7cqbvuuiswtnXrVr388svDdj9JfW/X4sWLVVVV5ei+ciQsAWC44QoeADAgLAHAgLAE\nAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAz+D4GsMlewG9H3AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f647b314be0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "6g0CKyv3SFxp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Get a Batch of the Data"
      ]
    },
    {
      "metadata": {
        "id": "L0dt5XIqSK_p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "def get_batch(batch_size = 32):\n",
        "  #get a random index to extract a batch \n",
        "  r = np.random.randint(0, 60000-batch_size)\n",
        "  return x_train[r: r + batch_size], y_train[r: r + batch_size]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1Ao4i46DR2M9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Simple Model"
      ]
    },
    {
      "metadata": {
        "id": "oUcac6HAR5-S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 486
        },
        "outputId": "2512bacb-ce78-42dd-ca56-0650ae2d7886"
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.layers import Dense, Convolution2D, MaxPooling2D, Flatten, BatchNormalization, Dropout\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "def create_model():\n",
        "  model = Sequential()\n",
        "  model.add(Convolution2D(filters = 16, kernel_size = 3, padding = 'same', input_shape = [28, 28, 1], activation = 'relu'))\n",
        "  model.add(MaxPooling2D(pool_size = (2,2)))\n",
        "  model.add(BatchNormalization())\n",
        "  model.add(Convolution2D(filters = 32, kernel_size = 3, padding = 'same', activation = 'relu'))\n",
        "  model.add(MaxPooling2D(pool_size = (2,2)))\n",
        "  model.add(BatchNormalization())\n",
        "  model.add(Flatten())\n",
        "  model.add(Dense(units = 100, activation = 'relu'))\n",
        "  model.add(Dropout(0.5))\n",
        "  model.add(Dense(units = 10 , activation = 'softmax'))\n",
        "  return model\n",
        "\n",
        "model = create_model()\n",
        "model.summary()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d (Conv2D)              (None, 28, 28, 16)        160       \n",
            "_________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D) (None, 14, 14, 16)        0         \n",
            "_________________________________________________________________\n",
            "batch_normalization (BatchNo (None, 14, 14, 16)        64        \n",
            "_________________________________________________________________\n",
            "conv2d_1 (Conv2D)            (None, 14, 14, 32)        4640      \n",
            "_________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32)          0         \n",
            "_________________________________________________________________\n",
            "batch_normalization_1 (Batch (None, 7, 7, 32)          128       \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 1568)              0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 100)               156900    \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 162,902\n",
            "Trainable params: 162,806\n",
            "Non-trainable params: 96\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DH6vYwpXScJG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculate the Loss and Gradient "
      ]
    },
    {
      "metadata": {
        "id": "ZTBv2wsJSnFz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def loss(model, x, y):\n",
        "  #use cross entropy\n",
        "  prediction = model(x)\n",
        "  return tf.losses.softmax_cross_entropy(y, logits=prediction)\n",
        "\n",
        "def grad(model, x, y):\n",
        "  #use tf.GradientTape() to record the gradient \n",
        "  with tf.GradientTape() as tape:\n",
        "    loss_value = loss(model, x, y)\n",
        "  return tape.gradient(loss_value, model.variables)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aU8oNX2vSva3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calcuate Accuracy of the Model "
      ]
    },
    {
      "metadata": {
        "id": "icQFu_AFSzcs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def accuracy(model, x, y):\n",
        "  #calcuate the model prediction eagerly \n",
        "  yhat = model(x)\n",
        "  \n",
        "  #compare the predictions to the truth\n",
        "  yhat = tf.argmax(yhat, 1).numpy()\n",
        "  y    = tf.argmax(y   , 1).numpy()\n",
        "  return np.sum(y == yhat)/len(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zei0Ga5jTEJC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training\n",
        "Initialize the variables for training. Note that we need to know the length of an epoch in order to evaluate the metrics of the model. Also, we are using SGD to update the model paramteres "
      ]
    },
    {
      "metadata": {
        "id": "X-iPCeBwTHJv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#iterations and epochs variables \n",
        "i = 1\n",
        "epoch = 0\n",
        "epochs = 4 \n",
        "\n",
        "#running loss and accuracy\n",
        "running_loss = 0.\n",
        "running_acc  = 0.\n",
        "\n",
        "#calcuate the length of the epoch \n",
        "batch_size = 32 \n",
        "epoch_length = x_train.numpy().shape[0] // batch_size\n",
        "\n",
        "#optimizer to update the parmeters of the model \n",
        "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.05)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yJTCnxusTczk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "bdd9609a-25f9-4e82-8cf1-7651ffebdad0"
      },
      "cell_type": "code",
      "source": [
        "while epoch <= epochs:\n",
        "  #get next batch\n",
        "  x, y = get_batch(batch_size = batch_size)\n",
        "\n",
        "  # Calculate derivatives of the input function with respect to its parameters.\n",
        "  grads = grad(model, x, y)\n",
        "\n",
        "  # Apply the gradient to the model\n",
        "  optimizer.apply_gradients(zip(grads, model.variables),\n",
        "                            global_step=tf.train.get_or_create_global_step())\n",
        "  loss_value = loss(model, x, y)\n",
        "  \n",
        "  #calcuate running loss and accuracy \n",
        "  running_loss += loss_value\n",
        "  running_acc  += accuracy(model, x, y)\n",
        "  \n",
        "  #report values at the end of the epoch \n",
        "  if i % epoch_length == 0:\n",
        "    print(\"Epoch: {:d} Loss: {:.3f}, Acc: {:.3f}\".format(epoch, running_loss/epoch_length, running_acc/epoch_length))\n",
        "    \n",
        "    #reset the running loss an accuracy \n",
        "    running_loss = 0 \n",
        "    running_acc  = 0 \n",
        "    \n",
        "    epoch += 1\n",
        "  i += 1"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0 Loss: 1.706, Acc: 0.780\n",
            "Epoch: 1 Loss: 1.498, Acc: 0.965\n",
            "Epoch: 2 Loss: 1.485, Acc: 0.977\n",
            "Epoch: 3 Loss: 1.480, Acc: 0.982\n",
            "Epoch: 4 Loss: 1.477, Acc: 0.985\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SN2lmMTOTx1L",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ]
    },
    {
      "metadata": {
        "id": "NsOsTt-WT461",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "23891420-1306-4190-edac-079fcd638ab1"
      },
      "cell_type": "code",
      "source": [
        "print('Accuracy on the test set , ', accuracy(model, x_test, y_test))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy on the test set ,  0.9788\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "S09UdnjoXvE9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Read more ** https://www.tensorflow.org/guide/eager"
      ]
    }
  ]
}