tensorflow_serve_rnn_prod.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tensorflow_serve_rnn_prod.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/mdasadul/MNIST/blob/master/tensorflow_serve_rnn_prod.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "fWGnnJWxAlrr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "I am interested to do following\n",
        "\n",
        "\n",
        "*   Train a RNN model\n",
        "*   Save the model\n",
        "*   Export the model\n",
        "*   Serve it to production using Tensorflow \n",
        "\n",
        "\n",
        "\n",
        "Let's load necessary libraries"
      ]
    },
    {
      "metadata": {
        "id": "drkDcGL276QA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from __future__ import print_function\n",
        "import os\n",
        "import numpy as np\n",
        "import shutil\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow.contrib import rnn\n",
        "\n",
        "# Import MNIST data\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "21Eh2UmkZRzK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We are downloading data into a temporary location and we are converting each example of our dataset into one_hot vector. To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then\n",
        "handle 28 sequences of 28 steps for every sample.\n"
      ]
    },
    {
      "metadata": {
        "id": "W_nDgNloZ-9_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "09af4b40-8278-4a5b-9312-b5316ef27caf"
      },
      "cell_type": "code",
      "source": [
        "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
            "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
            "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
            "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dlgENxxmZ8au",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We are setting up training parameters and network parameters. We are keeping learning rate low and making sure the model is not going to overfit training data."
      ]
    },
    {
      "metadata": {
        "id": "3xEskJmJkCBO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "learning_rate = 0.001\n",
        "training_steps = 1000\n",
        "batch_size = 128\n",
        "display_step = 200\n",
        "\n",
        "# Network Parameters\n",
        "num_input = 28 # MNIST data input (img shape: 28*28)\n",
        "timesteps = 28 # timesteps\n",
        "num_hidden = 128 # hidden layer num of features\n",
        "num_classes = 10 # MNIST total classes (0-9 digits)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1meN6hxMkIM1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "It is very common in tensorflow to use placeholder and use feed_dict to feed data during training and inference. We are naming input to \"Input_X\" so that we can use it in future when we will be saving the graph and also during inferencing."
      ]
    },
    {
      "metadata": {
        "id": "kKlYb_fQn9J5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Training Parameters\n",
        "\n",
        "output_layer = ['output_layer/add']\n",
        "\n",
        "# tf Graph input\n",
        "X = tf.placeholder(\"float\", [None, timesteps, num_input],name ='Input_X')\n",
        "Y = tf.placeholder(\"float\", [None, num_classes])\n",
        "\n",
        "# Define weights\n",
        "weights = {\n",
        "    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))\n",
        "}\n",
        "biases = {\n",
        "    'out': tf.Variable(tf.random_normal([num_classes]))}\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mI0l3gXVoB5B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We are defining our simple RNN model here. We atart by preparing data shape to match `rnn` function requirements. Then we are creating LSTM cell with  BasicLSTMCell. after that we are applying that on the input we prepared for. We did create those cell within a variable scope and set auto_reuse=true to reuse the module.  Finally we are calculating logit by applying linear activation. In the inference graph we can use this linear activation as output layer and apply softmax during inference for output. If we would like to use this linear activation layer as output we should put them into scope and assign name to the logit opearion. "
      ]
    },
    {
      "metadata": {
        "id": "XkVyYl1sq03W",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def RNN(x, weights, biases):\n",
        "\n",
        "    # Prepare data shape to match `rnn` function requirements\n",
        "    # Current data input shape: (batch_size, timesteps, n_input)\n",
        "    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)\n",
        "\n",
        "    x = tf.unstack(x, timesteps, 1)\n",
        "\n",
        "    # Define a lstm cell \n",
        "    with tf.variable_scope(\"rnn\", reuse=tf.AUTO_REUSE):\n",
        "      lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
        "      \n",
        "      outputs, _ = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)\n",
        "\n",
        "    # Linear activation, using rnn inner loop last output\n",
        "    with tf.name_scope('output_layer'):\n",
        "        logit = tf.add(tf.matmul(outputs[-1], weights['out']) , biases['out'],name ='add')\n",
        "    return logit"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FAmw653sq5_W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "After getting logits we will apply softmax operation for prediction. An alternative to the linear activation layer in the model we can use the softmax layer as our output layer. Inorder to use this as a output layer we should provide a name for future use."
      ]
    },
    {
      "metadata": {
        "id": "gV0RXUcRrlPL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "logits = RNN(X, weights, biases)\n",
        "prediction = tf.nn.softmax(logits,name='y_')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "g9F2Uv1arnsK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next step is very typical to use a loss function on our case softmax_cross_entropy_with_logits_v2 to comapre the predicted output with the actual label and use an optimizer( Adam, SGD, RMSProp etc) to minimize the loss.  We are also calculating correct prediction and finally accuracy "
      ]
    },
    {
      "metadata": {
        "id": "EOREEibLsW44",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Define loss and optimizer\n",
        "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(\n",
        "    logits=logits, labels=Y))\n",
        "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
        "train_op = optimizer.minimize(loss_op)\n",
        "\n",
        "# Evaluate model\n",
        "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
        "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "g_9qUcuBsbm3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This is one of the important step were we are initializing variables including weights and biases. We are getting input and output tensor by their name from the graph. One important details here that we defined input variable as \"Input_X\" and output_layer as \"output_layer/add\" but we are adding \":0\" at the end of their name. The name itself in the graph are appeared as operation and if we would like to use them as tensor we have to add \":0\" at the end of the operation."
      ]
    },
    {
      "metadata": {
        "id": "i-m2T6QjDb_I",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize the variables (i.e. assign their default value)\n",
        "init = tf.global_variables_initializer()\n",
        "output_tensor = tf.get_default_graph().get_tensor_by_name(\"output_layer/add:0\")\n",
        "input_tensor = tf.get_default_graph().get_tensor_by_name(\"Input_X:0\")\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DM0wyWsaDe2e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally we are defining Saver to save the model as a checkpint. We can load these checkpint for retraining and inference puspose. But the resulted graph has many operation which are not necessary for inferencing and the size of the save model is quite large as a result inferencing will be somewhat 20/30 % slower. We defined input and output tensor above and we will use these once we are done with training to expot frozen model for inferencing purpose.\n",
        "\n",
        "We are starting training by creating new ssession. Ater running the varibles initializer  we are starting the trainig loop upto predefined number of steps. We are iterating through training data and feed them to the model batch by batch and do the optimization to minimize the loss we defined above. Once in a while (disply step) we will print the loss and accuracy of the training to make sure that the loss is going downword and accuracy is upword. After finishing the training loop we will save it so that we can use it for future retraining. "
      ]
    },
    {
      "metadata": {
        "id": "UoHtteu2IJwH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "saver = tf.train.Saver()\n",
        "\n",
        "# start training\n",
        "with tf.Session() as sess:\n",
        "\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "\n",
        "    for step in range(1, training_steps+1):\n",
        "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
        "        # Reshape data to get 28 seq of 28 elements\n",
        "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
        "        # Run optimization op (backprop)\n",
        "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
        "        if step % display_step == 0 or step == 1:\n",
        "            # Calculate batch loss and accuracy\n",
        "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
        "                                                                 Y: batch_y})\n",
        "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
        "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
        "                  \"{:.3f}\".format(acc))\n",
        "    #for op in tf.get_default_graph().get_operations():\n",
        "    #    if output_layer[0] in op.name:\n",
        "    #            print(op.name)\n",
        "    print(\"Optimization Finished!\")\n",
        "    saver.save(sess,'./model.ckpt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hVuX1vH9INS3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once we are done with training we will select a subset of test data and do the prediction and make sure that test accuracy is not far off from the train accuracy."
      ]
    },
    {
      "metadata": {
        "id": "f5RVoRoFKQ0j",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "   # Calculate accuracy for 128 mnist test images\n",
        "    test_len = 128\n",
        "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
        "    test_label = mnist.test.labels[:test_len]\n",
        "    print(\"Testing prediction:\", \\\n",
        "        sess.run(prediction, feed_dict={X: test_data}))\n",
        "\n",
        "    print(\"Testing Accuracy:\", \\\n",
        "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "F9aXYRSsKTcz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally we will export the trained model for inference. We will SavedModelBuilder for exporting.\n",
        "SavedModelBuilder will create the directory if doesn't exists. It is possible to add model version here but for this demo example we will remove previously saved model before saving another one. __tensor_info_x__ and __tensor_info_y__ are protocol buffer defined by using SavedModel API."
      ]
    },
    {
      "metadata": {
        "id": "riGkmZ7JNnos",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "    # Export the model for prediction\n",
        "    export_path =  './exportmodel'\n",
        "    # removing previously exported model\n",
        "    shutil.rmtree(export_path)\n",
        "    builder = tf.saved_model.builder.SavedModelBuilder(export_path)\n",
        "\n",
        "    \n",
        "    tensor_info_x = tf.saved_model.utils.build_tensor_info(input_tensor)\n",
        "    tensor_info_y = tf.saved_model.utils.build_tensor_info(output_tensor)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jGjXMa7uN56w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We are defining the signature which is useful for prediction. The signature defination should be build by using key, value stracture. We are naming the key of the input as \"x_input\" which is protocol buffer for input_X and th output as \"y_output\" which is the protocol buffer for the tensor for logit.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "fT6NpTeEQHUl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "   prediction_signature = (\n",
        "         tf.saved_model.signature_def_utils.build_signature_def(\n",
        "         inputs={'x_input': tensor_info_x},\n",
        "          outputs={'y_output': tensor_info_y},\n",
        "         method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-gLEgzhbQPl7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally we are adding meta graph and variables such as Input_X and logit to the builder by using SavedModelBuilder.add_meta_graph_and_variables() with the following arguments:\n",
        "\n",
        "\n",
        "*   sess: The rensorflow session that holds the trained model\n",
        "*   tags: The set of tags with which to save the meta graph. In this case, since we intend to use the graph in serving, we use the serve tag from predefined SavedModel tag constants. \n",
        "*   signature_def_map: The map of user-supplied key for a signature to a tensorflow::SignatureDef to add to the meta graph. Signature specifies what type of model is being exported, and the input/output tensors to bind to when running inference.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "lZhwHsuORxTt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 504
        },
        "outputId": "fb542947-dc79-46b9-a740-0201c6a21d47"
      },
      "cell_type": "code",
      "source": [
        " \n",
        "    builder.add_meta_graph_and_variables(\n",
        "       sess, [tf.saved_model.tag_constants.SERVING],\n",
        "       signature_def_map={\n",
        "      tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:\n",
        "          prediction_signature\n",
        "       },\n",
        "      )\n",
        "    builder.save()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Step 1, Minibatch Loss= 2.3642, Training Accuracy= 0.102\n",
            "Step 200, Minibatch Loss= 2.0342, Training Accuracy= 0.383\n",
            "Step 400, Minibatch Loss= 1.8062, Training Accuracy= 0.469\n",
            "Step 600, Minibatch Loss= 1.7246, Training Accuracy= 0.469\n",
            "Step 800, Minibatch Loss= 1.6795, Training Accuracy= 0.500\n",
            "Step 1000, Minibatch Loss= 1.6217, Training Accuracy= 0.500\n",
            "output_layer/add\n",
            "gradients/output_layer/add_grad/Shape\n",
            "gradients/output_layer/add_grad/Shape_1\n",
            "gradients/output_layer/add_grad/BroadcastGradientArgs\n",
            "gradients/output_layer/add_grad/Sum\n",
            "gradients/output_layer/add_grad/Reshape\n",
            "gradients/output_layer/add_grad/Sum_1\n",
            "gradients/output_layer/add_grad/Reshape_1\n",
            "gradients/output_layer/add_grad/tuple/group_deps\n",
            "gradients/output_layer/add_grad/tuple/control_dependency\n",
            "gradients/output_layer/add_grad/tuple/control_dependency_1\n",
            "Optimization Finished!\n",
            "Testing prediction: [[0.00468715 0.0743486  0.00581111 ... 0.5039283  0.06872662 0.12138908]\n",
            " [0.08121999 0.01353763 0.29797387 ... 0.01006361 0.18041241 0.0291665 ]\n",
            " [0.0015001  0.7218817  0.014951   ... 0.07694507 0.02122275 0.04575193]\n",
            " ...\n",
            " [0.02510563 0.02112834 0.12790453 ... 0.11117595 0.05063821 0.13330224]\n",
            " [0.45245665 0.00471736 0.14605731 ... 0.02836649 0.01829042 0.02736905]\n",
            " [0.02147828 0.01703928 0.05430871 ... 0.11795196 0.25505272 0.14612368]]\n",
            "Testing Accuracy: 0.4609375\n",
            "INFO:tensorflow:No assets to save.\n",
            "INFO:tensorflow:No assets to write.\n",
            "INFO:tensorflow:SavedModel written to: b'./save/saved_model.pb'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "iaIv0lJFRzIB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We exported the model in export model directory and we are ready to do inference. Inferencing can be done few different ways. We can use standard python API or we can use google RPC along with Tensorflow Serving API. We are desiging our inferencing technique which can be used by both technique with little modification. \n",
        "\n",
        "We are starting by creating a session. At first we get the default signature defination key. After that we assign the key which we used to save input and output of the model. After that we load the model from where we exported by using tensorflow SavedModel API as a meta graph. After that we extract the signature so that we can extract input and output tensor by name. Finally we get the tensor from the session by thier name and assign input of the graph as x and logit as y. Now we are reay for inferening. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "x-waBLAfVRd9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sess=tf.Session() \n",
        "signature_key = tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY\n",
        "input_key = 'x_input'\n",
        "output_key = 'y_output'\n",
        "\n",
        "export_path =  './exportmodel'\n",
        "meta_graph_def = tf.saved_model.loader.load(\n",
        "           sess,\n",
        "          [tf.saved_model.tag_constants.SERVING],\n",
        "          export_path)\n",
        "signature =  meta_graph_def.signature_def\n",
        "\n",
        "x_tensor_name = signature[signature_key].inputs[input_key].name\n",
        "y_tensor_name = signature[signature_key].outputs[output_key].name\n",
        "\n",
        "x = sess.graph.get_tensor_by_name(x_tensor_name)\n",
        "y = sess.graph.get_tensor_by_name(y_tensor_name)\n",
        "# Import MNIST data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xHmpUVJYVSMk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we can get some test data and send the data to the model one by one. I am showing inference for one particular example if you need to do more than one it is possible to achieve that by using a loop"
      ]
    },
    {
      "metadata": {
        "id": "SFFFt8NR9bf4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
        "batch_x= mnist.test.images[:1].reshape(-1,28,28)\n",
        "# Reshape data to get 28 seq of 28 elements\n",
        "batch_y = mnist.test.labels[:1][0]\n",
        "prediction = sess.run([y], feed_dict={x: batch_x})\n",
        "print(np.argmax(batch_y))\n",
        "print(np.argmax(prediction))\n",
        "correct_pred = tf.equal(np.argmax(prediction), np.argmax(batch_y))\n",
        "print(sess.run(correct_pred))"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}