flowers_tf_lite.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "flowers_tf_lite.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jvishnuvardhan/Stackoverflow_Questions/blob/master/flowers_tf_lite.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n6ecAvsmQp1I"
      },
      "source": [
        "## To run this colab, press the \"Runtime\" button in the menu tab and then press the \"Run all\" button."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "77gENRVX40S7"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "both",
        "colab_type": "code",
        "id": "d8jyt37T42Vf",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hRTa3Ee15WsJ"
      },
      "source": [
        "# Recognize Flowers using Transfer Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dQHMcypT3vDT"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/community/en/flowers_tf_lite.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/community/en/flowers_tf_lite.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBMcobPHdD8O",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "assert tf.__version__.startswith('2')\n",
        "\n",
        "import os\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NOG3l_MsBO1A",
        "colab": {}
      },
      "source": [
        "tf.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v77rlkCKW0IJ"
      },
      "source": [
        "## Setup Input Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "j4QOy2uA3P_p"
      },
      "source": [
        "Download the flowers dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xxL2mjVVGIrV",
        "colab": {}
      },
      "source": [
        "_URL = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
        "\n",
        "zip_file = tf.keras.utils.get_file(origin=_URL, \n",
        "                                   fname=\"flower_photos.tgz\", \n",
        "                                   extract=True)\n",
        "\n",
        "base_dir = os.path.join(os.path.dirname(zip_file), 'flower_photos')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "z4gTv7ig2vMh"
      },
      "source": [
        "Use `ImageDataGenerator` to rescale the images.\n",
        "\n",
        "Create the train generator and specify where the train dataset directory, image size, batch size.\n",
        "\n",
        "Create the validation generator with similar approach as the train generator with the flow_from_directory() method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "aCLb_yV5JfF3",
        "colab": {}
      },
      "source": [
        "IMAGE_SIZE = 224\n",
        "BATCH_SIZE = 64\n",
        "\n",
        "datagen = tf.keras.preprocessing.image.ImageDataGenerator(\n",
        "    rescale=1./255, \n",
        "    validation_split=0.2)\n",
        "\n",
        "train_generator = datagen.flow_from_directory(\n",
        "    base_dir,\n",
        "    target_size=(IMAGE_SIZE, IMAGE_SIZE),\n",
        "    batch_size=BATCH_SIZE, \n",
        "    subset='training')\n",
        "\n",
        "val_generator = datagen.flow_from_directory(\n",
        "    base_dir,\n",
        "    target_size=(IMAGE_SIZE, IMAGE_SIZE),\n",
        "    batch_size=BATCH_SIZE, \n",
        "    subset='validation')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tx1L7fxxWA_G",
        "colab": {}
      },
      "source": [
        "for image_batch, label_batch in train_generator:\n",
        "  break\n",
        "image_batch.shape, label_batch.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZrFFcwUb3iK9"
      },
      "source": [
        "Save the labels in a file which will be downloaded later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-QFZIhWs4dsq",
        "colab": {}
      },
      "source": [
        "print (train_generator.class_indices)\n",
        "\n",
        "labels = '\\n'.join(sorted(train_generator.class_indices.keys()))\n",
        "\n",
        "with open('labels.txt', 'w') as f:\n",
        "  f.write(labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "duxD_UDSOmng",
        "colab": {}
      },
      "source": [
        "!cat labels.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OkH-kazQecHB"
      },
      "source": [
        "## Create the base model from the pre-trained convnets\n",
        "\n",
        "Create the base model from the **MobileNet V2** model developed at Google, and pre-trained on the ImageNet dataset, a large dataset of 1.4M images and 1000 classes of web images.\n",
        "\n",
        "First, pick which intermediate layer of MobileNet V2 will be used for feature extraction. A common practice is to use the output of the very last layer before the flatten operation, the so-called \"bottleneck layer\". The reasoning here is that the following fully-connected layers will be too specialized to the task the network was trained on, and thus the features learned by these layers won't be very useful for a new task. The bottleneck features, however, retain much generality.\n",
        "\n",
        "Let's instantiate an MobileNet V2 model pre-loaded with weights trained on ImageNet. By specifying the `include_top=False` argument, we load a network that doesn't include the classification layers at the top, which is ideal for feature extraction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "19IQ2gqneqmS",
        "colab": {}
      },
      "source": [
        "IMG_SHAPE = (IMAGE_SIZE, IMAGE_SIZE, 3)\n",
        "\n",
        "# Create the base model from the pre-trained model MobileNet V2\n",
        "base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,\n",
        "                                              include_top=False, \n",
        "                                              weights='imagenet')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rlx56nQtfe8Y"
      },
      "source": [
        "## Feature extraction\n",
        "You will freeze the convolutional base created from the previous step and use that as a feature extractor, add a classifier on top of it and train the top-level classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Tts8BbAtRGvk",
        "colab": {}
      },
      "source": [
        "base_model.trainable = False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wdMRM8YModbk"
      },
      "source": [
        "### Add a classification head"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eApvroIyn1K0",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  base_model,\n",
        "  tf.keras.layers.Conv2D(32, 3, activation='relu'),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.GlobalAveragePooling2D(),\n",
        "  tf.keras.layers.Dense(5, activation='softmax')\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g0ylJXE_kRLi"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "You must compile the model before training it.  Since there are two classes, use a binary cross-entropy loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RpR8HdyMhukJ",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer=tf.keras.optimizers.Adam(), \n",
        "              loss='categorical_crossentropy', \n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "I8ARiyMFsgbH",
        "colab": {}
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "krvBumovycVA",
        "colab": {}
      },
      "source": [
        "print('Number of trainable variables = {}'.format(len(model.trainable_variables)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RxvgOYTDSWTx"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "<!-- TODO(markdaoust): delete steps_per_epoch in TensorFlow r1.14/r2.0 -->"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JsaRFlZ9B6WK",
        "colab": {}
      },
      "source": [
        "epochs = 10\n",
        "with tf.device(\"/device:GPU:0\"):\n",
        "  history = model.fit_generator(train_generator, steps_per_epoch=len(train_generator),\n",
        "                    epochs=epochs, \n",
        "                    validation_data=val_generator,validation_steps=len(val_generator))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Hd94CKImf8vi"
      },
      "source": [
        "### Learning curves\n",
        "\n",
        "Let's take a look at the learning curves of the training and validation accuracy/loss when using the MobileNet V2 base model as a fixed feature extractor. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "53OTCh3jnbwV",
        "colab": {}
      },
      "source": [
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.plot(acc, label='Training Accuracy')\n",
        "plt.plot(val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.ylim([min(plt.ylim()),1])\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(loss, label='Training Loss')\n",
        "plt.plot(val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.ylabel('Cross Entropy')\n",
        "plt.ylim([0,1.0])\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CqwV-CRdS6Nv"
      },
      "source": [
        "## Fine tuning\n",
        "In our feature extraction experiment, you were only training a few layers on top of an MobileNet V2 base model. The weights of the pre-trained network were **not** updated during training.\n",
        "\n",
        "One way to increase performance even further is to train (or \"fine-tune\") the weights of the top layers of the pre-trained model alongside the training of the classifier you added. The training process will force the weights to be tuned from generic features maps to features associated specifically to our dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CPXnzUK0QonF"
      },
      "source": [
        "### Un-freeze the top layers of the model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rfxv_ifotQak"
      },
      "source": [
        "All you need to do is unfreeze the `base_model` and set the bottom layers be un-trainable. Then, recompile the model (necessary for these changes to take effect), and resume training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "4nzcagVitLQm",
        "colab": {}
      },
      "source": [
        "base_model.trainable = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-4HgVAacRs5v",
        "colab": {}
      },
      "source": [
        "# Let's take a look to see how many layers are in the base model\n",
        "print(\"Number of layers in the base model: \", len(base_model.layers))\n",
        "\n",
        "# Fine tune from this layer onwards\n",
        "fine_tune_at = 100\n",
        "\n",
        "# Freeze all the layers before the `fine_tune_at` layer\n",
        "for layer in base_model.layers[:fine_tune_at]:\n",
        "  layer.trainable =  False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4Uk1dgsxT0IS"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "Compile the model using a much lower training rate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NtUnaz0WUDva",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer = tf.keras.optimizers.Adam(1e-5),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WwBWy7J2kZvA",
        "colab": {}
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bNXelbMQtonr",
        "colab": {}
      },
      "source": [
        "print('Number of trainable variables = {}'.format(len(model.trainable_variables)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4G5O4jd6TuAG"
      },
      "source": [
        "### Continue Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ECQLkAsFTlun",
        "colab": {}
      },
      "source": [
        "with tf.device(\"/device:GPU:0\"):\n",
        "  history_fine = model.fit_generator(train_generator, steps_per_epoch=len(train_generator),\n",
        "                         epochs=5,\n",
        "                         validation_data=val_generator,validation_steps=len(val_generator))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kRDabW_u1wnv"
      },
      "source": [
        "## Convert to TFLite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hNvMl6CM6lG4"
      },
      "source": [
        "Saved the model using `tf.saved_model.save` and then convert the saved model to a tf lite compatible format."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_LZiKVInWNGy",
        "colab": {}
      },
      "source": [
        "saved_model_dir = 'save/fine_tuning'\n",
        "tf.saved_model.save(model, saved_model_dir)\n",
        "\n",
        "converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)\n",
        "tflite_model = converter.convert()\n",
        "\n",
        "with open('model.tflite', 'wb') as f:\n",
        "  f.write(tflite_model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GE4w-9S410Dk"
      },
      "source": [
        "Download the converted model and labels"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "x47uW_lI1DoV",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "files.download('model.tflite')\n",
        "files.download('labels.txt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TfXEmsxQf6eP"
      },
      "source": [
        "Let's take a look at the learning curves of the training and validation accuracy/loss, when fine tuning the last few layers of the MobileNet V2 base model and training the classifier on top of it. The validation loss is much higher than the training loss, so you may get some overfitting.\n",
        "\n",
        "You may also get some overfitting as the new training set is relatively small and similar to the original MobileNet V2 datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "chW103JUItdk",
        "colab": {}
      },
      "source": [
        "acc = history_fine.history['accuracy']\n",
        "val_acc = history_fine.history['val_accuracy']\n",
        "\n",
        "loss = history_fine.history['loss']\n",
        "val_loss = history_fine.history['val_loss']\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.plot(acc, label='Training Accuracy')\n",
        "plt.plot(val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.ylim([min(plt.ylim()),1])\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(loss, label='Training Loss')\n",
        "plt.plot(val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.ylabel('Cross Entropy')\n",
        "plt.ylim([0,1.0])\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_TZTwG7nhm0C"
      },
      "source": [
        "## Summary:\n",
        "\n",
        "* **Using a pre-trained model for feature extraction**:  When working with a small dataset, it is common to take advantage of features learned by a model trained on a larger dataset in the same domain. This is done by instantiating the pre-trained model and adding a fully-connected classifier on top. The pre-trained model is \"frozen\" and only the weights of the classifier get updated during training.\n",
        "In this case, the convolutional base extracted all the features associated with each image and you just trained a classifier that determines the image class given that set of extracted features.\n",
        "\n",
        "* **Fine-tuning a pre-trained model**: To further improve performance, one might want to repurpose the top-level layers of the pre-trained models to the new dataset via fine-tuning.\n",
        "In this case, you tuned your weights such that your model learned high-level features specific to the dataset. This technique is usually recommended when the training dataset is large and very similar to the orginial dataset that the pre-trained model was trained on.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N-o4oLrCEkGH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V7ZTdX3TEnSz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}