C3/W3/ungraded_labs/C3_W3_Lab_3_Conv1D.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hobbang2/TensorFlowCertification/blob/main/C3/W3/ungraded_labs/C3_W3_Lab_3_Conv1D.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qp17FpQfOiZf"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/https-deeplearning-ai/tensorflow-1-public/blob/master/C3/W3/ungraded_labs/C3_W3_Lab_3_Conv1D.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFiCyWQ-NC5D"
      },
      "source": [
        "# Ungraded Lab: Using Convolutional Neural Networks\n",
        "\n",
        "In this lab, you will look at another way of building your text classification model and this will be with a convolution layer. As you learned in Course 2 of this specialization, convolutions extract features by applying filters to the input. Let's see how you can use that for text data in the next sections."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fXV3zToEOiZj"
      },
      "outputs": [],
      "source": [
        "# Install this package to use Colab's GPU for training\n",
        "!apt install --allow-change-held-packages libcudnn8=8.4.1.50-1+cuda11.6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djvGxIRDHT5e"
      },
      "source": [
        "## Download and prepare the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y20Lud2ZMBhW"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "\n",
        "# Download the subword encoded pretokenized dataset\n",
        "dataset, info = tfds.load('imdb_reviews/subwords8k', with_info=True, as_supervised=True)\n",
        "\n",
        "# Get the tokenizer\n",
        "tokenizer = info.features['text'].encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AW-4Vo4TMUHb"
      },
      "outputs": [],
      "source": [
        "BUFFER_SIZE = 10000\n",
        "BATCH_SIZE = 256\n",
        "\n",
        "# Get the train and test splits\n",
        "train_data, test_data = dataset['train'], dataset['test'], \n",
        "\n",
        "# Shuffle the training data\n",
        "train_dataset = train_data.shuffle(BUFFER_SIZE)\n",
        "\n",
        "# Batch and pad the datasets to the maximum length of the sequences\n",
        "train_dataset = train_dataset.padded_batch(BATCH_SIZE)\n",
        "test_dataset = test_data.padded_batch(BATCH_SIZE)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfatNr6-IAcd"
      },
      "source": [
        "## Build the Model\n",
        "\n",
        "In Course 2, you were using 2D convolution layers because you were applying it on images. For temporal data such as text sequences, you will use [Conv1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D) instead so the convolution will happen over a single dimension. You will also append a pooling layer to reduce the output of the convolution layer. For this lab, you will use [GlobalMaxPooling1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalMaxPool1D) to get the max value across the time dimension. You can also use average pooling and you will do that in the next labs. See how these layers behave as standalone layers in the cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ay87qbqwIJaV"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# Hyperparameters\n",
        "batch_size = 1\n",
        "timesteps = 20\n",
        "features = 20\n",
        "filters = 128\n",
        "kernel_size = 5\n",
        "\n",
        "print(f'batch_size: {batch_size}')\n",
        "print(f'timesteps (sequence length): {timesteps}')\n",
        "print(f'features (embedding size): {features}')\n",
        "print(f'filters: {filters}')\n",
        "print(f'kernel_size: {kernel_size}')\n",
        "\n",
        "# Define array input with random values\n",
        "random_input = np.random.rand(batch_size,timesteps,features)\n",
        "print(f'shape of input array: {random_input.shape}')\n",
        "\n",
        "# Pass array to convolution layer and inspect output shape\n",
        "conv1d = tf.keras.layers.Conv1D(filters=filters, kernel_size=kernel_size, activation='relu')\n",
        "result = conv1d(random_input)\n",
        "print(f'shape of conv1d output: {result.shape}')\n",
        "\n",
        "# Pass array to max pooling layer and inspect output shape\n",
        "gmp = tf.keras.layers.GlobalMaxPooling1D()\n",
        "result = gmp(result)\n",
        "print(f'shape of global max pooling output: {result.shape}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lNNYF7tqO7it"
      },
      "source": [
        "You can build the model by simply appending the convolution and pooling layer after the embedding layer as shown below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jo1jjO3vn0jo"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Hyperparameters\n",
        "embedding_dim = 64\n",
        "filters = 128\n",
        "kernel_size = 5\n",
        "dense_dim = 64\n",
        "\n",
        "# Build the model\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(tokenizer.vocab_size, embedding_dim),\n",
        "    tf.keras.layers.Conv1D(filters=filters, kernel_size=kernel_size, activation='relu'),\n",
        "    tf.keras.layers.GlobalMaxPooling1D(),\n",
        "    tf.keras.layers.Dense(dense_dim, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "# Print the model summary\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Uip7QOVzMoMq"
      },
      "outputs": [],
      "source": [
        "# Set the training parameters\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLJu8HEvPG0L"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "Training will take around 30 seconds per epoch and you will notice that it reaches higher accuracies than the previous models you've built."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7mlgzaRDMtF6"
      },
      "outputs": [],
      "source": [
        "NUM_EPOCHS = 10\n",
        "\n",
        "# Train the model\n",
        "history = model.fit(train_dataset, epochs=NUM_EPOCHS, validation_data=test_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mp1Z7P9pYRSK"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Plot utility\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.plot(history.history['val_'+string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.legend([string, 'val_'+string])\n",
        "  plt.show()\n",
        "\n",
        "# Plot the accuracy and results \n",
        "plot_graphs(history, \"accuracy\")\n",
        "plot_graphs(history, \"loss\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rD7ZS84PlUp"
      },
      "source": [
        "## Wrap Up\n",
        "\n",
        "In this lab, you explored another model architecture you can use for text classification. In the next lessons, you will revisit full word encoding of the IMDB reviews and compare which model works best when the data is prepared that way."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "C3_W3_Lab_3_Conv1D.ipynb",
      "private_outputs": true,
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}