NLP/Word_Embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Word Embeddings.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ramapriyakp/Portfolio/blob/master/NLP/Word_Embeddings.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GbmPqswtG_ns",
        "colab_type": "text"
      },
      "source": [
        "## Word Embedding \n",
        "\n",
        "Word embeddings are N-dimensional vectors that try to capture word-meaning and context in their values.  Word embeddings can be generated using various methods like neural networks, co-occurrence matrix, probabilistic models, etc.\n",
        "\n",
        "There’s a few key characteristics to a set of useful word embeddings:\n",
        "\n",
        "*  Every word has a unique word embedding (or “vector”), which is just a list of numbers for each word.\n",
        "*  The word embeddings are multidimensional; typically for a good model, embeddings are between 50 and 500 in length.\n",
        "*  For each word, the embedding captures the “meaning” of the word.\n",
        "*  Similar words end up with similar embedding values.\n",
        "\n",
        "\n",
        "### Word2Vec \n",
        "consists of models for generating word embedding. These models are shallow two layer neural networks having one input layer, one hidden layer and one output layer. Word2Vec utilizes two architectures :\n",
        "\n",
        "### CBOW (Continuous Bag of Words) : \n",
        "CBOW model predicts the current word given context words within specific window. The input layer contains the context words and the output layer contains the current word. The hidden layer contains the number of dimensions in which we want to represent current word present at the output layer.\n",
        "![alt text](https://media.geeksforgeeks.org/wp-content/uploads/cbow-1.png)\n",
        "### Skip Gram \n",
        " Skip gram predicts the surrounding context words within specific window given current word. The input layer contains the current word and the output layer contains the context words. The hidden layer contains the number of dimensions in which we want to represent current word present at the input layer.\n",
        " ![alt text](https://media.geeksforgeeks.org/wp-content/uploads/skip_gram.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OrokD7V5PTYZ",
        "colab_type": "code",
        "outputId": "cd3fd7aa-1597-4dce-9460-92e4b98ca179",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3juooBQjC3xr",
        "colab_type": "text"
      },
      "source": [
        "##Word Embeddings\n",
        "*  Using embedding layer in Keras\n",
        "*  Another way of learning word embeddings is via pre-training word vectors in another network (e.g., word2vec, GloVe, fasttext, etc.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "moaQuUlUPyk-",
        "colab_type": "code",
        "outputId": "679b9e9d-b845-4c05-e8af-3cfa0c4a3b55",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "cd /content/drive/My Drive/NLP"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/NLP\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LG2noVBGDQPz",
        "colab_type": "text"
      },
      "source": [
        "##Word vectors\n",
        "*  Long story short, word embedding is process of converting each word to a fixed dimensional \"(word) vector\"\n",
        "*  Dimensionality of embedding space (i.e., vector space) is a hyperparameter; one can set dimensionality as any positive integer.\n",
        "\n",
        "Unfortunately, this approach to word representation does not addres polysemy, or the co-existence of many possible meanings for a given word or phrase."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TPmNctb4QIhw",
        "colab_type": "code",
        "outputId": "c7d035ac-3d7d-44f2-e4b0-a895ea508f49",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import *\n",
        "from keras.datasets import reuters\n",
        "from keras.preprocessing import sequence\n",
        "from keras.utils import to_categorical"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YPXpps2ADctu",
        "colab_type": "text"
      },
      "source": [
        "##Embedding layer\n",
        "As a result, embedding layer bears 3-D tensors\n",
        "*  Output shape = (batch_size, input_length, output_dim)\n",
        "*  input_dim: dimensionality of input space (number of unique tokens of interest)\n",
        "*  output_dim: dimensionality of embedding space\n",
        "*  input_length: length of input sequence (if None, can vary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4W1mup_0QSCk",
        "colab_type": "code",
        "outputId": "b4a5253c-f837-4437-87ac-739db85c48fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "# when input length is constant\n",
        "model = Sequential()\n",
        "model.add(Embedding(input_dim = 10, output_dim = 5, input_length = 3))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iykf9D3_QXvB",
        "colab_type": "code",
        "outputId": "4840bcbe-067b-4f58-dcfd-1f322fffaeb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "model.output_shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, 3, 5)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UsEANi0ODz7W",
        "colab_type": "text"
      },
      "source": [
        "##Using embedding layer in network\n",
        "Usually, embedding layer are used as first layer in network to model text format data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LfJhYzQ-QcEf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# parameters to import dataset\n",
        "num_words = 3000\n",
        "maxlen = 50"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nul99a5FQg_t",
        "colab_type": "code",
        "outputId": "cc47082c-f56e-42c9-e170-2e6977b77619",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "(X_train, y_train), (X_test, y_test) = reuters.load_data(num_words = num_words, maxlen = maxlen)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/text-datasets/reuters.npz\n",
            "2113536/2110848 [==============================] - 1s 1us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hR2S1JNkQmLe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'post')\n",
        "X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'post')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nq5QnLzAQn8q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_train = to_categorical(y_train, num_classes = 46)\n",
        "y_test = to_categorical(y_test, num_classes = 46)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eOuymVsvQrk7",
        "colab_type": "code",
        "outputId": "fc03d463-d5f6-4815-d679-50ac57291ef0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "print(X_train.shape)\n",
        "print(X_test.shape)\n",
        "print(y_train.shape)\n",
        "print(y_test.shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1595, 50)\n",
            "(399, 50)\n",
            "(1595, 46)\n",
            "(399, 46)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lcaq-oKGQwhg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "input_dim = num_words\n",
        "output_dim = 100     # we set dimensionality of embedding space as 100\n",
        "input_length = maxlen"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GQS7z79cQ0jG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def reuters_model():\n",
        "    model = Sequential()\n",
        "    model.add(Embedding(input_dim = input_dim, output_dim = output_dim, input_length = input_length))\n",
        "    model.add(CuDNNGRU(50, return_sequences = False))\n",
        "    model.add(Dense(100))\n",
        "    model.add(Activation('relu'))\n",
        "    model.add(Dense(46, activation = 'softmax'))\n",
        "    \n",
        "    model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4GfUX8TkQ4qc",
        "colab_type": "code",
        "outputId": "1a0589e5-c4fb-4ea9-e7e3-3d64f9f765da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        }
      },
      "source": [
        "model = reuters_model()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aZoqbNdmQ76h",
        "colab_type": "code",
        "outputId": "8833d305-17f7-4c70-8187-4ac4bbf849e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_2 (Embedding)      (None, 50, 100)           300000    \n",
            "_________________________________________________________________\n",
            "cu_dnngru_1 (CuDNNGRU)       (None, 50)                22800     \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 100)               5100      \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 100)               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 46)                4646      \n",
            "=================================================================\n",
            "Total params: 332,546\n",
            "Trainable params: 332,546\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5OWVI5GMQ_ic",
        "colab_type": "code",
        "outputId": "16e565cc-03fa-4d3a-cfdd-5eee36bb2c43",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "model.fit(X_train, y_train, epochs = 100, batch_size = 100, verbose = 0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f6df2b0a470>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "udB182cWRDWC",
        "colab_type": "code",
        "outputId": "2e82a20d-10aa-4aa1-d803-b3f20d7f56fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "\n",
        "result = model.evaluate(X_test, y_test)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "399/399 [==============================] - 0s 205us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ff2gVfoHRHSt",
        "colab_type": "code",
        "outputId": "8504421d-2e76-47a3-b007-6d19b4936c6f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print('Test Accuracy: ', result[1])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Accuracy:  0.8571428575910124\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}