Basics/Text-Preprocessing/Bag-of-Words.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.10"
    },
    "colab": {
      "name": "Bag-of-Words.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Gladiator07/Natural-Language-Processing/blob/main/Basics/Text-Preprocessing/Bag-of-Words.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hFL76tVmFmx"
      },
      "source": [
        "# Bag Of Words"
      ],
      "id": "-hFL76tVmFmx"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJIg8LJKKoDN"
      },
      "source": [
        "### Bag of Words (using plain Python and nltk)"
      ],
      "id": "gJIg8LJKKoDN"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J6DjHGuXntv0"
      },
      "source": [
        "Bag of words keeps count of the total occurences of most frequently used words"
      ],
      "id": "J6DjHGuXntv0"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "biological-shape",
        "outputId": "90741f95-684e-4631-b2fe-00d6d476a499"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "import re\n",
        "import numpy as np"
      ],
      "id": "biological-shape",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lb7DjcfEmJy5"
      },
      "source": [
        "text = \"\"\"\n",
        "Beans. I was trying to explain to somebody as we were flying in, that’s corn. \n",
        "That’s beans. And they were very impressed at my agricultural knowledge. \n",
        "Please give it up for Amaury once again for that outstanding introduction. \n",
        "I have a bunch of good friends here today, including somebody who I served with, \n",
        "who is one of the finest senators in the country, and we’re lucky to have him, \n",
        "your Senator, Dick Durbin is here. I also noticed, by the way, former Governor \n",
        "Edgar here, who I haven’t seen in a long time, and somehow he has not aged and \n",
        "I have. And it’s great to see you, Governor. I want to thank President Killeen \n",
        "and everybody at the U of I System for making it possible for me to be here today. \n",
        "And I am deeply honored at the Paul Douglas Award that is being given to me. \n",
        "He is somebody who set the path for so much outstanding public service here in Illinois. \n",
        "Now, I want to start by addressing the elephant in the room. I know people are \n",
        "still wondering why I didn’t speak at the commencement.\"\"\""
      ],
      "id": "Lb7DjcfEmJy5",
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDIYsG0mn8fM"
      },
      "source": [
        "#### Step-1 \n",
        "- Convert text to lower case\n",
        "- Remove all non-word charachters\n",
        "- Remove all punctuations"
      ],
      "id": "mDIYsG0mn8fM"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MfB3swfpoD0d"
      },
      "source": [
        "data = nltk.sent_tokenize(text)\n",
        "for i in range(len(data)):\n",
        "    data[i] = data[i].lower()\n",
        "    data[i] = re.sub(r'\\W', ' ', data[i])\n",
        "    data[i] = re.sub(r'\\s+', ' ', data[i])"
      ],
      "id": "MfB3swfpoD0d",
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4jhv6yBcoQhK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "00f5d433-879a-4dc1-c0a6-a805ad64c540"
      },
      "source": [
        "data"
      ],
      "id": "4jhv6yBcoQhK",
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[' beans ',\n",
              " 'i was trying to explain to somebody as we were flying in that s corn ',\n",
              " 'that s beans ',\n",
              " 'and they were very impressed at my agricultural knowledge ',\n",
              " 'please give it up for amaury once again for that outstanding introduction ',\n",
              " 'i have a bunch of good friends here today including somebody who i served with who is one of the finest senators in the country and we re lucky to have him your senator dick durbin is here ',\n",
              " 'i also noticed by the way former governor edgar here who i haven t seen in a long time and somehow he has not aged and i have ',\n",
              " 'and it s great to see you governor ',\n",
              " 'i want to thank president killeen and everybody at the u of i system for making it possible for me to be here today ',\n",
              " 'and i am deeply honored at the paul douglas award that is being given to me ',\n",
              " 'he is somebody who set the path for so much outstanding public service here in illinois ',\n",
              " 'now i want to start by addressing the elephant in the room ',\n",
              " 'i know people are still wondering why i didn t speak at the commencement ']"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjaWAuRSCQqh"
      },
      "source": [
        "#### Step-2:\n",
        "- We declare a dictionary to hold our bag of words\n",
        "- Next we tokenize each sentence to words\n",
        "- Now for each word in sentence, we check if the words exists in our dictionary\n",
        "- If it does, then we increment its count by 1. If it doesn't, we add it to our dictionary and set its count as 1"
      ],
      "id": "PjaWAuRSCQqh"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BXqVljRRocTs"
      },
      "source": [
        "word2count = {}\n",
        "for d in data:\n",
        "    words = nltk.word_tokenize(d)\n",
        "    for word in words:\n",
        "        if word not in word2count.keys():\n",
        "            word2count[word] = 1\n",
        "        else:\n",
        "            word2count[word] += 1"
      ],
      "id": "BXqVljRRocTs",
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_WtseL6dC6fw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3c5ec1a9-b5e6-43a7-a15d-4a5275affd4d"
      },
      "source": [
        "word2count"
      ],
      "id": "_WtseL6dC6fw",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'a': 2,\n",
              " 'addressing': 1,\n",
              " 'again': 1,\n",
              " 'aged': 1,\n",
              " 'agricultural': 1,\n",
              " 'also': 1,\n",
              " 'am': 1,\n",
              " 'amaury': 1,\n",
              " 'and': 7,\n",
              " 'are': 1,\n",
              " 'as': 1,\n",
              " 'at': 4,\n",
              " 'award': 1,\n",
              " 'be': 1,\n",
              " 'beans': 2,\n",
              " 'being': 1,\n",
              " 'bunch': 1,\n",
              " 'by': 2,\n",
              " 'commencement': 1,\n",
              " 'corn': 1,\n",
              " 'country': 1,\n",
              " 'deeply': 1,\n",
              " 'dick': 1,\n",
              " 'didn': 1,\n",
              " 'douglas': 1,\n",
              " 'durbin': 1,\n",
              " 'edgar': 1,\n",
              " 'elephant': 1,\n",
              " 'everybody': 1,\n",
              " 'explain': 1,\n",
              " 'finest': 1,\n",
              " 'flying': 1,\n",
              " 'for': 5,\n",
              " 'former': 1,\n",
              " 'friends': 1,\n",
              " 'give': 1,\n",
              " 'given': 1,\n",
              " 'good': 1,\n",
              " 'governor': 2,\n",
              " 'great': 1,\n",
              " 'has': 1,\n",
              " 'have': 3,\n",
              " 'haven': 1,\n",
              " 'he': 2,\n",
              " 'here': 5,\n",
              " 'him': 1,\n",
              " 'honored': 1,\n",
              " 'i': 12,\n",
              " 'illinois': 1,\n",
              " 'impressed': 1,\n",
              " 'in': 5,\n",
              " 'including': 1,\n",
              " 'introduction': 1,\n",
              " 'is': 4,\n",
              " 'it': 3,\n",
              " 'killeen': 1,\n",
              " 'know': 1,\n",
              " 'knowledge': 1,\n",
              " 'long': 1,\n",
              " 'lucky': 1,\n",
              " 'making': 1,\n",
              " 'me': 2,\n",
              " 'much': 1,\n",
              " 'my': 1,\n",
              " 'not': 1,\n",
              " 'noticed': 1,\n",
              " 'now': 1,\n",
              " 'of': 3,\n",
              " 'once': 1,\n",
              " 'one': 1,\n",
              " 'outstanding': 2,\n",
              " 'path': 1,\n",
              " 'paul': 1,\n",
              " 'people': 1,\n",
              " 'please': 1,\n",
              " 'possible': 1,\n",
              " 'president': 1,\n",
              " 'public': 1,\n",
              " 're': 1,\n",
              " 'room': 1,\n",
              " 's': 3,\n",
              " 'see': 1,\n",
              " 'seen': 1,\n",
              " 'senator': 1,\n",
              " 'senators': 1,\n",
              " 'served': 1,\n",
              " 'service': 1,\n",
              " 'set': 1,\n",
              " 'so': 1,\n",
              " 'somebody': 3,\n",
              " 'somehow': 1,\n",
              " 'speak': 1,\n",
              " 'start': 1,\n",
              " 'still': 1,\n",
              " 'system': 1,\n",
              " 't': 2,\n",
              " 'thank': 1,\n",
              " 'that': 4,\n",
              " 'the': 9,\n",
              " 'they': 1,\n",
              " 'time': 1,\n",
              " 'to': 8,\n",
              " 'today': 2,\n",
              " 'trying': 1,\n",
              " 'u': 1,\n",
              " 'up': 1,\n",
              " 'very': 1,\n",
              " 'want': 2,\n",
              " 'was': 1,\n",
              " 'way': 1,\n",
              " 'we': 2,\n",
              " 'were': 2,\n",
              " 'who': 4,\n",
              " 'why': 1,\n",
              " 'with': 1,\n",
              " 'wondering': 1,\n",
              " 'you': 1,\n",
              " 'your': 1}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SdJUR1udDDRl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3fc6ca2e-1bfb-451f-a2e1-1546d132f9d6"
      },
      "source": [
        "len(word2count)"
      ],
      "id": "SdJUR1udDDRl",
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "118"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QALk8SNDJX4"
      },
      "source": [
        "We have 118 words in our vocabulary\n",
        "\n",
        "However, when processing large texts, the number of words could reach millions. We do not need to use all those words. Hence, we select a particular number of most frequently used words. Like this:\n",
        "\n"
      ],
      "id": "1QALk8SNDJX4"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1MH3UBH5DIAt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95db52bb-176b-44a3-dc30-b4b38e66cc6f"
      },
      "source": [
        "import heapq\n",
        "freq_words = heapq.nlargest(100, word2count, key=word2count.get)\n",
        "len(freq_words)"
      ],
      "id": "1MH3UBH5DIAt",
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "100"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JoluqaWBFfUD"
      },
      "source": [
        "#### Step-3: Building the Bag of Words model"
      ],
      "id": "JoluqaWBFfUD"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "em50753CFmU3"
      },
      "source": [
        "In this step we construct a vector, which would tell us whether a word in each sentence is a frequent word or not. If a word in a sentence is a frequent word, we set it as 1, else we set it as 0.\n"
      ],
      "id": "em50753CFmU3"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5lZpUYvqFV-k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "18820f0e-7337-4de0-8f35-3edcf8cae592"
      },
      "source": [
        "X = []\n",
        "for d in data:\n",
        "    vector = []\n",
        "    for word in freq_words:\n",
        "        if word in nltk.word_tokenize(d):\n",
        "            vector.append(1)\n",
        "        else:\n",
        "            vector.append(0)\n",
        "    X.append(vector)\n",
        "X = np.asarray(X)\n",
        "print(X)"
      ],
      "id": "5lZpUYvqFV-k",
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0 0 0 ... 0 0 0]\n",
            " [1 0 1 ... 0 0 0]\n",
            " [0 0 0 ... 0 0 0]\n",
            " ...\n",
            " [0 1 0 ... 1 1 1]\n",
            " [1 1 1 ... 0 0 0]\n",
            " [1 1 0 ... 0 0 0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S11U8NezF3Ju",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "280725ae-f0c0-4814-d380-5eb3e6034e57"
      },
      "source": [
        "X.shape"
      ],
      "id": "S11U8NezF3Ju",
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(13, 100)"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KUSdbfY-GF4q"
      },
      "source": [
        "So, here we have 13 sentences and 100 dim vector (0 or 1) (vocabulary) in each sentence"
      ],
      "id": "KUSdbfY-GF4q"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QiuBo8ygGWyb"
      },
      "source": [
        "## Bag of Words (using sklearn and nltk)"
      ],
      "id": "QiuBo8ygGWyb"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AMWVFM6BGCsv"
      },
      "source": [
        "import nltk"
      ],
      "id": "AMWVFM6BGCsv",
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z4tfmmvDGhcT"
      },
      "source": [
        "paragraph =  \"\"\"I have three visions for India. In 3000 years of our history, people from all over \n",
        "               the world have come and invaded us, captured our lands, conquered our minds. \n",
        "               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,\n",
        "               the French, the Dutch, all of them came and looted us, took over what was ours. \n",
        "               Yet we have not done this to any other nation. We have not conquered anyone. \n",
        "               We have not grabbed their land, their culture, \n",
        "               their history and tried to enforce our way of life on them. \n",
        "               Why? Because we respect the freedom of others.That is why my \n",
        "               first vision is that of freedom. I believe that India got its first vision of \n",
        "               this in 1857, when we started the War of Independence. It is this freedom that\n",
        "               we must protect and nurture and build on. If we are not free, no one will respect us.\n",
        "               My second vision for India’s development. For fifty years we have been a developing nation.\n",
        "               It is time we see ourselves as a developed nation. We are among the top 5 nations of the world\n",
        "               in terms of GDP. We have a 10 percent growth rate in most areas. Our poverty levels are falling.\n",
        "               Our achievements are being globally recognised today. Yet we lack the self-confidence to\n",
        "               see ourselves as a developed nation, self-reliant and self-assured. Isn’t this incorrect?\n",
        "               I have a third vision. India must stand up to the world. Because I believe that unless India \n",
        "               stands up to the world, no one will respect us. Only strength respects strength. We must be \n",
        "               strong not only as a military power but also as an economic power. Both must go hand-in-hand. \n",
        "               My good fortune was to have worked with three great minds. Dr. Vikram Sarabhai of the Dept. of \n",
        "               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.\n",
        "               I was lucky to have worked with all three of them closely and consider this the great opportunity of my life. \n",
        "               I see four milestones in my career\"\"\""
      ],
      "id": "z4tfmmvDGhcT",
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H931BupYGjdw"
      },
      "source": [
        "import re\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import SnowballStemmer\n",
        "from nltk.stem import WordNetLemmatizer"
      ],
      "id": "H931BupYGjdw",
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kmC34SbuG9jh"
      },
      "source": [
        "# creating objects of stemmers and lemmatizers\n",
        "stemmer = SnowballStemmer(language=\"english\")\n",
        "lemmatizer = WordNetLemmatizer()"
      ],
      "id": "kmC34SbuG9jh",
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "44zeRoprHIgz"
      },
      "source": [
        "sentences = nltk.sent_tokenize(paragraph)\n",
        "corpus = []\n",
        "\n",
        "for i in range(len(sentences)):\n",
        "    text = re.sub('[^a-zA-Z]', ' ', sentences[i])\n",
        "    text = text.lower()\n",
        "    text = nltk.word_tokenize(text)\n",
        "    text = [stemmer.stem(word) for word in text if not word in set(stopwords.words('english'))]\n",
        "    text = ' '.join(text)\n",
        "    corpus.append(text)"
      ],
      "id": "44zeRoprHIgz",
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mhbfbt09KeUz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e4ec86c-e0d1-4d53-e1aa-1e910d4b2747"
      },
      "source": [
        "corpus"
      ],
      "id": "Mhbfbt09KeUz",
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['three vision india',\n",
              " 'year histori peopl world come invad us captur land conquer mind',\n",
              " 'alexand onward greek turk mogul portugues british french dutch came loot us took',\n",
              " 'yet done nation',\n",
              " 'conquer anyon',\n",
              " 'grab land cultur histori tri enforc way life',\n",
              " '',\n",
              " 'respect freedom other first vision freedom',\n",
              " 'believ india got first vision start war independ',\n",
              " 'freedom must protect nurtur build',\n",
              " 'free one respect us',\n",
              " 'second vision india develop',\n",
              " 'fifti year develop nation',\n",
              " 'time see develop nation',\n",
              " 'among top nation world term gdp',\n",
              " 'percent growth rate area',\n",
              " 'poverti level fall',\n",
              " 'achiev global recognis today',\n",
              " 'yet lack self confid see develop nation self reliant self assur',\n",
              " 'incorrect',\n",
              " 'third vision',\n",
              " 'india must stand world',\n",
              " 'believ unless india stand world one respect us',\n",
              " 'strength respect strength',\n",
              " 'must strong militari power also econom power',\n",
              " 'must go hand hand',\n",
              " 'good fortun work three great mind',\n",
              " 'dr vikram sarabhai dept',\n",
              " 'space professor satish dhawan succeed dr brahm prakash father nuclear materi',\n",
              " 'lucki work three close consid great opportun life',\n",
              " 'see four mileston career']"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T7FJMGjOJzfD"
      },
      "source": [
        "# creating bag of words vector\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "cv = CountVectorizer(max_features = 100)\n",
        "X = cv.fit_transform(corpus).toarray()"
      ],
      "id": "T7FJMGjOJzfD",
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3AS88GmuKjk5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c663eb2b-debd-4351-b7b1-7a5ef4918a21"
      },
      "source": [
        "X"
      ],
      "id": "3AS88GmuKjk5",
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 1, 1, 0],\n",
              "       [0, 1, 0, ..., 0, 0, 0],\n",
              "       ...,\n",
              "       [0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 0, 0, 0],\n",
              "       [0, 0, 0, ..., 0, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yM-hb5C2KkBD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "da0c151a-ae11-45eb-a566-810b8b7681c7"
      },
      "source": [
        "X.shape"
      ],
      "id": "yM-hb5C2KkBD",
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(31, 100)"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L-uNNzewCxsE"
      },
      "source": [
        ""
      ],
      "id": "L-uNNzewCxsE",
      "execution_count": null,
      "outputs": []
    }
  ]
}