05_Text_Parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "Natural Language Processing (NLP) has gained a lot of traction as a sub-field of Artificial Intelligence. It is focused on enabling computers to understand and process human languages. Some common applications include Chatbots, Sentiment Analysis, Translation, Spam Classification, and many more.\n",
    "\n",
    "However, there is a significant difference between NLP and traditional machine learning tasks, with the former dealing with unstructured text data while the latter deals with structured tabular data. Hence, it is necessary to understand how to deal with text before applying machine learning techniques to it. This is where text parsing comes into the picture.\n",
    "\n",
    "So, what is text parsing? In simple terms, it is a common programming task that separates the given series of text into smaller components based on some rules. Its application ranges from document parsing to deep learning NLP.\n",
    "\n",
    "In this guide, we will be applying the rich functionalities available within python to do text parsing. The two popular options are regular expressions and word tokenization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regular Expressions\n",
    "\n",
    "Regular Expressions, or Regex, are strings with a special syntax that allow us to match patterns in other strings. In python, there is a module called re to work with regular expressions. Some of the common regex patterns and its usage are shown below:\n",
    "\n",
    "    '\\d': matches any decimal digit; for example, 5. A variance of this is '\\D' that matches any non-digit character.\n",
    "\n",
    "    '\\s': matches any whitespace character; for example, ''. A variance of this is '\\S' that matches any non-whitespace character.\n",
    "\n",
    "    '\\w': matches any alphanumeric character; for example, 'Pluralsight'. A variance of this is '\\W' that matches any non-alphanumeric character.\n",
    "\n",
    "    '+ or *': performs a greedy match. For example, 'eeeeee'.\n",
    "\n",
    "    ‘a-z’: matches lowercase groups.\n",
    "\n",
    "    ‘A-Za-z’: matches upper and lowercase English alphabet.\n",
    "\n",
    "    ‘0-9’: matches numbers from 0 to 9.\n",
    "\n",
    "These are a few of the many regex patterns available. We will understand regex better in the subsequent sections of this guide with the help of examples. We will start by importing the re module which is done in the first line of code below. We will also need a text object or corpus, for which we use the brief description of the popular movie, 'Avengers'. We store this text in the corpus 'regex_example'. This is done in the second line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Avengers: Infinity War was a 2018 American superhero film based on the Marvel Comics superhero team the Avengers. It is the 19th film in the Marvel Cinematic Universe (MCU). The running time of the movie was 149 minutes and the box office collection was around 2 billion dollars. (Source: Wikipedia)\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "regex_example = \"Avengers: Infinity War was a 2018 American superhero film based on the Marvel Comics superhero team the Avengers. It is the 19th film in the Marvel Cinematic Universe (MCU). The running time of the movie was 149 minutes and the box office collection was around 2 billion dollars. (Source: Wikipedia)\"\n",
    "\n",
    "print(regex_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common Python Regex Methods\n",
    "#### re.findall()\n",
    "\n",
    "The re.findall() method returns all patterns in the string.In case of no match, an empty list is returned.\n",
    "\n",
    "The first line of code below extracts numbers from the text, 'regex_example', that we created earlier.\n",
    "\n",
    "We will now work with words and try to find the number of vowels in the text. The second and third lines of code perform this task. There are 87 occurrences of vowels in the text.\n",
    "\n",
    "Suppose you want to find the number of times the word 'Avengers' was used in the corpus. This is achieved by the fourth line of code and the answer is 2.\n",
    "\n",
    "We can also find all capitalized words and print the result, which is done in fifth and sixth lines of the code. The output only contains the capitalized words of the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2018', '19', '149', '2']\n",
      "87\n",
      "2\n",
      "['Avengers', 'Infinity', 'War', 'American', 'Marvel', 'Comics', 'Avengers', 'It', 'Marvel', 'Cinematic', 'Universe', 'MCU', 'The', 'Source', 'Wikipedia']\n"
     ]
    }
   ],
   "source": [
    "print(re.findall('\\d+', regex_example))  #line1\n",
    "\n",
    " \n",
    "\n",
    "vowels = re.findall('[aeiou]', regex_example)   #line2\n",
    "\n",
    "print(len(vowels))   #line 3\n",
    "\n",
    " \n",
    "\n",
    "print(len(re.findall('Avengers', regex_example))) #line 4\n",
    "\n",
    " \n",
    "\n",
    "capitalwords = \"[A-Z]\\w+\" \t#line 5\n",
    "\n",
    "print(re.findall(capitalwords, regex_example))   #line 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### re.split()\n",
    "\n",
    "The other useful method is re.split(), that splits the string in case of a match. In case of no match, it returns a list containing an empty string.\n",
    "\n",
    "In our example, let's apply this method and split the corpus with a pattern of numbers. The below chunk of code does this task and prints the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Avengers: Infinity War was a ', ' American superhero film based on the Marvel Comics superhero team the Avengers. It is the ', 'th film in the Marvel Cinematic Universe (MCU). The running time of the movie was ', ' minutes and the box office collection was around ', ' billion dollars. (Source: Wikipedia)']\n"
     ]
    }
   ],
   "source": [
    "print(re.split(r\"\\d+\", regex_example))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above output shows the split was done on digits. It is possible to add the ‘maxsplit’ argument to the re.split() method, that indicates the maximum number of splits that will occur. The default value is zero. The code below uses the maxsplit value as 2, and the method only splits for the first couple of digit occurrences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Avengers: Infinity War was a ', ' American superhero film based on the Marvel Comics superhero team the Avengers. It is the ', 'th film in the Marvel Cinematic Universe (MCU). The running time of the movie was 149 minutes and the box office collection was around 2 billion dollars. (Source: Wikipedia)']\n"
     ]
    }
   ],
   "source": [
    "print(re.split(r\"\\d+\", regex_example,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another application would be to split the corpus on spaces. That is achieved with the below code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Avengers:', 'Infinity', 'War', 'was', 'a', '2018', 'American', 'superhero', 'film', 'based', 'on', 'the', 'Marvel', 'Comics', 'superhero', 'team', 'the', 'Avengers.', 'It', 'is', 'the', '19th', 'film', 'in', 'the', 'Marvel', 'Cinematic', 'Universe', '(MCU).', 'The', 'running', 'time', 'of', 'the', 'movie', 'was', '149', 'minutes', 'and', 'the', 'box', 'office', 'collection', 'was', 'around', '2', 'billion', 'dollars.', '(Source:', 'Wikipedia)']\n"
     ]
    }
   ],
   "source": [
    "print(re.split(r\"\\s+\", regex_example))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### re.sub()\n",
    "\n",
    "This method is used to substitute the matched text with the contents in the replace variable. If the pattern is not found, the original string is returned.\n",
    "\n",
    "In our example, let's substitute the word 'Avengers' with 'A'. The below chunk of code does this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A: Infinity War was a 2018 American superhero film based on the Marvel Comics superhero team the A. It is the 19th film in the Marvel Cinematic Universe (MCU). The running time of the movie was 149 minutes and the box office collection was around 2 billion dollars. (Source: Wikipedia)\n"
     ]
    }
   ],
   "source": [
    "print(re.sub(\"Avengers\", \"A\", regex_example))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### re.search()\n",
    "\n",
    "This method is used to search the pattern in the string and return the match object if successful. If the search fails, it returns None.\n",
    "\n",
    "Let's understand this in the example below, where we will try to find if the word 'Python' is at the beginning of the sentence \"Scikit Learn is a great Python library\". In our example, since the search fails, the output is 'None'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "example = \"Scikit Learn is a great Python library\"\n",
    "\n",
    "match = re.search('\\APython', example)\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions are a humongous area and it is impossible to cover all of it in one guide. However, we have got a basic understanding of the most commonly used regex methods and their working in Python, which will be useful in the majority of the cases. Let’s now turn towards another important text parsing technique called word tokenization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word Tokenization\n",
    "\n",
    "Tokenization is the process of converting a text or corpus into tokens (smaller pieces). The conversion into tokens is carried out based on certain rules. Using regex, own rules can also be created. Tokenization helps in text pre-processing tasks such as mapping parts of speech, finding and matching common words, cleaning text, and getting the data ready for advanced text analytics techniques like sentiment analysis.\n",
    "\n",
    "Python has a very popular natural language toolkit library, called 'nltk', that has a rich set of functions for performing many NLP jobs. It can be downloaded through pip and is also included in the Anaconda distribution.\n",
    "\n",
    "We will be working on the common applications of word tokenization using nltk. To start with, we should import the nltk library. Then, we import the sent_tokenize and word_tokenize functions. This is done in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "\n",
    "import nltk\n",
    "\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pluralsight is the technology skills platform. It has more than 6000 courses and 1100+ employees. Pluralsight is headquartered in Utah and has over 1500 experts authoring high quality courses. Pluralsight grew 161 percent between 2014 and 2017, earning the company a spot in the Deloitte Technology Fast 500 list for five consecutive years.\n"
     ]
    }
   ],
   "source": [
    "textdata = \"Pluralsight is the technology skills platform. It has more than 6000 courses and 1100+ employees. Pluralsight is headquartered in Utah and has over 1500 experts authoring high quality courses. Pluralsight grew 161 percent between 2014 and 2017, earning the company a spot in the Deloitte Technology Fast 500 list for five consecutive years.\"\n",
    "\n",
    "print(textdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now perform word tokenization tasks on this textdata. The most common forms of tokenization are word and sentence tokenization.\n",
    "\n",
    "The first code below tokenizes the text into words and prints the output. The second code also does tokenization, but this time it's done on sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pluralsight', 'is', 'the', 'technology', 'skills', 'platform', '.', 'It', 'has', 'more', 'than', '6000', 'courses', 'and', '1100+', 'employees', '.', 'Pluralsight', 'is', 'headquartered', 'in', 'Utah', 'and', 'has', 'over', '1500', 'experts', 'authoring', 'high', 'quality', 'courses', '.', 'Pluralsight', 'grew', '161', 'percent', 'between', '2014', 'and', '2017', ',', 'earning', 'the', 'company', 'a', 'spot', 'in', 'the', 'Deloitte', 'Technology', 'Fast', '500', 'list', 'for', 'five', 'consecutive', 'years', '.']\n"
     ]
    }
   ],
   "source": [
    "print(word_tokenize(textdata))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pluralsight is the technology skills platform.', 'It has more than 6000 courses and 1100+ employees.', 'Pluralsight is headquartered in Utah and has over 1500 experts authoring high quality courses.', 'Pluralsight grew 161 percent between 2014 and 2017, earning the company a spot in the Deloitte Technology Fast 500 list for five consecutive years.']\n"
     ]
    }
   ],
   "source": [
    "print(sent_tokenize(textdata))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to understand the difference between the two functions 'word_tokenize' and 'sent_tokenize'. In the above outputs, we saw that both the functions create different outputs (tokens) for the same input (text data), and that's because of the difference in their functionalities.\n",
    "\n",
    "We will now look at the unique tokens generated with both the methods in the two lines of code below. In the first output below, the term 'Pluralsight' appears only once as it's a word-level token. However, in the second output, the term appears many times because the tokenization happened at the sentence level. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'high', 'quality', 'headquartered', ',', 'technology', 'grew', 'authoring', 'Pluralsight', 'spot', '1500', 'Technology', 'list', 'experts', '.', 'Utah', 'Deloitte', 'company', 'than', '2017', 'has', 'and', '500', 'years', 'a', 'courses', 'between', 'It', 'five', 'over', 'more', 'percent', 'for', '2014', 'earning', '161', 'employees', 'skills', 'platform', 'Fast', 'is', 'consecutive', '6000', '1100+', 'the', 'in'}\n"
     ]
    }
   ],
   "source": [
    "print(set(word_tokenize(textdata)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Pluralsight is headquartered in Utah and has over 1500 experts authoring high quality courses.', 'Pluralsight grew 161 percent between 2014 and 2017, earning the company a spot in the Deloitte Technology Fast 500 list for five consecutive years.', 'It has more than 6000 courses and 1100+ employees.', 'Pluralsight is the technology skills platform.'}\n"
     ]
    }
   ],
   "source": [
    "print(set(sent_tokenize(textdata)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the number of unique tokens generated using both the functions, 'word_tokenize' and 'sent_tokenize', which is done in the two lines of code below. There are 45 unique 'word' tokens, while there are only 4 unique 'sentence' tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "print(len(set(word_tokenize(textdata))))\n",
    "\n",
    "print(len(set(sent_tokenize(textdata))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a fair understanding of tokenization. We will now add some visualization to tokenization through the powerful python library, matplotlib. The first line of code imports the matplotlib library, while the second line creates word tokens and stores it in the object 't'. The third line calculates the number of occurrences of each word in the corpus. The fourth and fifth lines of code plot the histogram. We can see that there are many words which have been repeated quite a few times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZ4UlEQVR4nO3de2yV9f3A8U+lWhhpq2Xj0kilM2QoIOJQo5gNIpEQRMmiToNIMNFdqoAsDroNHCpU3H6GqQTEZMIS8fKHoMOoYQxhZnKtGM02LhGxkQBbpi2X0JH2/P5YbFK5SN3p95wDr1fy/HGe5+n5fvKEyzvPOaenKJPJZAIAIJFzcj0AAHB2ER8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJBUca4H+LLW1tbYu3dvlJaWRlFRUa7HAQBOQyaTiYMHD0ZlZWWcc86p723kXXzs3bs3+vbtm+sxAICvoaGhIS688MJTnpN38VFaWhoR/x2+rKwsx9MAAKejqakp+vbt2/b/+KnkXXx88VJLWVmZ+ACAAnM6b5nwhlMAICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJNXh+Fi/fn2MGzcuKisro6ioKFauXHnSc3/84x9HUVFRLFiw4H8YEQA4k3Q4Pg4fPhxDhgyJhQsXnvK8FStWxIYNG6KysvJrDwcAnHk6/MVyY8aMiTFjxpzynE8//TTuv//+eOutt2Ls2LFfezgA4MyT9W+1bW1tjYkTJ8aDDz4YAwcO/Mrzm5ubo7m5ue1xU1NTtkcCAPJI1uNj/vz5UVxcHFOmTDmt8+vq6mLOnDnZHoMc6zfz9VyP0GEfP+YuHUAKWf20y9atW+N3v/tdLF26NIqKik7rZ2pra6OxsbFta2hoyOZIAECeyWp8/OUvf4kDBw5EVVVVFBcXR3FxcezZsyd+9rOfRb9+/U74MyUlJVFWVtZuAwDOXFl92WXixIkxatSodvtGjx4dEydOjMmTJ2dzKQCgQHU4Pg4dOhS7du1qe7x79+7Ytm1bVFRURFVVVfTo0aPd+eeee2707t07vvOd7/zv0wIABa/D8bFly5YYOXJk2+Pp06dHRMSkSZNi6dKlWRsMADgzdTg+RowYEZlM5rTP//jjjzu6BABwBvPdLgBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJBUh+Nj/fr1MW7cuKisrIyioqJYuXJl27Fjx47FjBkzYvDgwdG9e/eorKyMu+66K/bu3ZvNmQGAAtbh+Dh8+HAMGTIkFi5ceNyxI0eORH19fcyaNSvq6+vjlVdeie3bt8dNN92UlWEBgMJX3NEfGDNmTIwZM+aEx8rLy2P16tXt9j399NNx1VVXxSeffBJVVVVfb0oA4IzR6e/5aGxsjKKiojj//PM7eykAoAB0+M5HRxw9ejRmzJgRd9xxR5SVlZ3wnObm5mhubm573NTU1JkjAQA51ml3Po4dOxa33XZbZDKZWLRo0UnPq6uri/Ly8ratb9++nTUSAJAHOiU+vgiPPXv2xOrVq0961yMiora2NhobG9u2hoaGzhgJAMgTWX/Z5Yvw2LlzZ6xduzZ69OhxyvNLSkqipKQk22MAAHmqw/Fx6NCh2LVrV9vj3bt3x7Zt26KioiL69OkTt9xyS9TX18eqVauipaUl9u3bFxERFRUVcd5552VvcgCgIHU4PrZs2RIjR45sezx9+vSIiJg0aVL8+te/jtdeey0iIi6//PJ2P7d27doYMWLE158UADgjdDg+RowYEZlM5qTHT3UMAMB3uwAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkVZzrAVLrN/P1XI8AAGc1dz4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBSHY6P9evXx7hx46KysjKKiopi5cqV7Y5nMpmYPXt29OnTJ7p16xajRo2KnTt3ZmteAKDAdTg+Dh8+HEOGDImFCxee8Pjjjz8eTz75ZCxevDg2btwY3bt3j9GjR8fRo0f/52EBgMJX3NEfGDNmTIwZM+aExzKZTCxYsCB+9atfxc033xwREX/4wx+iV69esXLlyrj99tv/t2kBgIKX1fd87N69O/bt2xejRo1q21deXh5XX311vPvuuyf8mebm5mhqamq3AQBnrg7f+TiVffv2RUREr1692u3v1atX27Evq6urizlz5mRzDCCP9Zv5eq5H6LCPHxub6xHgjJLzT7vU1tZGY2Nj29bQ0JDrkQCATpTV+Ojdu3dEROzfv7/d/v3797cd+7KSkpIoKytrtwEAZ66sxkd1dXX07t071qxZ07avqakpNm7cGNdcc002lwIAClSH3/Nx6NCh2LVrV9vj3bt3x7Zt26KioiKqqqpi2rRp8eijj0b//v2juro6Zs2aFZWVlTF+/Phszg0AFKgOx8eWLVti5MiRbY+nT58eERGTJk2KpUuXxs9//vM4fPhw3HvvvfH555/HddddF2+++WZ07do1e1MDAAWrw/ExYsSIyGQyJz1eVFQUDz/8cDz88MP/02AAwJkp5592AQDOLuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJBU1uOjpaUlZs2aFdXV1dGtW7e4+OKL45FHHolMJpPtpQCAAlSc7SecP39+LFq0KJYtWxYDBw6MLVu2xOTJk6O8vDymTJmS7eUAgAKT9fj461//GjfffHOMHTs2IiL69esXL7zwQmzatCnbSwEABSjrL7tce+21sWbNmtixY0dERLz//vvxzjvvxJgxY054fnNzczQ1NbXbAIAzV9bvfMycOTOamppiwIAB0aVLl2hpaYm5c+fGhAkTTnh+XV1dzJkzJ9tjAAB5Kut3Pl5++eV4/vnnY/ny5VFfXx/Lli2L3/72t7Fs2bITnl9bWxuNjY1tW0NDQ7ZHAgDySNbvfDz44IMxc+bMuP322yMiYvDgwbFnz56oq6uLSZMmHXd+SUlJlJSUZHsMACBPZf3Ox5EjR+Kcc9o/bZcuXaK1tTXbSwEABSjrdz7GjRsXc+fOjaqqqhg4cGC899578cQTT8Tdd9+d7aUAgAKU9fh46qmnYtasWfHTn/40Dhw4EJWVlfGjH/0oZs+ene2lAIAClPX4KC0tjQULFsSCBQuy/dQAwBnAd7sAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUp0SH59++mnceeed0aNHj+jWrVsMHjw4tmzZ0hlLAQAFpjjbT/jZZ5/F8OHDY+TIkfHGG2/Et771rdi5c2dccMEF2V4KAChAWY+P+fPnR9++feO5555r21ddXZ3tZQCAApX1l11ee+21GDZsWNx6663Rs2fPGDp0aDz77LMnPb+5uTmamprabQDAmSvr8fHRRx/FokWLon///vHWW2/FT37yk5gyZUosW7bshOfX1dVFeXl529a3b99sjwQA5JGsx0dra2tcccUVMW/evBg6dGjce++9cc8998TixYtPeH5tbW00Nja2bQ0NDdkeCQDII1mPjz59+sSll17abt8ll1wSn3zyyQnPLykpibKysnYbAHDmynp8DB8+PLZv395u344dO+Kiiy7K9lIAQAHKenw88MADsWHDhpg3b17s2rUrli9fHkuWLImamppsLwUAFKCsx8eVV14ZK1asiBdeeCEGDRoUjzzySCxYsCAmTJiQ7aUAgAKU9d/zERFx4403xo033tgZTw0AFDjf7QIAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AIKlOj4/HHnssioqKYtq0aZ29FABQADo1PjZv3hzPPPNMXHbZZZ25DABQQDotPg4dOhQTJkyIZ599Ni644ILOWgYAKDCdFh81NTUxduzYGDVq1CnPa25ujqampnYbAHDmKu6MJ33xxRejvr4+Nm/e/JXn1tXVxZw5czpjDAAKTL+Zr+d6hA77+LGxuR6h4GT9zkdDQ0NMnTo1nn/++ejatetXnl9bWxuNjY1tW0NDQ7ZHAgDySNbvfGzdujUOHDgQV1xxRdu+lpaWWL9+fTz99NPR3NwcXbp0aTtWUlISJSUl2R4DAMhTWY+P66+/Pj744IN2+yZPnhwDBgyIGTNmtAsPAODsk/X4KC0tjUGDBrXb17179+jRo8dx+wGAs4/fcAoAJNUpn3b5srfffjvFMgBAAXDnAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJLKenzU1dXFlVdeGaWlpdGzZ88YP358bN++PdvLAAAFKuvxsW7duqipqYkNGzbE6tWr49ixY3HDDTfE4cOHs70UAFCAirP9hG+++Wa7x0uXLo2ePXvG1q1b43vf+162lwMACkzW4+PLGhsbIyKioqLihMebm5ujubm57XFTU1NnjwQA5FCnxkdra2tMmzYthg8fHoMGDTrhOXV1dTFnzpzOHANOS7+Zr+d6hA77+LGxuR7hrFCIfzYgn3Xqp11qamriww8/jBdffPGk59TW1kZjY2Pb1tDQ0JkjAQA51ml3Pu67775YtWpVrF+/Pi688MKTnldSUhIlJSWdNQYAkGeyHh+ZTCbuv//+WLFiRbz99ttRXV2d7SUAgAKW9fioqamJ5cuXx6uvvhqlpaWxb9++iIgoLy+Pbt26ZXs5AKDAZP09H4sWLYrGxsYYMWJE9OnTp2176aWXsr0UAFCAOuVlFwCAk/HdLgBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASKo41wMAX1+/ma/negQ46xXi38OPHxub0/Xd+QAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEl1WnwsXLgw+vXrF127do2rr746Nm3a1FlLAQAFpFPi46WXXorp06fHQw89FPX19TFkyJAYPXp0HDhwoDOWAwAKSKfExxNPPBH33HNPTJ48OS699NJYvHhxfOMb34jf//73nbEcAFBAirP9hP/5z39i69atUVtb27bvnHPOiVGjRsW777573PnNzc3R3Nzc9rixsTEiIpqamrI9WkREtDYf6ZTnBYBC0Rn/x37xnJlM5ivPzXp8/Otf/4qWlpbo1atXu/29evWKf/zjH8edX1dXF3PmzDluf9++fbM9GgAQEeULOu+5Dx48GOXl5ac8J+vx0VG1tbUxffr0tsetra3x73//O3r06BFFRUU5nCyNpqam6Nu3bzQ0NERZWVmux8l7rtfpc61On2t1+lyr03e2XatMJhMHDx6MysrKrzw36/HxzW9+M7p06RL79+9vt3///v3Ru3fv484vKSmJkpKSdvvOP//8bI+V98rKys6KP5zZ4nqdPtfq9LlWp8+1On1n07X6qjseX8j6G07PO++8+O53vxtr1qxp29fa2hpr1qyJa665JtvLAQAFplNedpk+fXpMmjQphg0bFldddVUsWLAgDh8+HJMnT+6M5QCAAtIp8fHDH/4w/vnPf8bs2bNj3759cfnll8ebb7553JtQ+e/LTg899NBxLz1xYq7X6XOtTp9rdfpcq9PnWp1cUeZ0PhMDAJAlvtsFAEhKfAAASYkPACAp8QEAJCU+cqSuri6uvPLKKC0tjZ49e8b48eNj+/btuR6rIDz22GNRVFQU06ZNy/UoeenTTz+NO++8M3r06BHdunWLwYMHx5YtW3I9Vt5paWmJWbNmRXV1dXTr1i0uvvjieOSRR07reynOBuvXr49x48ZFZWVlFBUVxcqVK9sdz2QyMXv27OjTp09069YtRo0aFTt37szNsDl2qmt17NixmDFjRgwePDi6d+8elZWVcdddd8XevXtzN3AeEB85sm7duqipqYkNGzbE6tWr49ixY3HDDTfE4cOHcz1aXtu8eXM888wzcdlll+V6lLz02WefxfDhw+Pcc8+NN954I/72t7/F//3f/8UFF1yQ69Hyzvz582PRokXx9NNPx9///veYP39+PP744/HUU0/lerS8cPjw4RgyZEgsXLjwhMcff/zxePLJJ2Px4sWxcePG6N69e4wePTqOHj2aeNLcO9W1OnLkSNTX18esWbOivr4+Xnnlldi+fXvcdNNNOZg0j2TICwcOHMhERGbdunW5HiVvHTx4MNO/f//M6tWrM9///vczU6dOzfVIeWfGjBmZ6667LtdjFISxY8dm7r777nb7fvCDH2QmTJiQo4nyV0RkVqxY0fa4tbU107t378xvfvObtn2ff/55pqSkJPPCCy/kYML88eVrdSKbNm3KRERmz549aYbKQ+585InGxsaIiKioqMjxJPmrpqYmxo4dG6NGjcr1KHnrtddei2HDhsWtt94aPXv2jKFDh8azzz6b67Hy0rXXXhtr1qyJHTt2RETE+++/H++8806MGTMmx5Plv927d8e+ffva/V0sLy+Pq6++Ot59990cTlYYGhsbo6io6Kz8HrMv5Pxbbfnvd99MmzYthg8fHoMGDcr1OHnpxRdfjPr6+ti8eXOuR8lrH330USxatCimT58ev/jFL2Lz5s0xZcqUOO+882LSpEm5Hi+vzJw5M5qammLAgAHRpUuXaGlpiblz58aECRNyPVre27dvX0TEcb+1ulevXm3HOLGjR4/GjBkz4o477jhrvmzuRMRHHqipqYkPP/ww3nnnnVyPkpcaGhpi6tSpsXr16ujatWuux8lrra2tMWzYsJg3b15ERAwdOjQ+/PDDWLx4sfj4kpdffjmef/75WL58eQwcODC2bdsW06ZNi8rKSteKTnHs2LG47bbbIpPJxKJFi3I9Tk552SXH7rvvvli1alWsXbs2LrzwwlyPk5e2bt0aBw4ciCuuuCKKi4ujuLg41q1bF08++WQUFxdHS0tLrkfMG3369IlLL7203b5LLrkkPvnkkxxNlL8efPDBmDlzZtx+++0xePDgmDhxYjzwwANRV1eX69HyXu/evSMiYv/+/e3279+/v+0Y7X0RHnv27InVq1ef1Xc9IsRHzmQymbjvvvtixYoV8ec//zmqq6tzPVLeuv766+ODDz6Ibdu2tW3Dhg2LCRMmxLZt26JLly65HjFvDB8+/LiPbO/YsSMuuuiiHE2Uv44cORLnnNP+n8AuXbpEa2trjiYqHNXV1dG7d+9Ys2ZN276mpqbYuHFjXHPNNTmcLD99ER47d+6MP/3pT9GjR49cj5RzXnbJkZqamli+fHm8+uqrUVpa2vY6aXl5eXTr1i3H0+WX0tLS494L07179+jRo4f3yHzJAw88ENdee23Mmzcvbrvttti0aVMsWbIklixZkuvR8s64ceNi7ty5UVVVFQMHDoz33nsvnnjiibj77rtzPVpeOHToUOzatavt8e7du2Pbtm1RUVERVVVVMW3atHj00Uejf//+UV1dHbNmzYrKysoYP3587obOkVNdqz59+sQtt9wS9fX1sWrVqmhpaWn7976ioiLOO++8XI2dW7n+uM3ZKiJOuD333HO5Hq0g+Kjtyf3xj3/MDBo0KFNSUpIZMGBAZsmSJbkeKS81NTVlpk6dmqmqqsp07do18+1vfzvzy1/+MtPc3Jzr0fLC2rVrT/hv1KRJkzKZzH8/bjtr1qxMr169MiUlJZnrr78+s3379twOnSOnula7d+8+6b/3a9euzfXoOVOUyfh1fgBAOt7zAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACS+n+Q4yHfN0XDhQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Combining NLP data extraction with plotting\n",
    "\n",
    "from matplotlib import pyplot as plt  #line 1\n",
    "\n",
    "t = word_tokenize(textdata)\n",
    "\n",
    "wordlen = [len(w) for w in t]\n",
    "\n",
    "plt.hist(wordlen)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In this guide, we have come a long way from understanding basic regular expressions to tokenizing words and sentences. We learned the usage of python's two powerful libraries, re and nltk, using interesting text examples. We also learned how to create visualizations of word tokens using nltk and matplotlib.\n",
    "\n",
    "Both Regex and NLTK can play a vital role in the text pre-processing phase. However, the domain of these techniques is too big to be covered in this one single guide, which was aimed to be a good building block for individuals aspiring to start working with natural language processing problems using Python’s inbuilt libraries."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "63f9911de6f9547d26d6ccd5b997d4c24c9ba8fc8843a26db797dceee8b60692"
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit (windows store)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}