_notebooks/2020-07-15-01-Regular-expressions-and-word-tokenization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular expressions and word tokenization\n",
    "> This chapter will introduce some basic NLP concepts, such as word tokenization and regular expressions to help parse text. You'll also learn how to handle non-English text and more difficult tokenization you might find. This is the Summary of lecture \"Introduction to Natural Language Processing in Python\", via datacamp.\n",
    "\n",
    "- toc: true \n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Chanseok Kang\n",
    "- categories: [Python, Datacamp, Natural_Language_Processing]\n",
    "- image: images/line_num_words.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to regular expressions\n",
    "- Regular expressions\n",
    "    - Strings with a special syntax\n",
    "    - Allow us to match patterns in other strings\n",
    "    - Applications of regular expressions\n",
    "        - Find all web links in a document\n",
    "        - Parse email addresses, remove/replace unwanted characters\n",
    "- Common Regex patterns\n",
    "\n",
    "| pattern | matches | examples |\n",
    "| ------- | ------- | -------- |\n",
    "| \\w+ | word | 'Magic' |\n",
    "| \\d | digit | 9 |\n",
    "| \\s | space | '' |\n",
    "| .* | wildcard | 'username74' |\n",
    "| + or * | greedy match | 'aaaaaaa' |\n",
    "| \\S | not space | 'no_spaces' |\n",
    "| [a-z] | lowercase group | 'abcdfg' |\n",
    "\n",
    "- Python's re Module\n",
    "    - `split`: split a string on regex\n",
    "    - `findall`: find all patterns in a string\n",
    "    - `search`: search for a pattern\n",
    "    - `match`: match an entire string or substring based on a pattern\n",
    "    - Pattern first, and the string second\n",
    "    - May return an iterator, string, or match object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Let', 's', 'write', 'RegEx']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_string = \"Let's write RegEx!\"\n",
    "PATTERN = r\"\\w+\"\n",
    "re.findall(PATTERN, my_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practicing regular expressions - re.split() and re.findall()\n",
    "Now you'll get a chance to write some regular expressions to match digits, strings and non-alphanumeric characters. Take a look at `my_string` first by printing it in the IPython Shell, to determine how you might best match the different steps.\n",
    "\n",
    "Note: It's important to prefix your regex patterns with `r` to ensure that your patterns are interpreted in the way you want them to. Else, you may encounter problems to do with escape sequences in strings. For example, `\"\\n\"` in Python is used to indicate a new line, but if you use the `r` prefix, it will be interpreted as the raw string `\"\\n\"` - that is, the character `\"\\\"` followed by the character `\"n\"` - and not as a new line.\n",
    "\n",
    "Remember from the video that the syntax for the regex library is to always to pass the **pattern first**, and then the **string second**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_string = \"Let's write RegEx!  Won't that be fun?  I sure think so.  Can you find 4 sentences?  Or perhaps, all 19 words?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Let's write RegEx\", \"  Won't that be fun\", '  I sure think so', '  Can you find 4 sentences', '  Or perhaps, all 19 words', '']\n",
      "['Let', 'RegEx', 'Won', 'Can', 'Or']\n",
      "[\"Let's\", 'write', 'RegEx!', \"Won't\", 'that', 'be', 'fun?', 'I', 'sure', 'think', 'so.', 'Can', 'you', 'find', '4', 'sentences?', 'Or', 'perhaps,', 'all', '19', 'words?']\n",
      "['4', '19']\n"
     ]
    }
   ],
   "source": [
    "# Write a pattern to match sentence endings: sentence_endings\n",
    "sentence_endings = r\"[.?!]\"\n",
    "\n",
    "# Split my_string on sentence endings and print the result\n",
    "print(re.split(sentence_endings, my_string))\n",
    "\n",
    "# Find all capicalized words in my_string and print the result\n",
    "capitalized_words = r\"[A-Z]\\w+\"\n",
    "print(re.findall(capitalized_words, my_string))\n",
    "\n",
    "# Split my_string on spaces and print the result\n",
    "spaces = r\"\\s+\"\n",
    "print(re.split(spaces, my_string))\n",
    "\n",
    "# Find all digits in my_string and print the result\n",
    "digits = r\"\\d+\"\n",
    "print(re.findall(digits, my_string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to tokenization\n",
    "- Tokenization\n",
    "    - Turning a string or document into **tokens** (smaller chunks)\n",
    "    - One step in preparing a text for NLP\n",
    "    - Many different theories and rules\n",
    "    - You can create your own rules using regular expressions\n",
    "    - Some examples:\n",
    "        - Breaking out words or sentences\n",
    "        - Separating punctuation\n",
    "        - Separating all hashtags in a tweet\n",
    "- Why tokenize?\n",
    "    - Easier to map part of speech\n",
    "    - Matching common words\n",
    "    - Removing unwanted tokens\n",
    "- Other `nltk` tokenizers\n",
    "    - `sent_tokenize`: tokenize a document into sentences\n",
    "    - `regexp_tokenize`: tokenize a string or document based on a regular expression pattern\n",
    "    - `TweetTokenizer`: special class just for tweet tokenization, allowing you to separate hashtags, mentions and lots of exclamation points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word tokenization with NLTK\n",
    "Here, you'll be using the first scene of Monty Python's Holy Grail, which has been pre-loaded as `scene_one`.\n",
    "\n",
    "Your job in this exercise is to utilize `word_tokenize` and `sent_tokenize` from `nltk.tokenize` to tokenize both words and sentences from Python strings - in this case, the first scene of Monty Python's Holy Grail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: Before using NLTK, you must install `punkt` package for tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./dataset/grail.txt', 'r') as file:\n",
    "    holy_grail = file.read()\n",
    "    scene_one = re.split('SCENE 2:', holy_grail)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"SCENE 1: [wind] [clop clop clop] \\nKING ARTHUR: Whoa there!  [clop clop clop] \\nSOLDIER #1: Halt!  Who goes there?\\nARTHUR: It is I, Arthur, son of Uther Pendragon, from the castle of Camelot.  King of the Britons, defeator of the Saxons, sovereign of all England!\\nSOLDIER #1: Pull the other one!\\nARTHUR: I am, ...  and this is my trusty servant Patsy.  We have ridden the length and breadth of the land in search of knights who will join me in my court at Camelot.  I must speak with your lord and master.\\nSOLDIER #1: What?  Ridden on a horse?\\nARTHUR: Yes!\\nSOLDIER #1: You're using coconuts!\\nARTHUR: What?\\nSOLDIER #1: You've got two empty halves of coconut and you're bangin' 'em together.\\nARTHUR: So?  We have ridden since the snows of winter covered this land, through the kingdom of Mercea, through--\\nSOLDIER #1: Where'd you get the coconuts?\\nARTHUR: We found them.\\nSOLDIER #1: Found them?  In Mercea?  The coconut's tropical!\\nARTHUR: What do you mean?\\nSOLDIER #1: Well, this is a temperate zone.\\nARTHUR: The swallow may fly south with the sun or the house martin or the plover may seek warmer climes in winter, yet these are not strangers to our land?\\nSOLDIER #1: Are you suggesting coconuts migrate?\\nARTHUR: Not at all.  They could be carried.\\nSOLDIER #1: What?  A swallow carrying a coconut?\\nARTHUR: It could grip it by the husk!\\nSOLDIER #1: It's not a question of where he grips it!  It's a simple question of weight ratios!  A five ounce bird could not carry a one pound coconut.\\nARTHUR: Well, it doesn't matter.  Will you go and tell your master that Arthur from the Court of Camelot is here.\\nSOLDIER #1: Listen.  In order to maintain air-speed velocity, a swallow needs to beat its wings forty-three times every second, right?\\nARTHUR: Please!\\nSOLDIER #1: Am I right?\\nARTHUR: I'm not interested!\\nSOLDIER #2: It could be carried by an African swallow!\\nSOLDIER #1: Oh, yeah, an African swallow maybe, but not a European swallow.  That's my point.\\nSOLDIER #2: Oh, yeah, I agree with that.\\nARTHUR: Will you ask your master if he wants to join my court at Camelot?!\\nSOLDIER #1: But then of course a-- African swallows are non-migratory.\\nSOLDIER #2: Oh, yeah...\\nSOLDIER #1: So they couldn't bring a coconut back anyway...  [clop clop clop] \\nSOLDIER #2: Wait a minute!  Supposing two swallows carried it together?\\nSOLDIER #1: No, they'd have to have it on a line.\\nSOLDIER #2: Well, simple!  They'd just use a strand of creeper!\\nSOLDIER #1: What, held under the dorsal guiding feathers?\\nSOLDIER #2: Well, why not?\\n\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scene_one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'lord', 'bangin', 'strangers', 'swallows', 'them', 'right', 'KING', '!', 'goes', 'Ridden', 'Yes', 'snows', 'I', 'weight', 'matter', 'yeah', 'they', 'Supposing', 'where', 'all', \"'s\", 'simple', 'So', 'seek', 'line', 'who', 'swallow', 'beat', 'land', 'with', 'Mercea', 'from', 'get', 'No', 'one', 'second', 'sovereign', 'temperate', 'Oh', 'to', 'Camelot', ']', '1', 'bird', 'tell', 'an', 'are', 'our', 'servant', 'interested', '2', 'Patsy', 'am', \"n't\", 'got', 'must', 'England', 'Court', 'if', 'feathers', 'anyway', 'carrying', 'use', 'does', 'climes', 'back', 'carried', 'Wait', 'air-speed', 'velocity', '#', 'pound', 'on', \"'d\", 'found', 'European', 'of', 'every', 'ARTHUR', 'your', 'horse', 'these', 'not', 'that', 'You', 'since', 'times', 'agree', 'winter', 'yet', 'breadth', 'speak', 'carry', 'Well', 'wings', 'King', 'Listen', 'martin', 'kingdom', 'maybe', 'there', 'ask', 'master', 'creeper', 'will', 'maintain', 'ridden', 'here', 'then', 'the', 'Please', 'through', 'together', 'why', 'using', 'a', 'non-migratory', 'clop', 'grips', 'have', 'SOLDIER', 'you', \"'\", 'plover', ',', 'Saxons', 'just', 'question', 'may', 'two', 'south', 'join', 'mean', 'They', 'strand', 'Halt', 'this', 'What', 'son', '[', 'search', 'bring', 'Not', 'house', 'forty-three', 'minute', 'guiding', 'other', 'wind', \"'em\", 'Pendragon', 'five', 'by', 'Uther', 'castle', 'order', 'is', 'but', ':', 'wants', 'under', 'Are', 'knights', 'held', 'Am', \"'m\", 'he', 'That', 'Arthur', 'at', 'zone', 'my', 'covered', 'It', 'We', 'coconuts', 'coconut', 'Found', 'African', 'Will', 'Whoa', 'Pull', 'trusty', 'grip', 'sun', 'Who', 'halves', 'defeator', 'and', '?', 'me', 'The', 'it', 'ounce', 'course', 'dorsal', 'its', 'point', 'But', 'husk', 'in', '...', 'court', 'empty', 'A', 'length', '--', 'go', 'migrate', 'or', 'suggesting', 'Britons', 'tropical', 'be', \"'re\", 'Where', '.', 'do', 'fly', 'ratios', 'SCENE', 'warmer', 'In', 'could', \"'ve\", 'needs'}\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "\n",
    "# Split scene_one into sentences: sentences\n",
    "sentences = sent_tokenize(scene_one)\n",
    "\n",
    "# Use word_tokenize to tokenize the fourth sentence: tokenized_sent\n",
    "tokenized_sent = word_tokenize(sentences[3])\n",
    "\n",
    "# Make a set of unique tokens in the entire scene: unique_tokens\n",
    "unique_tokens = set(word_tokenize(scene_one))\n",
    "\n",
    "# Print the unique tokens result\n",
    "print(unique_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More regex with re.search()\n",
    "In this exercise, you'll utilize `re.search()` and `re.match()` to find specific tokens. Both search and match expect regex patterns, similar to those you defined in an earlier exercise. You'll apply these regex library methods to the same Monty Python text from the `nltk` corpora.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "580 588\n"
     ]
    }
   ],
   "source": [
    "# Search for the first occurrence of \"coconuts\" in scene_one: match\n",
    "match = re.search(\"coconuts\", scene_one)\n",
    "\n",
    "# Print the start and end indexes of match\n",
    "print(match.start(), match.end())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(9, 32), match='[wind] [clop clop clop]'>\n"
     ]
    }
   ],
   "source": [
    "# Write a regular expression to search for anything in square brackets: pattern1\n",
    "pattern1 = r\"\\[.*\\]\"\n",
    "\n",
    "# Use re.search to find the first text in square brackets\n",
    "print(re.search(pattern1, scene_one))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 7), match='ARTHUR:'>\n"
     ]
    }
   ],
   "source": [
    "# Find the script notation at the beginning of the fourth sentence and print it\n",
    "pattern2 = r\"[\\w\\s]+:\"\n",
    "print(re.match(pattern2, sentences[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced tokenization with NLTK and regex\n",
    "- Regex groups using or `|`\n",
    "    - OR is represented using `|`\n",
    "    - You can define a group using `()`\n",
    "    - You can define explicit character ranges using `[]`\n",
    "- Regex ranges and groups\n",
    "\n",
    "| pattern | matches | example |\n",
    "| ------- | ------- | ------- |\n",
    "| [A-Za-z]+ | upper and lowercase English alphabet | 'ABCDEFghijk' |\n",
    "| [0-9] | numbers from 0 to 9 | 9 |\n",
    "| [A-Za-z\\-\\.]+ | upper and lowercase English alphabet, - and . | 'My-Website.com' |\n",
    "| (a-z) | a, - and z | 'a-z' |\n",
    "| (\\s+|,) | spaces or a comma | ', ' |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing a tokenizer\n",
    "Given the following string, which of the below patterns is the best tokenizer? If possible, you want to retain sentence punctuation as separate tokens, but have `'#1'` remain a single token.\n",
    "```python\n",
    "my_string = \"SOLDIER #1: Found them? In Mercea? The coconut's tropical!\"\n",
    "```\n",
    "\n",
    "Additionally, `regexp_tokenize` has been imported from `nltk.tokenize`. You can use `regexp_tokenize(string, pattern)` with `my_string` and one of the patterns as arguments to experiment for yourself and see which is the best tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import regexp_tokenize\n",
    "\n",
    "my_string = \"SOLDIER #1: Found them? In Mercea? The coconut's tropical!\"\n",
    "\n",
    "pattern1 = r'(\\\\w+|\\\\?|!)'\n",
    "pattern2 = r\"(\\w+|#\\d|\\?|!)\"\n",
    "pattern3 = r'(#\\\\d\\\\w+\\\\?!)'\n",
    "pattern4 = r'\\\\s+'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['SOLDIER',\n",
      " '#1',\n",
      " 'Found',\n",
      " 'them',\n",
      " '?',\n",
      " 'In',\n",
      " 'Mercea',\n",
      " '?',\n",
      " 'The',\n",
      " 'coconut',\n",
      " 's',\n",
      " 'tropical',\n",
      " '!']\n"
     ]
    }
   ],
   "source": [
    "pprint(regexp_tokenize(my_string, pattern2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regex with NLTK tokenization\n",
    "Twitter is a frequently used source for NLP text and tasks. In this exercise, you'll build a more complex tokenizer for tweets with hashtags and mentions using nltk and regex. The `nltk.tokenize.TweetTokenizer` class gives you some extra methods and attributes for parsing tweets.\n",
    "\n",
    "Here, you're given some example tweets to parse using both `TweetTokenizer` and `regexp_tokenize` from the `nltk.tokenize` module. \n",
    "\n",
    "Unlike the syntax for the regex library, with `nltk_tokenize()` you pass the pattern as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets = ['This is the best #nlp exercise ive found online! #python',\n",
    " '#NLP is super fun! <3 #learning',\n",
    " 'Thanks @datacamp :) #nlp #python']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['#nlp', '#python']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import regexp_tokenize, TweetTokenizer\n",
    "\n",
    "# Define a regex pattern to find hashtags: pattern1\n",
    "pattern1 = r\"#\\w+\"\n",
    "\n",
    "# Use the pattern on the first tweet in the tweets list\n",
    "hashtags = regexp_tokenize(tweets[0], pattern1)\n",
    "print(hashtags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['@datacamp', '#nlp', '#python']\n"
     ]
    }
   ],
   "source": [
    "# write a pattern that matches both mentions (@) and hashtags\n",
    "pattern2 = r\"[@|#]\\w+\"\n",
    "\n",
    "# Use the pattern on the last tweet in the tweets list\n",
    "mentions_hashtags = regexp_tokenize(tweets[-1], pattern2)\n",
    "print(mentions_hashtags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['This', 'is', 'the', 'best', '#nlp', 'exercise', 'ive', 'found', 'online', '!', '#python'], ['#NLP', 'is', 'super', 'fun', '!', '<3', '#learning'], ['Thanks', '@datacamp', ':)', '#nlp', '#python']]\n"
     ]
    }
   ],
   "source": [
    "# Use the TweetTokenizer to tokenize all tweets into one list\n",
    "tknzr = TweetTokenizer()\n",
    "all_tokens = [tknzr.tokenize(t) for t in tweets]\n",
    "print(all_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-ascii tokenization\n",
    "In this exercise, you'll practice advanced tokenization by tokenizing some non-ascii based text. You'll be using German with emoji!\n",
    "\n",
    "Here, you have access to a string called `german_text`, which has been printed for you in the Shell. Notice the emoji and the German characters!\n",
    "\n",
    "Unicode ranges for emoji are:\n",
    "\n",
    "`('\\U0001F300'-'\\U0001F5FF')`, `('\\U0001F600-\\U0001F64F')`, `('\\U0001F680-\\U0001F6FF')`, and `('\\u2600'-\\u26FF-\\u2700-\\u27BF')`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "german_text = 'Wann gehen wir Pizza essen? 🍕 Und fährst du mit Über? 🚕'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Wann', 'gehen', 'wir', 'Pizza', 'essen', '?', '🍕', 'Und', 'fährst', 'du', 'mit', 'Über', '?', '🚕']\n",
      "['Wann', 'Pizza', 'Und', 'Über']\n",
      "['🍕', '🚕']\n"
     ]
    }
   ],
   "source": [
    "# Tokenize and print all words in german_text\n",
    "all_words = word_tokenize(german_text)\n",
    "print(all_words)\n",
    "\n",
    "# Tokenize and print only capital words\n",
    "capital_words = r\"[A-ZÜ]\\w+\"\n",
    "print(regexp_tokenize(german_text, capital_words))\n",
    "\n",
    "# Tokenize and print only emoji\n",
    "emoji = \"['\\U0001F300-\\U0001F5FF'|'\\U0001F600-\\U0001F64F'|'\\U0001F680-\\U0001F6FF'|'\\u2600-\\u26FF\\u2700-\\u27BF']\"\n",
    "print(regexp_tokenize(german_text, emoji))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Charting word length with NLTK\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Charting practice\n",
    "Try using your new skills to find and chart the number of words per line in the script using matplotlib. The Holy Grail script is loaded for you, and you need to use regex to find the words per line.\n",
    "\n",
    "Using list comprehensions here will speed up your computations. For example: `my_lines = [tokenize(l) for l in lines]` will call a function tokenize on each line in the list lines. The new transformed list will be saved in the `my_lines` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAHiCAYAAAA9Am/ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAazUlEQVR4nO3debBmd13n8c9Xmn0xZAElCXQcI4KOSowYgQGGWFNsGmoGFAc0MnHiTKGCMAXBsQqXYQxVKkuNg0MlQlBkMSBkIKNSLEFUogm4ABETISZNAmnIwr4EvvPHc5pcOre7bzrdud/cvF5Vt/o55znPc37Puaf7fc95zn26ujsAwAzftNkDAABuIMwAMIgwA8AgwgwAgwgzAAwizAAwiDCzJVXVvavq3VX1mar6rU0cxyOrasdmrX9PqupdVfUzy+2nVNWfHYR13LeqPltVt9uPx+73dquqn66q9+zPYw+Wqvp/VXXycnvc+Jhl22YPANaqqr9O8pQkX01ydncft59PdWqSTya5R/tl/b3q7lcnefVBeN7LktztQD/vrVF3P2azx8CthyNmxqiq2ye5X5JLknx/kvfdjKe7X5IP3ZJRrqqRP+juzxErGzf1+86tlzAzyXfnhpgen32EuaoeUlV/U1XXLX8+ZJn/yiQnJ3nOcir1h3d73DFVdW1VfdMyfUZVXbXm/j+oqmcut+9TVedU1dVVdUlV/ec1y/1KVZ29LP/pJD9dVXeuqldW1TVV9aEkP7Dbup9bVR9bTrF/uKpO3MNre2VV/W5VvW1Z9ryqut+a+79zue/q5Xl+bLfHvqyqzq2qzyX5t/vYjt9warWquqr+S1VdvLyO36mqWnP/f6qqi5b7/nTtuHZ73u3Lc21bpt9VVb9eVX+xvKY/q6rD9zG2Z1fVVVV1ZVU9bc38b66qV1XVzqr6l6r65V3fz90e/zu7v5VRVf931/d3L+s9rqrev4zzj6rqdVX1P5b7HllVO5bv5ceTvKKq7llVb1nGc81y+6g1z/f1tw5gn7rbl69N/UrytCTXJvl8ki8ut69P8pnl9jHrPObQJNck+cms3pL5iWX6sOX+Vyb5H3tZ52VJvn+5/eEkH0nygDX3PWi5fV6S/53kTkm+L8nOJCcu9/1Kkq8keUJWP+TeOcnpSf58Gd/RST6QZMey/P2TXJ7kPsv09iT/ag/je+Xy+h+e5I5JXpLkPct9d12e52nLaz8uq9P237Xmsdcleegyrjut8/zvSvIzy+2f3vXcy3QneUuSQ5Lcd3nNj17ue0JWZzQesKz7l5P85R5ew/blubatWec/J/mOZVu9K8npe3jsI5d94NeS3D7JY5f9457L/a9K8uYkd1/W809JTtn99SR5cJIrknzTMn348jz33su+cYck/5LkGcu6/32SL2fZn9aM7YXL9+bOSQ5L8h+S3GUZ0x8ledNGtrcvX7t/OWJm03X3K7r7kCQXJjkhyfdkFbR7dPch3f3RdR72uCQXd/fvd/f13f2aJP+Y5Ec2uNrzkjyiqr5lmT57mT4myT2S/F1VHZ3kYUme291f7O6/TXJGVj8M7PJX3f2m7v5ad38hyY8leUF3X93dlyd56Zplv5rVP+QPrKrbd/el3f3PexnjW7v73d39pST/PckPLWN6fJJLl+12fXe/L8kbkjxxzWPf3N1/sYzrixvcJmud3t3X9up94ndm9UNJkvxskt/o7ou6+/ok/zPJ9+3pqHkdr+juf1q21evXPO96vpLk17r7K919bpLPJrn/cmr+x5M8r7s/092XJvmtfOP3JUnS3X+d1Q8pu85MPDnJu7r7E3tZ7wlZ/dDx0mXdb0zy17st87Ukz+/uL3X3F7r7U939hu7+fHd/JskLkjxir1sC9kCY2VRVdehyWvm6JA/J6sjiw1kdXV6zl1OO98nqqGatf0ly5AZXfV5WRz4PT/LuZb2PWL7+vLu/tqzj6uUf2j2t4/J1xrV23tfH2N2XJHlmVkfaV1XVa6vqPnsZ49efp7s/m+Tq5fnvl+QHl+12bVVdm9UFc9+y3mP308fX3P58briI635JXrJmvVcnqWx8u+/pedfzqSX+uy9/eG44qt1lb9/7s5I8dbn91CS/v48x3ifJx7p77fUJu2/PnWt/4Kmqu1TV/1lOq386q33qkPL+PvtBmNlUy5HlIVkdiZ2x3P6TJD+yHC2/eA8PvSKrSKx13yQf2+Cqz0vyb7KK83lJ3pPVqd9HLNO71nFoVd19L+vY/eKyK7M6hb12+RsW7v7D7n7YMvbO6nTonnz9earqblmdHr8iq0ict2yfXV936+7/updxHSiXJ/nZ3dZ95+7+y4O0vvV8Mquj6bXf/7197/8gyUlV9b1ZnYJ/0z6e/8okR659Xz3f+D1Nbrx9n53VD5M/2N33yOoHvmT1QwvcJMLMFGuvwn5QVqe19+bcJN9RVf+xqrZV1Y8neWBW743uU3dfnOQLWR1Bvbu7P53kE1m9T3jesszlSf4yyW9U1Z2q6nuSnJK9/2rR65M8b7kY6KgkP7/rjqq6f1U9qqrumNV76V/I6vT2njy2qh5WVXdI8utJzl/G9Jbltf9kVd1++fqBqnrARl77zfS7Wb2+70q+fhHWk26B9X5dd381q+38gqq6+3Ia/VlZBXi95Xck+ZusjpTfsJxG35u/yur78nPLvnVSVu9V783ds/p+XltVhyZ5/oZfEOxGmJni+5O8r6oOS/LV7r5mbwt396eyeq/12Uk+leQ5SR7f3Z+8Ces8L6vTpZetma4k71+zzE9kdXHRFUn+OKv3Fd+2l+f81axOq340yZ/lG0+b3jGri8M+mdUp3Xsl+aW9PNcfZvUP/NVZbZ+nJMlyav3fZfV+6RXLc+26EOmg6u4/Xtb12uWU7QeSbMbv6P58ks9lddHee7LaVr+3l+XPSvKvs+/T2OnuL2d1wdcpWV18+NSsfhj60l4e9uKsLgL7ZJL3ZnXWB/ZLfePbKMAEtfqVrx3d/cubPZatoKoentUR9fbl+oGb+vjzk/xud7/igA8OduOIGdjSavXBNc/I6hqGDUW5qh5RVd+ynMo+OavfFHAUzC3CJ9YAW9byvvsFSf4uq9/73jX/vkk+tIeHPTCrC7len9VV4P+c5IndfeXBHS2sOJUNAIM4lQ0AgwgzAAwy4j3mww8/vLdv377ZwwCAW8SFF174ye4+Yr37RoR5+/btueCCCzZ7GABwi6iq3T9S+OucygaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhk22YP4GDYftpbN3sI+3Tp6Y/b7CEAMJAjZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQTYU5qr6xar6YFV9oKpeU1V3qqpjqur8qrq4ql5XVXdYlr3jMn3Jcv/2g/kCAGAr2WeYq+rIJL+Q5Pju/u4kt0vy5CQvTPKi7j42yTVJTlkeckqSa7r725O8aFkOANiAjZ7K3pbkzlW1LcldklyZ5FFJzl7uPyvJE5bbJy3TWe4/sarqwAwXALa2fYa5uz+W5DeTXJZVkK9LcmGSa7v7+mWxHUmOXG4fmeTy5bHXL8sfdmCHDQBb00ZOZd8zq6PgY5LcJ8ldkzxmnUV710P2ct/a5z21qi6oqgt27ty58REDwBa2kVPZP5zko929s7u/kuSNSR6S5JDl1HaSHJXkiuX2jiRHJ8ly/zcnuXr3J+3ul3f38d19/BFHHHEzXwYAbA0bCfNlSU6oqrss7xWfmORDSd6Z5InLMicnefNy+5xlOsv97+juGx0xAwA3tpH3mM/P6iKu9yX5h+UxL0/y3CTPqqpLsnoP+czlIWcmOWyZ/6wkpx2EcQPAlrRt34sk3f38JM/fbfZHkjx4nWW/mORJN39oAHDb45O/AGAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEE2FOaqOqSqzq6qf6yqi6rqh6rq0Kp6W1VdvPx5z2XZqqqXVtUlVfX3VXXcwX0JALB1bPSI+SVJ/qS7vzPJ9ya5KMlpSd7e3ccmefsynSSPSXLs8nVqkpcd0BEDwBa2zzBX1T2SPDzJmUnS3V/u7muTnJTkrGWxs5I8Ybl9UpJX9cp7kxxSVd96wEcOAFvQRo6Yvy3JziSvqKr3V9UZVXXXJPfu7iuTZPnzXsvyRya5fM3jdyzzAIB92EiYtyU5LsnLuvtBST6XG05br6fWmdc3Wqjq1Kq6oKou2Llz54YGCwBb3UbCvCPJju4+f5k+O6tQf2LXKerlz6vWLH/0mscfleSK3Z+0u1/e3cd39/FHHHHE/o4fALaUfYa5uz+e5PKquv8y68QkH0pyTpKTl3knJ3nzcvucJD+1XJ19QpLrdp3yBgD2btsGl/v5JK+uqjsk+UiSp2UV9ddX1SlJLkvypGXZc5M8NsklST6/LAsAbMCGwtzdf5vk+HXuOnGdZTvJ02/muADgNsknfwHAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCAbDnNV3a6q3l9Vb1mmj6mq86vq4qp6XVXdYZl/x2X6kuX+7Qdn6ACw9dyUI+ZnJLlozfQLk7you49Nck2SU5b5pyS5pru/PcmLluUAgA3YUJir6qgkj0tyxjJdSR6V5OxlkbOSPGG5fdIyneX+E5flAYB92OgR84uTPCfJ15bpw5Jc293XL9M7khy53D4yyeVJstx/3bI8ALAP+wxzVT0+yVXdfeHa2ess2hu4b+3znlpVF1TVBTt37tzQYAFgq9vIEfNDk/xoVV2a5LVZncJ+cZJDqmrbssxRSa5Ybu9IcnSSLPd/c5Krd3/S7n55dx/f3ccfccQRN+tFAMBWsc8wd/fzuvuo7t6e5MlJ3tHdT0nyziRPXBY7Ocmbl9vnLNNZ7n9Hd9/oiBkAuLGb83vMz03yrKq6JKv3kM9c5p+Z5LBl/rOSnHbzhggAtx3b9r3IDbr7XUnetdz+SJIHr7PMF5M86QCMDQBuc3zyFwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADCIMAPAIMIMAIMIMwAMIswAMIgwA8AgwgwAgwgzAAwizAAwiDADwCDCDACDCDMADCLMADDIPsNcVUdX1Tur6qKq+mBVPWOZf2hVva2qLl7+vOcyv6rqpVV1SVX9fVUdd7BfBABsFRs5Yr4+ybO7+wFJTkjy9Kp6YJLTkry9u49N8vZlOkkek+TY5evUJC874KMGgC1qn2Hu7iu7+33L7c8kuSjJkUlOSnLWsthZSZ6w3D4pyat65b1JDqmqbz3gIweALegmvcdcVduTPCjJ+Unu3d1XJqt4J7nXstiRSS5f87Ady7zdn+vUqrqgqi7YuXPnTR85AGxBGw5zVd0tyRuSPLO7P723RdeZ1zea0f3y7j6+u48/4ogjNjoMANjSNhTmqrp9VlF+dXe/cZn9iV2nqJc/r1rm70hy9JqHH5XkigMzXADY2jZyVXYlOTPJRd3922vuOifJycvtk5O8ec38n1quzj4hyXW7TnkDAHu3bQPLPDTJTyb5h6r622XeLyU5Pcnrq+qUJJcledJy37lJHpvkkiSfT/K0AzpiANjC9hnm7n5P1n/fOElOXGf5TvL0mzkuALhN8slfADCIMAPAIMIMAIMIMwAMIswAMMhGfl2Kg2D7aW/d7CHs06WnP26zhwBwm+OIGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYJBtmz0A5tp+2ls3ewh7denpj9vsIQAccI6YAWAQYQaAQYQZAAYRZgAYRJgBYBBhBoBBhBkABhFmABhEmAFgEGEGgEGEGQAGEWYAGESYAWAQ/7sUt1rT//erxP+ABdx0jpgBYBBhBoBBhBkABhFmABhEmAFgkINyVXZVPTrJS5LcLskZ3X36wVgPTDf9ynFXjcM8B/yIuapul+R3kjwmyQOT/ERVPfBArwcAtqKDccT84CSXdPdHkqSqXpvkpCQfOgjrAm6G6Uf0HBjOjNy6HIwwH5nk8jXTO5L84EFYDwAb4Aewm++W/OHmYIS51pnXN1qo6tQkpy6Tn62qDx/AMRye5JMH8PluS2y7/Wfb7T/bbv/ZdvvnJm23euEBX//99nTHwQjzjiRHr5k+KskVuy/U3S9P8vKDsP5U1QXdffzBeO6tzrbbf7bd/rPt9p9tt38mb7eD8etSf5Pk2Ko6pqrukOTJSc45COsBgC3ngB8xd/f1VfVzSf40q1+X+r3u/uCBXg8AbEUH5feYu/vcJOcejOfeoINyivw2wrbbf7bd/rPt9p9tt3/GbrfqvtF1WQDAJvGRnAAwyJYLc1U9uqo+XFWXVNVpmz2eyarq6Kp6Z1VdVFUfrKpnLPMPraq3VdXFy5/33OyxTlRVt6uq91fVW5bpY6rq/GW7vW65+JHdVNUhVXV2Vf3jsu/9kH1uY6rqF5e/qx+oqtdU1Z3sd+urqt+rqquq6gNr5q27n9XKS5du/H1VHbd5I99iYfZxoDfZ9Ume3d0PSHJCkqcv2+u0JG/v7mOTvH2Z5saekeSiNdMvTPKiZbtdk+SUTRnVfC9J8ifd/Z1JvjerbWif24eqOjLJLyQ5vru/O6uLa58c+92evDLJo3ebt6f97DFJjl2+Tk3ysltojOvaUmHOmo8D7e4vJ9n1caCso7uv7O73Lbc/k9U/kEdmtc3OWhY7K8kTNmeEc1XVUUkel+SMZbqSPCrJ2csitts6quoeSR6e5Mwk6e4vd/e1sc9t1LYkd66qbUnukuTK2O/W1d3vTnL1brP3tJ+dlORVvfLeJIdU1bfeMiO9sa0W5vU+DvTITRrLrUpVbU/yoCTnJ7l3d1+ZrOKd5F6bN7KxXpzkOUm+tkwfluTa7r5+mbbvre/bkuxM8orlbYAzququsc/tU3d/LMlvJrksqyBfl+TC2O9uij3tZ6PasdXCvKGPA+UbVdXdkrwhyTO7+9ObPZ7pqurxSa7q7gvXzl5nUfvejW1LclySl3X3g5J8Lk5bb8jyfuhJSY5Jcp8kd83qFOzu7Hc33ai/v1stzBv6OFBuUFW3zyrKr+7uNy6zP7HrNM7y51WbNb6hHprkR6vq0qzeLnlUVkfQhyynGBP73p7sSLKju89fps/OKtT2uX374SQf7e6d3f2VJG9M8pDY726KPe1no9qx1cLs40BvguV90TOTXNTdv73mrnOSnLzcPjnJm2/psU3W3c/r7qO6e3tW+9g7uvspSd6Z5InLYrbbOrr740kur6r7L7NOzOq/hLXP7dtlSU6oqrssf3d3bTv73cbtaT87J8lPLVdnn5Dkul2nvDfDlvuAkap6bFZHL7s+DvQFmzyksarqYUn+PMk/5Ib3Sn8pq/eZX5/kvln9Y/Ck7t79IgqSVNUjk/y37n58VX1bVkfQhyZ5f5KndveXNnN8E1XV92V10dwdknwkydOyOkiwz+1DVf1qkh/P6jcq3p/kZ7J6L9R+t5uqek2SR2b1v0h9Isnzk7wp6+xnyw86/yurq7g/n+Rp3X3BZow72YJhBoBbs612KhsAbtWEGQAGEWYAGESYAWAQYQaAQYQZAAYRZgAYRJgBYJD/DzkNcPF8k9eFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Split the script into lines: lines\n",
    "lines = holy_grail.split('\\n')\n",
    "\n",
    "# Replace all script lines for speaker\n",
    "pattern = \"[A-Z]{2,}(\\s)?(#\\d)?([A-Z]{2,})?:\"\n",
    "lines = [re.sub(pattern, '', l) for l in lines]\n",
    "\n",
    "# Tokenize each line: tokenized_lines\n",
    "tokenized_lines = [regexp_tokenize(s, '\\w+') for s in lines]\n",
    "\n",
    "# Make a frequency list of lengths: line_num_words\n",
    "line_num_words = [len(t_line) for t_line in tokenized_lines]\n",
    "\n",
    "# Plot a histogram of the line lengths\n",
    "plt.figure(figsize=(8,8))\n",
    "plt.hist(line_num_words);\n",
    "plt.title('# of words per line in holy_grail');"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}