02_02_intro_to_regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Using RegEx with spaCy</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>Dr. W.J.B. Mattingly</center>\n",
    "\n",
    "<center>Smithsonian Data Science Lab and United States Holocaust Memorial Museum</center>\n",
    "\n",
    "<center>January 2021</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Concepts in this Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) What is RegEx (Regular Expressions)?<br>\n",
    "2) The Strengths of RegEx<br>\n",
    "3) The Weaknesses of RegEx<br>\n",
    "4) How to use RegEx in Python<br>\n",
    "5) How to use RegEx in spaCy<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Regular Expressions (RegEx)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular Expressions, or RegEx for short, is a way of achieving complex string matching based on simple or complex patterns. It can be used to perform finding and retrieving patterns or replacing matching patterns in a string with some other pattern. It was invnted by an Stephen Cole Kleene in the 1950s and is still widely used today for numerous tasks, but particularly string matching in texts. RegEx are fully integrated with most search engines and can allow for more robust searching. Nearly all data scientists, especially those who work with texts, use RegEx at some stage in their workflow, from data searching, to cleaning data, to implementing machine learning models. It is an essential tool for any text-based researcher. For these reasons, it merits a few chapters in this textbook.\n",
    "\n",
    "In spaCy it can be leveraged in a few different pipes (depending on the task at hand as we shall see), to identify things such as entities or pattern matching."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Strengths of RegEx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several strengths to RegEx.\n",
    "\n",
    "1) Due to its complex syntax, it can allow for programmers to write robust rules in short spaces.<br>\n",
    "2) It can allow the researcher to find all types of variance in strings<br>\n",
    "3) It can perform remarkably quickly when compared to other methods.\n",
    "4) It is universally supported"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Weaknesses of RegEx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite these strengths, there are a few weaknesses to RegEx.\n",
    "1) Its syntax is quite difficult for beginners. (I still find myself looking up how to do certain things).<br>\n",
    "2) It order to work well, it requires a domain-expert to work alongside the programmer to think of all ways a pattern may vary in texts.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Use RegEx in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python comes prepackaged with a RegEx library. We can import it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have it imported, we can begin to write out some RegEx rules. Let's say we want to find an occurrence of a date in a text. As noted in an earlier notebook, there are a finite number of ways this can be represented. Let's try to grab all instances of a day followed by a month first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('2 February', '2', 'February'), ('14 August', '4', 'August')]\n"
     ]
    }
   ],
   "source": [
    "pattern = r\"((\\d){1,2} (January|February|March|April|May|June|July|August|September|October|November|December))\"\n",
    "\n",
    "text = \"This is a date 2 February. Another date would be 14 August.\"\n",
    "matches = re.findall(pattern, text)\n",
    "print (matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this bit of code, we see a real-life RegEx formula at work. While this looks quite complex, its syntax is fairly straight forward. Let's break it down.\n",
    "The first ( tells RegEx that I'm looking for something within the ending ). In other words, I'm looking for a pattern that's going to match the whole pattern, not just components.\n",
    "\n",
    "Next, we state (\\d){1,2}. This means that we are looking for any digit (0-9) that occurs either once or twice ({1,2}).\n",
    "\n",
    "Next, we have a space to indicate the space in the string that we would expect with a date.\n",
    "\n",
    "Next, we have (January|February|March|April|May|June|July|August|September|October|November|December) -- this indicates another component of the pattern (because it is parentheses). The | indicates the same concept as \"or\" in English, so either January, or February, or March, etc.\n",
    "\n",
    "When we bring it together, this pattern will match anything that functions as a set of one or two numbers followed by a month. What happens when we try and do this with a date that is formed the opposite way?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('14 August', '4', 'August')]\n"
     ]
    }
   ],
   "source": [
    "text = \"This is a date February 2. Another date would be 14 August.\"\n",
    "matches = re.findall(pattern, text)\n",
    "print (matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It fails. But this is no fault of RegEx. Our pattern cannot accommodate that variation. Nevertheless, we can account for it by adding it as a possible variation. Possible variations are accounted for with a *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('February 2', '', '', '', '', 'February 2', 'February ', 'February', '2'), ('14 August', '14 August', '4', ' August', 'August', '', '', '', '')]\n"
     ]
    }
   ],
   "source": [
    "pattern = r\"(((\\d){1,2}( (January|February|March|April|May|June|July|August|September|October|November|December)))|(((January|February|March|April|May|June|July|August|September|October|November|December) )(\\d){1,2}))\"\n",
    "\n",
    "text = \"This is a date February 2. Another date would be 14 August.\"\n",
    "matches = re.findall(pattern, text)\n",
    "print (matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are more concise ways to write the same RegEx formula. I have opted here to be more verbose to make it a bit easier to read. You can see that we've allowed for two main options for our pattern matcher.\n",
    "\n",
    "Notice, however, that we have a lot of superfluous information for each match. These are the components of each match. There are several ways we can remove them. One way is to use the command finditer, rather than findall in RegEx."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<callable_iterator object at 0x00000217A415BC10>\n"
     ]
    }
   ],
   "source": [
    "text = \"This is a date February 2. Another date would be 14 August.\"\n",
    "iter_matches = re.finditer(pattern, text)\n",
    "print (iter_matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an iterator object, we can loop over it, however, and get our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<callable_iterator object at 0x00000217A4256670>\n",
      "<re.Match object; span=(15, 25), match='February 2'>\n",
      "<re.Match object; span=(49, 58), match='14 August'>\n"
     ]
    }
   ],
   "source": [
    "text = \"This is a date February 2. Another date would be 14 August.\"\n",
    "iter_matches = re.finditer(pattern, text)\n",
    "print (iter_matches)\n",
    "for hit in iter_matches:\n",
    "    print (hit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within each of these is some very salient information, such as the start and end location (inside the span) and the text itself (match). We can use the start and end location to grab the text within the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "February 2\n",
      "14 August\n"
     ]
    }
   ],
   "source": [
    "text = \"This is a date February 2. Another date would be 14 August.\"\n",
    "iter_matches = re.finditer(pattern, text)\n",
    "for hit in iter_matches:\n",
    "    start = hit.start()\n",
    "    end = hit.end()\n",
    "    print (text[start:end])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Use RegEx in spaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things like dates, times, IP Addresses, etc. that have either consistent or fairly consistent structures are excellent candidates for RegEx. Fortunately, spaCy has easy ways to implement RegEx in three pipes: Matcher, PhraseMatcher, and EntityRuler. One of the major drawbacks to the Matcher and PhraseMatcher, is that they do not align the matches as doc.ents. Because this textbook is about NER and our goal is to store the entities in the doc.ents, we will focus on using RegEx with the EntityRuler. In the next notebook, we will examine other methods.\n",
    "\n",
    "In the previous notebook, we saw how the code below allowed for us to capture the phone number in the string. I have modified it a bit here for reasons that will become a bit more clear below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Enabling eager execution\n",
      "INFO:tensorflow:Enabling v2 tensorshape\n",
      "INFO:tensorflow:Enabling resource variables\n",
      "INFO:tensorflow:Enabling tensor equality\n",
      "INFO:tensorflow:Enabling control flow v2\n",
      "555-5555 PHONE_NUMBER\n"
     ]
    }
   ],
   "source": [
    "#Import the requisite library\n",
    "import spacy\n",
    "\n",
    "#Sample text\n",
    "text = \"This is a sample number 555-5555.\"\n",
    "\n",
    "#Build upon the spaCy Small Model\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "#Create the Ruler and Add it\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "\n",
    "#List of Entities and Patterns (source: https://spacy.io/usage/rule-based-matching)\n",
    "patterns = [\n",
    "                {\"label\": \"PHONE_NUMBER\", \"pattern\": [{\"SHAPE\": \"ddd\"},\n",
    "                {\"ORTH\": \"-\", \"OP\": \"?\"}, {\"SHAPE\": \"dddd\"}]}\n",
    "            ]\n",
    "#add patterns to ruler\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "#create the doc\n",
    "doc = nlp(text)\n",
    "\n",
    "#extract entities\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method worked well for grabbing the phone number. But what if we wanted to use RegEx as opposed to linguistic features, such as shape? First, let's write some RegEx to capturee 555-5555."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('555-5555', '5', '5')]\n"
     ]
    }
   ],
   "source": [
    "pattern = r\"((\\d){3}-(\\d){4})\"\n",
    "text = \"This is a sample number 555-5555.\"\n",
    "matches = re.findall(pattern, text)\n",
    "print (matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay. So, now we know that we have a RegEx pattern that works. Let's try and implement it in the spaCy EntityRuler. We can do that with the code below. When we execute the code below, we have no output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import the requisite library\n",
    "import spacy\n",
    "\n",
    "#Sample text\n",
    "text = \"This is a sample number (555) 555-5555.\"\n",
    "\n",
    "#Build upon the spaCy Small Model\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "#Create the Ruler and Add it\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "\n",
    "#List of Entities and Patterns (source: https://spacy.io/usage/rule-based-matching)\n",
    "patterns = [\n",
    "                {\n",
    "                    \"label\": \"PHONE_NUMBER\", \"pattern\": [{\"TEXT\": {\"REGEX\": \"((\\d){3}-(\\d){4})\"}}\n",
    "                                                        ]\n",
    "                }\n",
    "            ]\n",
    "#add patterns to ruler\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "\n",
    "#create the doc\n",
    "doc = nlp(text)\n",
    "\n",
    "#extract entities\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is for one very important reason. SpaCy's EntityRuler cannot use RegEx to pattern match across tokens. The dash in the phone number throws off the EntityRuler. So, what are we to do in this scenario? Well, we have a few different options that we will explore in the next notebook. But before we get to that, let's try and use RegEx to capture the phone number with no hyphen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5555555 PHONE_NUMBER\n"
     ]
    }
   ],
   "source": [
    "#Import the requisite library\n",
    "import spacy\n",
    "\n",
    "#Sample text\n",
    "text = \"This is a sample number 5555555.\"\n",
    "#Build upon the spaCy Small Model\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "#Create the Ruler and Add it\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "\n",
    "#List of Entities and Patterns (source: https://spacy.io/usage/rule-based-matching)\n",
    "patterns = [\n",
    "                {\n",
    "                    \"label\": \"PHONE_NUMBER\", \"pattern\": [{\"TEXT\": {\"REGEX\": \"((\\d){5})\"}}\n",
    "                                                        ]\n",
    "                }\n",
    "            ]\n",
    "#add patterns to ruler\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "\n",
    "#create the doc\n",
    "doc = nlp(text)\n",
    "\n",
    "#extract entities\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that without the dash and a few modifications to our RegEx, we were able to capture 5555555 because this is a single token in the spaCy doc object. Let's explore how to solve the problem in the next notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div align=\"center\">\n",
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/wpyCzodvO3A\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div align=\"center\">\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/wpyCzodvO3A\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}