regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What are regular expressions?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Regular expressions are designed for matching patterns in text.  \n",
    "- Examples are picking out three digits in a row or characters separated by white space or punctuation(i.e. words in a sentence).  \n",
    "- **They can be extremely useful for parsing & cleaning data - check that entries follow a certain format or extract parts of a string based on certain criteria**\n",
    "- The downside is that they aren't very human-readable.  \n",
    "- Sometimes called REs, regexes or regex patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regular Expressions in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# re is the regular expression module in python\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most characters match themselves\n",
    "For example `test` will match 'test'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 4), match='test'>\n"
     ]
    }
   ],
   "source": [
    "# Compiling the regular expression stores it as a regular expression object that you can use\n",
    "retest = re.compile('test')\n",
    "mystring = \"testing. 1. 2. 3. ...\"\n",
    "# The match method looks for matches at the beginning of a string\n",
    "# If there is a match it will return a match object \n",
    "# The match object has information about what matched\n",
    "# information about what part of the string matched\n",
    "m = retest.match(mystring)\n",
    "print(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "# If there is no match, it will return None\n",
    "mystring2 = \"You can learn to be a good test taker\"\n",
    "print(retest.match(mystring2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metacharacters - special characters\n",
    "These characters have special meanings and do not match themselves\n",
    ". ^ $ * + ? { } [ ] \\ | ( )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Square Brackets - specify a set of characters to match\n",
    "- Called a character class\n",
    "- Characters can be listed individually or with a `-` to indicate a range\n",
    "- `[aeiouAEIOU]` would match any vowel\n",
    "- `[a-z]` would match any lowercase letter\n",
    "- Metacharacters are not active in character classes `[ab\\]` would match a,b or \\\n",
    "- You can match charcters not listed in a set by complementing a set with `^` at the beginning\n",
    "- `[^aeiouAEIOU]` would match any character that is NOT a vowel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 1), match='I'>\n"
     ]
    }
   ],
   "source": [
    "rebrackets = re.compile('[aeiouAEIOU]')\n",
    "mystring3 = \"I like Python, cookies and tea!\"\n",
    "m = rebrackets.match(mystring3)\n",
    "print(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Asterix - match ZERO or more times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 2), match='ct'> <re.Match object; span=(0, 3), match='cat'> <re.Match object; span=(0, 12), match='caaaaaaaaaat'>\n"
     ]
    }
   ],
   "source": [
    "# the * matches zero or more 'a's\n",
    "recat = re.compile('ca*t')\n",
    "# all of these will match\n",
    "m1 = recat.match(\"ct\")\n",
    "m2 = recat.match(\"cat\")\n",
    "m3 = recat.match(\"caaaaaaaaaat\")\n",
    "print(m1, m2, m3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plus sign - match ONE or more times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the + matches zone or more 'a's\n",
    "recat = re.compile('ca+t')\n",
    "# all of these will match\n",
    "m1 = recat.match(\"ct\")\n",
    "m2 = recat.match(\"cat\")\n",
    "m3 = recat.match(\"caaaaaaaaaat\")\n",
    "print(m1, m2, m3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question mark - match zero or one times\n",
    "-think of this as an matching an optional character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 12), match='two year-old'> <re.Match object; span=(0, 11), match='two yearold'>\n"
     ]
    }
   ],
   "source": [
    "retwo = re.compile('two year-?old')\n",
    "m1 = retwo.match('two year-old')\n",
    "m2 = retwo.match('two yearold')\n",
    "print(m1, m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backslashes - escape out special characters\n",
    "If you want to match another special character the `\\` in front of it lets you do this\n",
    "\n",
    "**Backslashes are also escape characters for strings in Python which makes things tricky and weird - read more at this link:**\n",
    "[The Backslash Plague](https://docs.python.org/2/howto/regex.html#the-backslash-plague)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 3), match='2+2'>\n"
     ]
    }
   ],
   "source": [
    "# this will match '2+2' whereas '2+2' without the \\ would match any series of more than three 2s\n",
    "replus = re.compile('2\\+2')\n",
    "m1 = replus.match('2+2')\n",
    "print(m1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipe (bar, vertical line or whatever you want to call it) - match this OR that\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(7, 13), match='coffee'>\n"
     ]
    }
   ],
   "source": [
    "redrink = re.compile('coffee|tea')\n",
    "# using search looks for the regex anywhere in the string\n",
    "# using match as above just looks at the beginning\n",
    "m = redrink.search('I like coffee')\n",
    "print(m)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matching numbers, whitespace, characters, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are more of these that you can look up, but here are a few to get started with\n",
    "-  \\d will match any digit   (inverse \\D will match anything EXCEPT a digit)\n",
    "-  \\w will match a 'word character' - letter, digit or underscore  (inverse \\W will match anything EXCEPT a word character)\n",
    "-  \\s will match any whitespace character  (inverse \\S will match anything EXCEPT whitespace)\n",
    "-  \\t matches Tab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(5, 7), match='98'>\n"
     ]
    }
   ],
   "source": [
    "# This will match any number of digits\n",
    "renum = re.compile('\\d+')\n",
    "m1 = renum.search('I am 98 years old!')\n",
    "print(m1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groups - ()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Groups let you catch parts of a string to use seperately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cliff:25 Cliff 25\n"
     ]
    }
   ],
   "source": [
    "reperson = re.compile('(\\w+):(\\d+)')\n",
    "# let's say you have data in the form 'name:age'\n",
    "m1 = reperson.match('Cliff:25')\n",
    "# group(0) is always the entire result\n",
    "name = m1.group(1)\n",
    "age = m1.group(2)\n",
    "print(m1.group(0), name, age)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of matches\n",
    "-  match() - looks to match the beginning of a string\n",
    "-  search() - will match anywhere in the string\n",
    "-  findall()  - Find all substrings that match and return as a list\n",
    "-  finditer() - Gind all substrings that match and retrun as an iterator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['14', '8756', '2', '6']\n"
     ]
    }
   ],
   "source": [
    "renum = re.compile('\\d+')\n",
    "matches = renum.findall(\"There were 14 mice that lived at 8756 2nd Avenue in a 6 bedroom house.\")\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many options for things you can do with regular expressions.  Some of them include:\n",
    "-  Also testing what is ahead or behind your expression without capturing it\n",
    "-  Capturing and non-capturing groups\n",
    "-  Testing if something is before or after a new line  \n",
    "-  Split a string into substrings by where a regex is found\n",
    "-  Replace parts of a string\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommended Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Python.org Regular Expression HOWTO](https://docs.python.org/2/howto/regex.html#regex-howto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Match Parcel Numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Oklahoma, most counties follow a convention for creating a unique ID for each parcel of land owned in the county.  \n",
    "\n",
    "For rural parcels, this convention is:\n",
    "0000-SS-TTT-RRR-Q-PPP-00\n",
    "SS is a two digit section number from 01 to 36\n",
    "TTT is a township expressed as two digits and a N or a S (north or south)\n",
    "RRR is a range expressed as two digits and a E or a W (east or west)\n",
    "Q is for the quarter section- a number from 1 to 4 (NE, NW, SW, SE)\n",
    "PPP is a number assigned consecutively for parcels in a quarter section\n",
    "\n",
    "\n",
    "Create a regular expression to test if a string meets this format. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "testNumbers = ['0000-03-24N-19E-3-008-00',\n",
    "    '0000-22-24N-20E-1-002-00',\n",
    "    '0000-35-24N-20E-1-008-00',\n",
    "    '0000-35-24N-20E-1-005-00',\n",
    "    '000-24-24S-21E-1-005-00',\n",
    "    '0000-24-05N-21E-1-002-00',\n",
    "    '0000-24-24N-21E-4-001-00',\n",
    "    '0000-16-25N-19T-2-002-00',\n",
    "    '0000-34-25S-21E-2-002-00',\n",
    "    '0000-37-25B-21W-2-007-00',\n",
    "    '0000-24-26N-18E-1-004-00']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework : Match Phone Numbers\n",
    "Use a regular expression to pick just the phone numbers out from this list AND store them in a new list in the format XXX-XXX-XXXX.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contacts = ['Mary: 505-343-7644', 'Santana: (834)-434-5879', 'Brandon - 436.753.4956']"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}