Chapter 6 - Regular Expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 6 - Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions are another, powerful way of handling strings in Python. You can use them for all sorts of very common string operations, like searching words and replacing them in texts. In fact, regular expressions are often used across many programming languages and text editors. Because of that, you will often be able to reuse many of the things that you will learn about below. The functionality for using regular expressions in Python is included in the 're' package, which you should be able to import as usual: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On many occasions, you will want to search a string in your scripts: e.g. does the following word appear in a text? Is the format of the following email address valid and does it contain an @-symbol and a least one dot? To carry out such operations, the first thing you need is a string to search:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"In principio erat verbum, et verbum erat apud Deum.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next thing we define is the actual regular expression which we will use, or the string that we will use to search the sentence we defined above. We pass this string to the `compile()` function in the `re` package, which will allow fast searching later on. Note that we put an `r` in front of this string when we initialize it, which turns our string into a so-called 'raw string'. While this is not always necessary, it is a good idea to do this consistently when dealing with regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile(r\"verbum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can call the `sub()` function from the `re` package on this pattern, in order to replace (or 'substitute') our pattern with another word, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = pattern.sub(\"XXX\", s)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the order of the arguments passed to `sub()`: first, the word we would like to replace our pattern with, and secondly our original string. We can just as easily get back our original string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern2 = re.compile(r\"XXX\")\n",
    "text = pattern2.sub(\"verbum\", s)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far nothing special: we are simply replacing one word for another word. The smart ones among you will have noticed that we could have achieved the exact same result using the `replace()` function, which we came across in an earlier chapter. But now: say you would like to replace all vowels in a string. With regular expressions, this is a piece of cake:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vowel_pattern = re.compile(r\"a|e|o|u|i\")\n",
    "without_vowels = vowel_pattern.sub(\"X\", s)\n",
    "print(without_vowels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how our pattern allows for a special syntax: the pipe symbol which we used allows to express that one character OR another one is fine for the regular expression to match. Oops: the capital letter at the beginning of the sentence hasn't been replaced because we only included lowercase vowels in our pattern definition. Let's add the uppercase vowels to the regex:   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vowel_pattern = re.compile(r\"a|A|e|E|o|O|u|U|i|I\")\n",
    "without_vowels = vowel_pattern.sub(\"X\", s)\n",
    "print(without_vowels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is in fact an easy way to match all lowercase and uppercase characters in a string, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ups = re.compile(r\"[A-Z]\")\n",
    "lows = re.compile(r\"[a-z]\")\n",
    "without_ups = ups.sub(\"X\", s)\n",
    "print(without_ups)\n",
    "without_ups = lows.sub(\"X\", s)\n",
    "print(without_ups)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These specific patterns are called 'ranges': they will match any lowercase or uppercase letter. In fact, you can use such a range syntax using squared brackets, to replace the pipe syntax we used earlier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vowel_pattern = re.compile(r\"[aeoui]\")\n",
    "without_vowels = vowel_pattern.sub(\"X\", s)\n",
    "print(without_vowels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also look for more specific, as well as longer letter groups by arranging them with round brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "p = re.compile(r\"(ri)|(um)|(Th)\")\n",
    "print(vowel_pattern.sub(\"X\", s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also a syntax to match any character (except the newline):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "any_char = re.compile(r\".\")\n",
    "print(any_char.sub(\"X\", s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like your expression to match an actual dot, you have to escape it using a backslash:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dot = re.compile(r\"\\.\")\n",
    "print(dot.sub(\"X\", s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the way, there exist more characters that you might have to escape using a backslash. This is because they are part of the syntax that use to define regular expressions: if you don't escape them, Python will not know that you are looking for an literal match. Characters that you typically might want to escape include: ( + ? . * ^ $ ( ) [ ] { } | \\ ) ,. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"In principio [erat] verbum, et verbum erat apud Deum.\"\n",
    "brackets_wrong = re.compile(r\"[|]\")\n",
    "print(brackets_wrong.sub(\"X\", s))\n",
    "brackets_right = re.compile(r\"(\\[)|(\\])\")\n",
    "print(brackets_right.sub(\"X\", s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The syntax for regular expression includes a whole range of possibilities which we simply cannot all deal with it here. Because of that we will stick to a number of helpful examples. An interesting feature is that you can specify whether or not a character really has to occur. You can check whether the pattern occurs in a string using the `match()` function which will return `None` if it doesn't find the pattern in the string searched:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile(r\"m{2,4}\")\n",
    "print(pattern.match(\"\"))\n",
    "print(pattern.match(\"m\"))\n",
    "print(pattern.match(\"mm\"))\n",
    "print(pattern.match(\"mmm\"))\n",
    "print(pattern.match(\"mmmm\"))\n",
    "print(pattern.match(\"mmmmm\"))\n",
    "print(pattern.match(\"mmmmmm\"))\n",
    "print(pattern.match(\"mmmmammm\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the curly brackets, you indicate that you are only interested in the letter 'm' if it occurs 2,3 or 4 times in a row in the string you search. Because `None` is returned if not a single match was found, you can use the outcome of `match()`in an if-statement. The following example shows how you can also use the curly brackets to match an exact number of occurences (in this case three a's)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pattern = re.compile(r\"a{5}\")\n",
    "if pattern.match(\"aaaaa\"):\n",
    "    print(\"Found it!\")\n",
    "else:\n",
    "    print(\"Nope...\")\n",
    "# or:\n",
    "if pattern.match(\"aa\"):\n",
    "    print(\"Found it!\")\n",
    "else:\n",
    "    print(\"Nope...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a plus sign you can indicate whether you want to match multiple occurrences of a character. A good example from the world of paper writing are double spaces, which can be hard to spot. In the code block below, we replace all multiple occurences of a whitespace character by a single whitespace character. Note that you can use the whitespace character just like any other character (you don't have to escape it). Multiple occurences of the whitespace character will be matched: it doesn't matter how many, as long as there is at least one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "paper = \"My thesis on  biology     contains a lot of  double spaces.   I will remove  them.\"\n",
    "mult = re.compile(r\" +\")\n",
    "print(mult.sub(\" \", paper))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar piece of functionality is offered by the asterisk operator: here you can match multiple occurences of the same character in a row OR not a single one. Note the subtle difference with respect to the plus operator, which needs at least a single occurence of the character to match. Here we use the `search()` function which will search the entire string: the `match()` function which we used earlier will only look for matches at the very beginning of a string. Keep this in mind! The final pattern below yields a match, although there is not a single 'x' in the sentence. That is because the pattern with the asterisk says: \"a single x, or no x at all\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"In English some letters occur multiple times in a row.\"\n",
    "p1 = re.compile(r\"t\")\n",
    "p2 = re.compile(r\"t*\")\n",
    "p3 = re.compile(r\"x\")\n",
    "p4 = re.compile(r\"x*\")\n",
    "print(p1.search(s))\n",
    "print(p2.search(s))\n",
    "print(p3.search(s))\n",
    "print(p4.search(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, you also use regular expression to search inside words. Can you explain why the following patterns (don't) match? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "candidates = [\"good\", \"god\", \"gud\", \"gd\"]\n",
    "p = re.compile(r\"go+d\")\n",
    "for c in candidates:\n",
    "    print(p.match(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Speaking of words: it might be interesting to know that you can use regular expressions for advanced string splitting. If you want to split a sentence across all whitespace characters for instance, you can use an espaced `\\s`. This operator will match all whitespace characters, such as tabs, linebreaks, normal spaces etc. If you add a `+` sign, your pattern will match series of whitespace characters: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"\"\"This is a text  on three   lines\n",
    "with  multiple instances of  \n",
    "double spaces.\"\"\"\n",
    "whitespace = re.compile(r\"\\s+\")\n",
    "print(whitespace.split(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would have wanted to split on the linebreaks only (possible followed by e.g. spaces), you could have used the following pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"\"\"This is a text  on three   lines\n",
    "with  multiple instances of  \n",
    "double spaces.\"\"\"\n",
    "whitespace = re.compile(r\"\\s*\\n\\s*\")\n",
    "print(whitespace.split(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to correct the double spaces, we could now do:    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds = re.compile(r\" +\")\n",
    "for line in whitespace.split(s):\n",
    "    print ds.sub(\" \", line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One final feature we should mention is the `[^...]` syntax: this will match any character that is NOT between the brackets. Remember the vowel_pattern above? Using the caret symbol we can quickly 'invert' this pattern, so that it will match all consonants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = \"these are vowels and consonants\"\n",
    "consonants = re.compile(r\"[^aeuoi]\")\n",
    "print(consonants.sub(\"X\", s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions are really useful, but they can get tricky as well as difficult to read, because of the many different options that exist. There is a whole range of special symbols which you can use to match nearly everything in a text, from word boundaries (\\b) to digits (\\d) etc. Don't learn these by heart but look up a good reference list online (like http://www.tutorialspoint.com/python/python_reg_expressions.htm). As usual Stackoverflow will prove really useful when you search for information online."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Exercises Chapter 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 1 - Write Python code that loads data items from a file that has the format below. Use regular expresions to parse the lines and the data fields: take care of the multiple whitespace characters that might occur. Fill a dictionary using the two data fields. Use regular expressions as much as possible!\n",
    "\n",
    "Example data:\n",
    "\n",
    "color = red\n",
    "\n",
    "number =7\n",
    "\n",
    "name= joe\n",
    "\n",
    "age =   9\n",
    "...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 2 - In the scientific community you will often find data online that has been stored in '.csv' format. Each data item in these files is represented on separate line. Write a function that takes a csv-filename as only input parameter and return a lists of lists, containing the data fields for each item.\n",
    "\n",
    "Example data:\n",
    "\n",
    "Mike, 28, scientist, Belgium\n",
    "\n",
    "Lars, 49, research director, Luxemburg\n",
    "\n",
    "Matt, 52, rockstar, US\n",
    "\n",
    "Example output:\n",
    "[[\"Mike\",\"28\",\"scientist\",\"Belgium\"],[\"Lars\",\"49\",\"research director\",\"Luxemburg\"], ...]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 3 - Expand the previous excercise (don't throw away the original version!). Assume that the first line of your csv-file is not a real data-entry, but a so-called header-line that contains the names of the data fields stored in your csv-file. Now, have your function return a list of dictionaries: one for data item, containing for each item the value for each data field which you find.\n",
    "\n",
    "Example data:\n",
    "name, age, profession, country\n",
    "\n",
    "Mike, 28, scientist, Belgium\n",
    "\n",
    "Lars, 49, research director, Luxemburg\n",
    "\n",
    "Matt, 52, rockstar, US\n",
    "...\n",
    "Example output:\n",
    "[{\"name\": \"Mike\", \"age\": \"28\", \"profession\":\"scientist\", \"country\":\"Belgium\"}, {\"name\": \"Lars\", \"age\": \"49\", \"profession\":\"research director\", \"country\":\"Luxemburg\"]}, ...]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 4 - Write a function that reads a random text file, splits the words across whitespace instances and returns a set containing all words that contain at least two characters. Use regular expressions where possible!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 5 - Come up with a regular expression that matches time-of-day strings (such as 9:14 am or 11:20 pm)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ex. 6 - Write a function that can validate email addresses: a valid email address contains at least one dot, one (and only one!) at-symbol. It should not contain other punctuation symbols and it should end in a common extension like \".com\", \".net\" or \".org\". Again, use regular expressions where possible! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------\n",
    "\n",
    "You've reached the end of Chapter 6! Ignore the code below, it's only here to make the page pretty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    @font-face {\n",
       "        font-family: \"Computer Modern\";\n",
       "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "    }\n",
       "    div.cell {\n",
       "        width:800px;\n",
       "        margin-left:auto;\n",
       "        margin-right:auto;\n",
       "    }\n",
       "    div.cell, .input.hbox {\n",
       "    display:block;\n",
       "}\n",
       "    h1 {\n",
       "        font-family: \"Charis SIL\", Palatino, serif;\n",
       "    }\n",
       "    h4{\n",
       "        margin-top:12px;\n",
       "        margin-bottom: 3px;\n",
       "       }\n",
       "    div.text_cell_render{\n",
       "        font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
       "        line-height: 145%;\n",
       "        font-size: 120%;\n",
       "        width:800px;\n",
       "        margin-left:auto;\n",
       "        margin-right:auto;\n",
       "    }\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
       "    }\n",
       "    .prompt{\n",
       "        display: None;\n",
       "    }\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 300;\n",
       "        font-size: 16pt;\n",
       "        color: #4057A1;\n",
       "        font-style: italic;\n",
       "        margin-bottom: .5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "    }\n",
       "    \n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }\n",
       "\n",
       "    blockquote {\n",
       "    border-left: 4px solid #DDDDDD;\n",
       "    color: #777777;\n",
       "        padding: 0 15px;\n",
       "        }\n",
       "\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}