PythonTour/14-Strings-and-Regular-Expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<!--BOOK_INFORMATION-->\n",
    "<img align=\"left\" style=\"padding-right:10px;\" src=\"fig/cover-small.jpg\">\n",
    "*This notebook contains an excerpt from the [Whirlwind Tour of Python](http://www.oreilly.com/programming/free/a-whirlwind-tour-of-python.csp) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/WhirlwindTourOfPython).*\n",
    "\n",
    "*The text and code are released under the [CC0](https://github.com/jakevdp/WhirlwindTourOfPython/blob/master/LICENSE) license; see also the companion project, the [Python Data Science Handbook](https://github.com/jakevdp/PythonDataScienceHandbook).*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Modules and Packages](13-Modules-and-Packages.ipynb) | [Contents](Index.ipynb) | [A Preview of Data Science Tools](15-Preview-of-Data-Science-Tools.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# String Manipulation and Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This section will cover some of \n",
    "- Python's built-in string methods and formatting operations\n",
    "- a quick guide to the extremely useful subject of *regular expressions*.\n",
    "\n",
    "Strings in Python can be defined using either single or double quotations (they are functionally equivalent):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 'a string'\n",
    "y = \"a string\"\n",
    "x == y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In addition, it is possible to define multi-line strings using a triple-quote syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "multiline = \"\"\"\n",
    "one\n",
    "two\n",
    "three\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "With this, let's take a quick tour of some of Python's string manipulation tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simple String Manipulation in Python\n",
    "\n",
    "For basic manipulation of strings, Python's **built-in string methods** can be extremely convenient.\n",
    "\n",
    "> If you have a background working in C or another low-level language, you will likely find the simplicity of Python's methods extremely refreshing.\n",
    "\n",
    "We introduced Python's string type and a few of these methods earlier; here we'll dive a bit deeper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Formatting strings: Adjusting case\n",
    "\n",
    "To adjust the case of a string.\n",
    "- ``upper()``\n",
    "- ``lower()``\n",
    "- ``capitalize()``\n",
    "- ``title()``\n",
    "- ``swapcase()`` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fox = \"tHe qUICk bROWn fOx.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To convert the entire string into upper-case or lower-case, you can use the ``upper()`` or ``lower()`` methods respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'THE QUICK BROWN FOX.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the quick brown fox.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A common formatting need is to capitalize the first letter \n",
    "\n",
    "- ``title()``  to capitalize just the first letter of each word, \n",
    "- ``capitalize()`` to capitalize the first letter of each sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Quick Brown Fox.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox.title()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The quick brown fox.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox.capitalize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The cases can be swapped using the ``swapcase()`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ThE QuicK BrowN FoX.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox.swapcase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Formatting strings: Adding and removing spaces\n",
    "\n",
    "- the ``strip()`` method\n",
    "    - strips whitespace from the beginning and end of the line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is the content'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = '         this is the content         '\n",
    "line.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To remove just space to the right or left, use ``rstrip()`` or ``lstrip()`` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'         this is the content'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.rstrip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is the content         '"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.lstrip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To remove characters other than spaces, you can pass the desired character to the ``strip()`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:23:13.143061Z",
     "start_time": "2018-04-25T06:23:13.138715Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'40035'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num = \"0000000000004003500\"\n",
    "num.strip('0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "adding spaces or other characters, can be accomplished using the \n",
    "- ``center()``\n",
    "- ``ljust()``\n",
    "- ``rjust()`` \n",
    "\n",
    "For example, we can use the ``center()`` method \n",
    "- to center a given string within a given number of spaces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'     this is the content      '"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = \"this is the content\"\n",
    "line.center(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similarly, ``ljust()`` and ``rjust()`` will left-justify or right-justify the string within spaces of a given length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is the content           '"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.ljust(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'           this is the content'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.rjust(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "All these methods additionally accept any character which will be used to fill the space.\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:27:07.940615Z",
     "start_time": "2018-04-25T06:27:07.935900Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'00435'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'435'.rjust(5, '0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Because zero-filling is such a common need, \n",
    "- Python also provides ``zfill()``, \n",
    "    - which is a special method to right-pad a string with zeros:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0000000435'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'435'.zfill(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding and replacing substrings\n",
    "\n",
    "If you want to find occurrences of a certain character in a string, the `\n",
    "- ``find()``/``rfind()``\n",
    "- ``index()``/``rindex()``\n",
    "- ``replace()`` \n",
    "\n",
    "``find()`` and ``index()`` are very similar, in that they search for the first occurrence of a character or substring within a string, and return the index of the substring:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:29:54.276768Z",
     "start_time": "2018-04-25T06:29:54.272249Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = 'the quick brown fox jumped over a lazy dog and another yellow fox!'\n",
    "line.find('fox')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:30:28.663087Z",
     "start_time": "2018-04-25T06:30:28.658937Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.index('fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The only difference between ``find()`` and ``index()``\n",
    "- when the search string is not found; \n",
    "    - ``find()`` returns ``-1``, while \n",
    "    - ``index()`` raises a ``ValueError``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.find('bear')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "substring not found",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-21-4cbe6ee9b0eb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bear'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: substring not found"
     ]
    }
   ],
   "source": [
    "line.index('bear')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The related ``rfind()`` and ``rindex()`` work similarly, \n",
    "- except they search for the first occurrence **from the end of the string**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:31:57.040993Z",
     "start_time": "2018-04-25T06:31:57.036897Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.rfind('fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For the special case of checking for a substring **at the beginning or end of a string**, \n",
    "- Python provides the ``startswith()`` and ``endswith()`` methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:33:06.991889Z",
     "start_time": "2018-04-25T06:33:06.988291Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.endswith('fox!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:33:44.288231Z",
     "start_time": "2018-04-25T06:33:44.283892Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.startswith('the')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To replace a given substring with a new string, \n",
    "- you can use the ``replace()`` method.\n",
    "\n",
    "Here, let's replace ``'brown'`` with ``'red'``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:36:21.039443Z",
     "start_time": "2018-04-25T06:36:21.035317Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the quick brown fox jumped over a lazy pig and another yellow fox!'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.replace('dog', 'pig')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The ``replace()`` function returns a new string, and will replace all occurrences of the input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the quick br--wn f--x jumped --ver a lazy d--g'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.replace('o', '--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For a more flexible approach to this ``replace()`` functionality, see the discussion of regular expressions in [Flexible Pattern Matching with Regular Expressions](#Flexible-Pattern-Matching-with-Regular-Expressions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Splitting and partitioning strings\n",
    "\n",
    "If you would like to find a substring *and then* split the string based on its location, \n",
    "- ``partition()`` and/or ``split()`` methods \n",
    "    - Both will return a sequence of substrings.\n",
    "\n",
    "The ``partition()`` method returns a tuple with three elements: \n",
    "- the substring before the first instance of the split-point, \n",
    "- the split-point itself, and \n",
    "- the substring after:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:39:25.165089Z",
     "start_time": "2018-04-25T06:39:25.160672Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('the quick brown ', 'fox', ' jumped over a lazy dog and another yellow fox!')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.partition('fox')\n",
    "#line.rpartition('fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The ``rpartition()`` method is similar, but searches from the right of the string.\n",
    "\n",
    "The ``split()`` method \n",
    "- finds *all* instances of the split-point and returns the substrings **in between**.\n",
    "- The default is to split on any whitespace, returning a list of the individual words in a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:40:55.476025Z",
     "start_time": "2018-04-25T06:40:55.471633Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the quick brown ', ' jumped over a lazy dog and another yellow ', '!']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#line.split()\n",
    "line.split('fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A related method is ``splitlines()``, \n",
    "- which splits on newline characters.\n",
    "\n",
    "Let's do this with a Haiku, popularly attributed to the 17th-century poet Matsuo Bashō:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['matsushima-ya', 'aah matsushima-ya', 'matsushima-ya']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "haiku = \"\"\"matsushima-ya\n",
    "aah matsushima-ya\n",
    "matsushima-ya\"\"\"\n",
    "\n",
    "haiku.splitlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note that if you would like to undo a ``split()``, \n",
    "- you can use the ``join()`` method, \n",
    "    - which returns a string built from a splitpoint and an iterable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1--2--3'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'--'.join(['1', '2', '3'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A common pattern is to use the special character ``\"\\n\"`` (newline) to join together lines that have been previously split, and recover the input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:47:17.726739Z",
     "start_time": "2018-04-25T06:47:17.723201Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matsushima-ya\n",
      "aah matsushima-ya\n",
      "matsushima-ya\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(['matsushima-ya', 'aah matsushima-ya', 'matsushima-ya']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Format Strings\n",
    "\n",
    "to manipulate string *representations* of values of other types.\n",
    "- using the ``str()`` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:49:07.337106Z",
     "start_time": "2018-04-25T06:49:07.332739Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.14159'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pi = 3.14159\n",
    "str(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For more complicated formats, you might be tempted to use string arithmetic as outlined in [Basic Python Semantics: Operators](04-Semantics-Operators.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T06:49:33.501168Z",
     "start_time": "2018-04-25T06:49:33.497102Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The value of pi is 3.14159'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"The value of pi is\" + ' ' + str(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A more flexible way to do this is to use *format strings*, \n",
    "- which are strings with special markers (noted by curly braces) into which string-formatted values will be inserted.\n",
    "\n",
    "Here is a basic example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The value of pi is 3.14159'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"The value of pi is {}\".format(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Inside the ``{}`` marker you can also include information on exactly *what* you would like to appear there.\n",
    "\n",
    "If you include a number, it will refer to the index of the argument to insert:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'First letter: A. Last letter: Z.'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"First letter: {0}. Last letter: {1}.\"\"\".format('A', 'Z')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you include a string, it will refer to the key of any keyword argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'First letter: A. Last letter: Z.'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"First letter: {first}. Last letter: {last}.\"\"\".format(last='Z', first='A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For numerical inputs, you can include format codes which control how the value is converted to a string.\n",
    "\n",
    "For example, to print a number as a floating point with three digits after the decimal point, you can use the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pi = 3.142'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"pi = {0:.3f}\".format(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- The \"``0``\" refers to the index of the value to be inserted.\n",
    "- The \"``:``\" marks that format codes will follow.\n",
    "- The \"``.3f``\" encodes the desired precision: three digits beyond the decimal point, floating-point format.\n",
    "\n",
    "For more information on the syntax of these format strings, see the [Format Specification](https://docs.python.org/3/library/string.html#formatspec) section of Python's online documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Flexible Pattern Matching with Regular Expressions\n",
    "\n",
    "Even more powerful tools are available in Python's built-in *regular expression* module.\n",
    "\n",
    "Regular expressions are a huge topic; there are there are entire books written on the topic (including Jeffrey E.F. Friedl’s [*Mastering Regular Expressions, 3rd Edition*](http://shop.oreilly.com/product/9780596528126.do)), so it will be hard to do justice within just a single subsection.\n",
    "\n",
    "My goal here is to give you an idea of the types of problems that might be addressed using regular expressions, as well as a basic idea of how to use them in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Flexible Pattern Matching with Regular Expressions\n",
    "\n",
    "\n",
    "I'll suggest some references for learning more in [Further Resources on Regular Expressions](#Further-Resources-on-Regular-Expressions).\n",
    "\n",
    "> Fundamentally, regular expressions are a means of *flexible pattern matching* in strings.\n",
    "\n",
    "If you frequently use the command-line, you are probably familiar with this type of flexible matching with the \"``*``\" character, which acts as a wildcard.\n",
    "\n",
    "For example, we can list all the IPython notebooks (i.e., files with extension *.ipynb*) with \"Python\" in their filename by using the \"``*``\" wildcard to match any characters in between:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:10:21.294518Z",
     "start_time": "2018-04-25T07:10:21.178711Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m01-How-to-Run-Python-Code.ipynb\u001b[m\u001b[m \u001b[31m02-Basic-Python-Syntax.ipynb\u001b[m\u001b[m\r\n"
     ]
    }
   ],
   "source": [
    "!ls *Python*.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# The ``split()`` method \n",
    "\n",
    "> Regular expressions generalize this \"wildcard\" (通配符) idea to a wide range of flexible string-matching sytaxes.\n",
    "\n",
    "- the built-in ``re`` module \n",
    "\n",
    "as a simple example, let's use it to duplicate the functionality of the string ``split()`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'a', 'lazy', 'dog']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "regex = re.compile('\\s+')\n",
    "line = 'the quick brown  fox jumped over a lazy dog'\n",
    "regex.split(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'a', 'lazy', 'dog']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here we've first *compiled* a regular expression, then used it to *split* a string.\n",
    "\n",
    "- Python's ``split()`` method returns a list of all substrings between whitespace, \n",
    "- Regular expression ``split()`` method returns a list of all substrings between matches to the input pattern.\n",
    "    - the input is ``\"\\s+\"``: \"``\\s``\" is a special character that matches any whitespace (space, tab, newline, etc.), and \n",
    "    - the \"``+``\" is a character that indicates *one or more* of the entity **preceding** it.\n",
    "\n",
    "Thus, the regular expression matches any substring consisting of one or more spaces.\n",
    "\n",
    "The ``split()`` method here is basically a convenience routine built upon this *pattern matching* behavior; \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# The ``match()`` method \n",
    "tells you whether **<u>``the beginning of a string``</u>** matches the pattern\n",
    "\n",
    "<span style=\"border-bottom:2px dashed red;font-size:30px;\"> The match( ) methond only check the beginning of a string</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:21:01.970447Z",
     "start_time": "2018-04-25T07:21:01.965604Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(0, 5), match='     '>\n",
      "None\n",
      "<_sre.SRE_Match object; span=(0, 2), match='  '>\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "regex = re.compile('\\s+')\n",
    "\n",
    "for s in [\"     \", \"abc   \", \"  abc\"]:\n",
    "    print(regex.match(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:21:04.967622Z",
     "start_time": "2018-04-25T07:21:04.963055Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "re.compile(r'\\s+', re.UNICODE)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:21:06.440862Z",
     "start_time": "2018-04-25T07:21:06.435346Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'     ' matches\n",
      "'abc ' does not match\n",
      "'  abc' matches\n"
     ]
    }
   ],
   "source": [
    "for s in [\"     \", \"abc \", \"  abc\"]:\n",
    "    if regex.match(s):\n",
    "        print(repr(s), \"matches\")\n",
    "    else:\n",
    "        print(repr(s), \"does not match\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on built-in function repr in module builtins:\n",
      "\n",
      "repr(obj, /)\n",
      "    Return the canonical string representation of the object.\n",
      "    \n",
      "    For many object types, including most builtins, eval(repr(obj)) == obj.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(repr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'  '\n"
     ]
    }
   ],
   "source": [
    "print(repr(\"  \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# The ``search()`` & ``sub()`` method \n",
    "\n",
    "- ``regex.search()`` method operates a lot like ``str.index()`` or ``str.find()``\n",
    "- ``regex.sub()`` method operates much like ``str.replace()``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = 'the quick brown fox jumped over a lazy dog'\n",
    "line.index('fox')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:27:09.572710Z",
     "start_time": "2018-04-25T07:27:09.567431Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16, 19)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile('fox')\n",
    "match = regex.search(line)\n",
    "match.start(), match.end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the quick brown BEAR jumped over a lazy dog'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.replace('fox', 'BEAR')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:28:30.331280Z",
     "start_time": "2018-04-25T07:28:30.327149Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the quick brown BEAR jumped over a lazy dog and another yellow BEAR!'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex.sub('BEAR', line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "With a bit of thought, other native string operations can also be cast as regular expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A more sophisticated example\n",
    "\n",
    "The advantage is that regular expressions offer *far* more flexibility.\n",
    "\n",
    "Here we'll consider a more complicated example: the common task of ``matching email addresses``.\n",
    "\n",
    "- start by simply writing a (somewhat indecipherable) regular expression, \n",
    "- and then walk through what is going on.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:33:39.295007Z",
     "start_time": "2018-04-25T07:33:39.292268Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "email = re.compile('\\w+@\\w+\\.[a-z]{3}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Using this, if we're given a line from a document, we can quickly extract things that look like email addresses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:36:15.740482Z",
     "start_time": "2018-04-25T07:36:15.736000Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wang@nju.edu', 'chengjun@google.com']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"To email chengjun, try chengjun.wang@nju.edu.cn or the older address chengjun@google.com.\"\n",
    "email.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To replace these email addresses with another string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:37:10.308739Z",
     "start_time": "2018-04-25T07:37:10.304594Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'To email chengjun, try --@--.--.cn or the older address --@--.--.'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email.sub('--@--.--', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "However, this regular expression is far too simple.\n",
    "- For example, it only allows addresses made of alphanumeric characters that **end in one of several common domain suffixes**.\n",
    "\n",
    "So, for example, the period used here means that we only find part of the address:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['obama@whitehouse.gov']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email.findall('barack.obama@whitehouse.gov')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This goes to show how unforgiving regular expressions can be if you're not careful!\n",
    "\n",
    "Search around online, you can find some suggestions for regular expressions that will match *all* valid emails.\n",
    "- but beware: they are much more involved than the simple expression used here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Basics of regular expression syntax\n",
    "\n",
    "A bit of familiarity can go a long way: \n",
    "- I will walk through some of the basic constructs here, \n",
    "- and then list some more complete resources from which you can learn more.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Simple strings are matched directly\n",
    "\n",
    "If you build a regular expression on a simple string of characters or digits, it will match that exact string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ion']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile('ion')\n",
    "regex.findall('Great Expectations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Some characters have special meanings\n",
    "\n",
    "While simple letters or numbers are direct matches, there are a handful of characters that have special meanings within regular expressions. They are:\n",
    "```\n",
    ". ^ $ * + ? { } [ ] \\ | ( )\n",
    "```\n",
    "We will discuss the meaning of some of these momentarily.\n",
    "\n",
    "In the meantime, you should know that if you'd like to match any of these characters directly, \n",
    "- you can *escape* them with a back-slash ``\\``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:41:59.656974Z",
     "start_time": "2018-04-25T07:41:59.651810Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['$']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile(r'\\$')\n",
    "regex.findall(\"the cost is $20\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The ``r`` preface in ``r'\\$'`` indicates a *raw string*; \n",
    "- in standard Python strings, the backslash is used to indicate special characters.\n",
    "    - For example, a tab is indicated by ``\"\\t\"``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\tb\tc\n"
     ]
    }
   ],
   "source": [
    "print('a\\tb\\tc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Such substitutions are not made in a `raw string`, such as r'abc'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\\tb\\tc\n"
     ]
    }
   ],
   "source": [
    "print(r'a\\tb\\tc') # r means raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For this reason, whenever you use backslashes in a regular expression, it is good practice to use a raw string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Special characters can match character groups\n",
    "\n",
    "``\"\\\"`` character within regular expressions can escape special characters\n",
    "\n",
    "- ``\"\\w\"``, a special marker matching *any alphanumeric character*. \n",
    "- ``\"\\s\"``, a special marker indicating *any whitespace character*.\n",
    "\n",
    "Putting these together, \n",
    "- we can create a regular expression that will match *any two letters/digits with whitespace between them*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-25T07:47:32.189800Z",
     "start_time": "2018-04-25T07:47:32.185157Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['e f', 'x i', 's 9', 's o']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile(r'\\w\\s\\w') # letter-space-letter\n",
    "regex.findall('the fox is 9 years old')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This example begins to hint at the power and flexibility of regular expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The following table lists a few of these characters that are commonly useful:\n",
    "\n",
    "| Character | Description                 || Character | Description                     |\n",
    "|-----------|-----------------------------||-----------|---------------------------------|\n",
    "| ``\"\\d\"``  | Match any digit             || ``\"\\D\"``  | Match any non-digit             |\n",
    "| ``\"\\s\"``  | Match any whitespace        || ``\"\\S\"``  | Match any non-whitespace        |\n",
    "| ``\"\\w\"``  | Match any alphanumeric char || ``\"\\W\"``  | Match any non-alphanumeric char |\n",
    "\n",
    "This is *not* a comprehensive list or description; for more details, see Python's [regular expression syntax documentation](https://docs.python.org/3/library/re.html#re-syntax)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Square brackets match custom character groups\n",
    "\n",
    "If the built-in character groups aren't specific enough for you, you can use square brackets `[]` to specify any set of characters you're interested in.\n",
    "\n",
    "For example, the following will match any lower-case vowel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['c', 'ns', 'q', '', 'nt', '', 'l']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile('[aeiou]')\n",
    "regex.split('consequential')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similarly, you can use a dash to specify a range: for example, \n",
    "- ``\"[a-z]\"`` will match any lower-case letter, and \n",
    "- ``\"[1-3]\"`` will match any of ``\"1\"``, ``\"2\"``, or ``\"3\"``.\n",
    "\n",
    "For instance, you may need to extract from a document specific numerical codes that consist of a capital letter followed by a digit. You could do this as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G2', 'H6']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile('[A-Z][0-9]')\n",
    "regex.findall('1043879, G2, H6')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Wildcards match repeated characters\n",
    "\n",
    "If you would like to match a string with, say, three alphanumeric characters in a row, it is possible to write, for example, ``\"\\w\\w\\w\"``.\n",
    "\n",
    "Because this is such a common need, there is a specific syntax to match repetitions – curly braces with a number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The', 'qui', 'bro', 'fox']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile(r'\\w{3}')\n",
    "regex.findall('The quick brown fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There are also markers available to match any number of repetitions – for example, \n",
    "- the ``\"+\"`` character will match *one or more* repetitions of what precedes it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The', 'quick', 'brown', 'fox']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex = re.compile(r'\\w+')\n",
    "regex.findall('The quick brown fox')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The following is a table of the repetition markers available for use in regular expressions:\n",
    "\n",
    "| Character | Description | Example |\n",
    "|-----------|-------------|---------|\n",
    "| ``?`` | Match zero or one repetitions of preceding  | ``\"ab?\"`` matches ``\"a\"`` or ``\"ab\"`` |\n",
    "| ``*`` | Match zero or more repetitions of preceding | ``\"ab*\"`` matches ``\"a\"``, ``\"ab\"``, ``\"abb\"``, ``\"abbb\"``... |\n",
    "| ``+`` | Match one or more repetitions of preceding  | ``\"ab+\"`` matches ``\"ab\"``, ``\"abb\"``, ``\"abbb\"``... but not ``\"a\"`` |\n",
    "| ``{n}`` | Match ``n`` repetitions of preeeding | ``\"ab{2}\"`` matches ``\"abb\"`` |\n",
    "| ``{m,n}`` | Match between ``m`` and ``n`` repetitions of preceding | ``\"ab{2,3}\"`` matches ``\"abb\"`` or ``\"abbb\"`` |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "With these basics in mind, let's return to our email address matcher:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "email = re.compile(r'\\w+@\\w+\\.[a-z]{3}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can now understand what this means: \n",
    "- we want one or more alphanumeric character (``\"\\w+\"``) \n",
    "- followed by the *at sign* (``\"@\"``), \n",
    "- followed by one or more alphanumeric character (``\"\\w+\"``), \n",
    "- followed by a period (``\"\\.\"`` – note the need for a backslash escape), \n",
    "- followed by exactly three lower-case letters.\n",
    "\n",
    "If we want to now modify this so that the Obama email address matches, we can do so using the square-bracket notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['barack.obama@whitehouse.gov']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email2 = re.compile(r'[\\w.]+@\\w+\\.[a-z]{3}')\n",
    "email2.findall('barack.obama@whitehouse.gov')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We have changed ``\"\\w+\"`` to ``\"[\\w.]+\"``, \n",
    "- so we will match any alphanumeric character *or* a period.\n",
    "\n",
    "With this more flexible expression, we can match a wider range of email addresses\n",
    "- though still not all \n",
    "    - can you identify other shortcomings of this expression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Parentheses indicate *groups* to extract\n",
    "\n",
    "For compound regular expressions like our email matcher, we often want to extract their components rather than the full match. \n",
    "\n",
    "This can be done using parentheses `()` to *group* the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "email3 = re.compile(r'([\\w.]+)@(\\w+)\\.([a-z]{3})')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('guido', 'python', 'org'), ('guido', 'google', 'com')]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"To email Guido, try guido@python.org or the older address guido@google.com.\"\n",
    "email3.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As we see, this grouping actually extracts a list of the sub-components of the email address.\n",
    "\n",
    "We can go a bit further and *name* the extracted components using the ``\"(?P<name> )\"`` syntax, in which case the groups can be extracted as a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'domain': 'python', 'suffix': 'org', 'user': 'guido'}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email4 = re.compile(r'(?P<user>[\\w.]+)@(?P<domain>\\w+)\\.(?P<suffix>[a-z]{3})')\n",
    "match = email4.match('guido@python.org')\n",
    "match.groupdict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Combining these ideas (as well as some of the powerful regexp syntax that we have not covered here) allows you to flexibly and quickly extract information from strings in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Further Resources on Regular Expressions\n",
    "\n",
    "The above discussion is just a quick (and far from complete) treatment of this large topic.\n",
    "If you'd like to learn more, I recommend the following resources:\n",
    "\n",
    "- [Python's ``re`` package Documentation](https://docs.python.org/3/library/re.html): I find that I promptly forget how to use regular expressions just about every time I use them. Now that I have the basics down, I have found this page to be an incredibly valuable resource to recall what each specific character or sequence means within a regular expression.\n",
    "- [Python's official regular expression HOWTO](https://docs.python.org/3/howto/regex.html): a more narrative approach to regular expressions in Python.\n",
    "- [Mastering Regular Expressions (OReilly, 2006)](http://shop.oreilly.com/product/9780596528126.do) is a 500+ page book on the subject. If you want a really complete treatment of this topic, this is the resource for you.\n",
    "\n",
    "For some examples of string manipulation and regular expressions in action at a larger scale, see [Pandas: Labeled Column-oriented Data](15-Preview-of-Data-Science-Tools.ipynb#Pandas:-Labeled-Column-oriented-Data), where we look at applying these sorts of expressions across *tables* of string data within the Pandas package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Modules and Packages](13-Modules-and-Packages.ipynb) | [Contents](Index.ipynb) | [A Preview of Data Science Tools](15-Preview-of-Data-Science-Tools.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}