Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100755 440 lines (434 sloc) 56.699 kb
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
1 <!DOCTYPE html>
2 <meta charset=utf-8>
81abf54c » Mark Pilgrim
2009-09-18 consistent capitalization
3 <title>Regular expressions - Dive Into Python 3</title>
f6961f7e » mark@atlantis
2009-06-05 build fiddling
4 <!--[if IE]><script src=j/html5.js></script><![endif]-->
9a5aaf5e » Mark Pilgrim
2009-05-21 markup fiddling
5 <link rel=stylesheet href=dip3.css>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
6 <style>
e5b43fb4 » Mark Pilgrim
2009-07-25 wrote advanced-unit-testing chapter, decided to merge it into unit-te…
7 body{counter-reset:h1 5}
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
8 </style>
9a5aaf5e » Mark Pilgrim
2009-05-21 markup fiddling
9 <link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
8912c8ae » Mark Pilgrim
2009-05-29 added print stylesheet
10 <link rel=stylesheet media=print href=print.css>
11 <meta name=viewport content='initial-scale=1.0'>
b1b24100 » Mark Pilgrim
2009-09-12 placeholder text
12 <form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=root value=Search></div></form>
ae146df0 » Mark Pilgrim
2009-06-08 syntax highlighting for everyone!
13 <p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#regular-expressions>Dive Into Python 3</a> <span class=u>&#8227;</span>
49bd08c0 » Mark Pilgrim
2009-06-10 unicode fiddling
14 <p id=level>Difficulty level: <span class=u title=intermediate>&#x2666;&#x2666;&#x2666;&#x2662;&#x2662;</span>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
15 <h1>Regular Expressions</h1>
16 <blockquote class=q>
ae146df0 » Mark Pilgrim
2009-06-08 syntax highlighting for everyone!
17 <p><span class=u>&#x275D;</span> Some people, when confronted with a problem, think &#8220;I know, I&#8217;ll use regular expressions.&#8221; Now they have two problems. <span class=u>&#x275E;</span><br>&mdash; <a href=http://www.jwz.org/hacks/marginal.html>Jamie Zawinski</a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
18 </blockquote>
19 <p id=toc>&nbsp;
20 <h2 id=divingin>Diving In</h2>
66e867b8 » Mark Pilgrim
2010-02-11 this is all Philip's fault
21 <p class=f>Getting a small bit of text out of a large block of text is a challenge. In Python, strings have methods for searching and replacing: <code>index()</code>, <code>find()</code>, <code>split()</code>, <code>count()</code>, <code>replace()</code>, <i class=baa>&amp;</i>c. But these methods are limited to the simplest of cases. For example, the <code>index()</code> method looks for a single, hard-coded substring, and the search is always case-sensitive. To do case-insensitive searches of a string <var>s</var>, you must call <code>s.lower()</code> or <code>s.upper()</code> and make sure your search strings are the appropriate case to match. The <code>replace()</code> and <code>split()</code> methods have the same limitations.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
22 <p>If your goal can be accomplished with string methods, you should use them. They&#8217;re fast and simple and easy to read, and there&#8217;s a lot to be said for fast, simple, readable code. But if you find yourself using a lot of different string functions with <code>if</code> statements to handle special cases, or if you&#8217;re chaining calls to <code>split()</code> and <code>join()</code> to slice-and-dice your strings, you may need to move up to regular expressions.
23 <p>Regular expressions are a powerful and (mostly) standardized way of searching, replacing, and parsing text with complex patterns of characters. Although the regular expression syntax is tight and unlike normal code, the result can end up being <em>more</em> readable than a hand-rolled solution that uses a long chain of string functions. There are even ways of embedding comments within regular expressions, so you can include fine-grained documentation within them.
654b102d » Mark Pilgrim
2009-06-05 you wouldn't believe me if I told you
24 <blockquote class='note compare perl5'>
d160e10b » Mark Pilgrim
2009-07-05 changed Perl 5 reference
25 <p><span class=u>&#x261E;</span>If you&#8217;ve used regular expressions in other languages (like Perl, JavaScript, or PHP), Python&#8217;s syntax will be very familiar. Read the summary of the <a href=http://docs.python.org/dev/library/re.html#module-contents><code>re</code> module</a> to get an overview of the available functions and their arguments.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
26 </blockquote>
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
27 <p class=a>&#x2042;
28
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
29 <h2 id=streetaddresses>Case Study: Street Addresses</h2>
30 <p>This series of examples was inspired by a real-life problem I had in my day job several years ago, when I needed to scrub and standardize street addresses exported from a legacy system before importing them into a newer system. (See, I don&#8217;t just make this stuff up; it&#8217;s actually useful.) This example shows how I approached the problem.
31 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
32 <samp class=p>>>> </samp><kbd class=pp>s = '100 NORTH MAIN ROAD'</kbd>
33 <a><samp class=p>>>> </samp><kbd class=pp>s.replace('ROAD', 'RD.')</kbd> <span class=u>&#x2460;</span></a>
34 <samp class=pp>'100 NORTH MAIN RD.'</samp>
35 <samp class=p>>>> </samp><kbd class=pp>s = '100 NORTH BROAD ROAD'</kbd>
36 <a><samp class=p>>>> </samp><kbd class=pp>s.replace('ROAD', 'RD.')</kbd> <span class=u>&#x2461;</span></a>
37 <samp class=pp>'100 NORTH BRD. RD.'</samp>
38 <a><samp class=p>>>> </samp><kbd class=pp>s[:-4] + s[-4:].replace('ROAD', 'RD.')</kbd> <span class=u>&#x2462;</span></a>
39 <samp class=pp>'100 NORTH BROAD RD.'</samp>
40 <a><samp class=p>>>> </samp><kbd class=pp>import re</kbd> <span class=u>&#x2463;</span></a>
41 <a><samp class=p>>>> </samp><kbd class=pp>re.sub('ROAD$', 'RD.', s)</kbd> <span class=u>&#x2464;</span></a>
42 <samp class=pp>'100 NORTH BROAD RD.'</samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
43 <ol>
44 <li>My goal is to standardize a street address so that <code>'ROAD'</code> is always abbreviated as <code>'RD.'</code>. At first glance, I thought this was simple enough that I could just use the string method <code>replace()</code>. After all, all the data was already uppercase, so case mismatches would not be a problem. And the search string, <code>'ROAD'</code>, was a constant. And in this deceptively simple example, <code>s.replace()</code> does indeed work.
45 <li>Life, unfortunately, is full of counterexamples, and I quickly discovered this one. The problem here is that <code>'ROAD'</code> appears twice in the address, once as part of the street name <code>'BROAD'</code> and once as its own word. The <code>replace()</code> method sees these two occurrences and blindly replaces both of them; meanwhile, I see my addresses getting destroyed.
46 <li>To solve the problem of addresses with more than one <code>'ROAD'</code> substring, you could resort to something like this: only search and replace <code>'ROAD'</code> in the last four characters of the address (<code>s[-4:]</code>), and leave the string alone (<code>s[:-4]</code>). But you can see that this is already getting unwieldy. For example, the pattern is dependent on the length of the string you&#8217;re replacing. (If you were replacing <code>'STREET'</code> with <code>'ST.'</code>, you would need to use <code>s[:-6]</code> and <code>s[-6:].replace(...)</code>.) Would you like to come back in six months and debug this? I know I wouldn&#8217;t.
47 <li>It&#8217;s time to move up to regular expressions. In Python, all functionality related to regular expressions is contained in the <code>re</code> module.
48 <li>Take a look at the first parameter: <code>'ROAD$'</code>. This is a simple regular expression that matches <code>'ROAD'</code> only when it occurs at the end of a string. The <code>$</code> means &#8220;end of the string.&#8221; (There is a corresponding character, the caret <code>^</code>, which means &#8220;beginning of the string.&#8221;) Using the <code>re.sub()</code> function, you search the string <var>s</var> for the regular expression <code>'ROAD$'</code> and replace it with <code>'RD.'</code>. This matches the <code>ROAD</code> at the end of the string <var>s</var>, but does <em>not</em> match the <code>ROAD</code> that&#8217;s part of the word <code>BROAD</code>, because that&#8217;s in the middle of <var>s</var>.
49 </ol>
50 <aside>^ matches the start of a string. $ matches the end of a string.</aside>
51 <p>Continuing with my story of scrubbing addresses, I soon discovered that the previous example, matching <code>'ROAD'</code> at the end of the address, was not good enough, because not all addresses included a street designation at all. Some addresses simply ended with the street name. I got away with it most of the time, but if the street name was <code>'BROAD'</code>, then the regular expression would match <code>'ROAD'</code> at the end of the string as part of the word <code>'BROAD'</code>, which is not what I wanted.
52 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
53 <samp class=p>>>> </samp><kbd class=pp>s = '100 BROAD'</kbd>
54 <samp class=p>>>> </samp><kbd class=pp>re.sub('ROAD$', 'RD.', s)</kbd>
55 <samp class=pp>'100 BRD.'</samp>
56 <a><samp class=p>>>> </samp><kbd class=pp>re.sub('\\bROAD$', 'RD.', s)</kbd> <span class=u>&#x2460;</span></a>
57 <samp class=pp>'100 BROAD'</samp>
58 <a><samp class=p>>>> </samp><kbd class=pp>re.sub(r'\bROAD$', 'RD.', s)</kbd> <span class=u>&#x2461;</span></a>
59 <samp class=pp>'100 BROAD'</samp>
60 <samp class=p>>>> </samp><kbd class=pp>s = '100 BROAD ROAD APT. 3'</kbd>
61 <a><samp class=p>>>> </samp><kbd class=pp>re.sub(r'\bROAD$', 'RD.', s)</kbd> <span class=u>&#x2462;</span></a>
62 <samp class=pp>'100 BROAD ROAD APT. 3'</samp>
63 <a><samp class=p>>>> </samp><kbd class=pp>re.sub(r'\bROAD\b', 'RD.', s)</kbd> <span class=u>&#x2463;</span></a>
64 <samp class=pp>'100 BROAD RD. APT 3'</samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
65 <ol>
66 <li>What I <em>really</em> wanted was to match <code>'ROAD'</code> when it was at the end of the string <em>and</em> it was its own word (and not a part of some larger word). To express this in a regular expression, you use <code>\b</code>, which means &#8220;a word boundary must occur right here.&#8221; In Python, this is complicated by the fact that the <code>'\'</code> character in a string must itself be escaped. This is sometimes referred to as the backslash plague, and it is one reason why regular expressions are easier in Perl than in Python. On the down side, Perl mixes regular expressions with other syntax, so if you have a bug, it may be hard to tell whether it&#8217;s a bug in syntax or a bug in your regular expression.
3b4d43bb » Mark Pilgrim
2009-07-16 fixed FIXMEs
67 <li>To work around the backslash plague, you can use what is called a <i>raw string</i>, by prefixing the string with the letter <code>r</code>. This tells Python that nothing in this string should be escaped; <code>'\t'</code> is a tab character, but <code>r'\t'</code> is really the backslash character <code>\</code> followed by the letter <code>t</code>. I recommend always using raw strings when dealing with regular expressions; otherwise, things get too confusing too quickly (and regular expressions are confusing enough already).
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
68 <li><em>*sigh*</em> Unfortunately, I soon found more cases that contradicted my logic. In this case, the street address contained the word <code>'ROAD'</code> as a whole word by itself, but it wasn&#8217;t at the end, because the address had an apartment number after the street designation. Because <code>'ROAD'</code> isn&#8217;t at the very end of the string, it doesn&#8217;t match, so the entire call to <code>re.sub()</code> ends up replacing nothing at all, and you get the original string back, which is not what you want.
69 <li>To solve this problem, I removed the <code>$</code> character and added another <code>\b</code>. Now the regular expression reads &#8220;match <code>'ROAD'</code> when it&#8217;s a whole word by itself anywhere in the string,&#8221; whether at the end, the beginning, or somewhere in the middle.
70 </ol>
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
71 <p class=a>&#x2042;
72
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
73 <h2 id=romannumerals>Case Study: Roman Numerals</h2>
74 <p>You&#8217;ve most likely seen Roman numerals, even if you didn&#8217;t recognize them. You may have seen them in copyrights of old movies and television shows (&#8220;Copyright <code>MCMXLVI</code>&#8221; instead of &#8220;Copyright <code>1946</code>&#8221;), or on the dedication walls of libraries or universities (&#8220;established <code>MDCCCLXXXVIII</code>&#8221; instead of &#8220;established <code>1888</code>&#8221;). You may also have seen them in outlines and bibliographical references. It&#8217;s a system of representing numbers that really does date back to the ancient Roman empire (hence the name).
75 <p>In Roman numerals, there are seven characters that are repeated and combined in various ways to represent numbers.
76 <ul>
77 <li><code>I = 1</code>
78 <li><code>V = 5</code>
79 <li><code>X = 10</code>
80 <li><code>L = 50</code>
81 <li><code>C = 100</code>
82 <li><code>D = 500</code>
83 <li><code>M = 1000</code>
84 </ul>
85 <p>The following are some general rules for constructing Roman numerals:
86 <ul>
8b7a71d8 » Mark Pilgrim
2009-09-25 make repeated rules uniform between chapters
87 <li>Sometimes characters are additive. <code>I</code> is <code>1</code>, <code>II</code> is <code>2</code>, and <code>III</code> is <code>3</code>. <code>VI</code> is <code>6</code> (literally, &#8220;<code>5</code> and <code>1</code>&#8221;), <code>VII</code> is <code>7</code>, and <code>VIII</code> is <code>8</code>.
88 <li>The tens characters (<code>I</code>, <code>X</code>, <code>C</code>, and <code>M</code>) can be repeated up to three times. At <code>4</code>, you need to subtract from the next highest fives character. You can't represent <code>4</code> as <code>IIII</code>; instead, it is represented as <code>IV</code> (&#8220;<code>1</code> less than <code>5</code>&#8221;). <code>40</code> is written as <code>XL</code> (&#8220;<code>10</code> less than <code>50</code>&#8221;), <code>41</code> as <code>XLI</code>, <code>42</code> as <code>XLII</code>, <code>43</code> as <code>XLIII</code>, and then <code>44</code> as <code>XLIV</code> (&#8220;<code>10</code> less than <code>50</code>, then <code>1</code> less than <code>5</code>&#8221;).
89 <li>Sometimes characters are&hellip; the opposite of additive. By putting certain characters before others, you subtract from the final value. For example, at <code>9</code>, you need to subtract from the next highest tens character: <code>8</code> is <code>VIII</code>, but <code>9</code> is <code>IX</code> (&#8220;<code>1</code> less than <code>10</code>&#8221;), not <code>VIIII</code> (since the <code>I</code> character can not be repeated four times). <code>90</code> is <code>XC</code>, <code>900</code> is <code>CM</code>.
90 <li>The fives characters can not be repeated. <code>10</code> is always represented as <code>X</code>, never as <code>VV</code>. <code>100</code> is always <code>C</code>, never <code>LL</code>.
91 <li>Roman numerals are read left to right, so the order of characters matters very much. <code>DC</code> is <code>600</code>; <code>CD</code> is a completely different number (<code>400</code>, &#8220;<code>100</code> less than <code>500</code>&#8221;). <code>CI</code> is <code>101</code>; <code>IC</code> is not even a valid Roman numeral (because you can't subtract <code>1</code> directly from <code>100</code>; you would need to write it as <code>XCIX</code>, &#8220;<code>10</code> less than <code>100</code>, then <code>1</code> less than <code>10</code>&#8221;).
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
92 </ul>
93 <h3 id=thousands>Checking For Thousands</h3>
94 <p>What would it take to validate that an arbitrary string is a valid Roman numeral? Let&#8217;s take it one digit at a time. Since Roman numerals are always written highest to lowest, let&#8217;s start with the highest: the thousands place. For numbers 1000 and higher, the thousands are represented by a series of <code>M</code> characters.
95 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
96 <samp class=p>>>> </samp><kbd class=pp>import re</kbd>
97 <a><samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?$'</kbd> <span class=u>&#x2460;</span></a>
98 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'M')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
99 <samp>&lt;_sre.SRE_Match object at 0106FB58></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
100 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MM')</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
101 <samp>&lt;_sre.SRE_Match object at 0106C290></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
102 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMM')</kbd> <span class=u>&#x2463;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
103 <samp>&lt;_sre.SRE_Match object at 0106AA38></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
104 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMMM')</kbd> <span class=u>&#x2464;</span></a>
105 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, '')</kbd> <span class=u>&#x2465;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
106 <samp>&lt;_sre.SRE_Match object at 0106F4A8></samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
107 <ol>
108 <li>This pattern has three parts. <code>^</code> matches what follows only at the beginning of the string. If this were not specified, the pattern would match no matter where the <code>M</code> characters were, which is not what you want. You want to make sure that the <code>M</code> characters, if they&#8217;re there, are at the beginning of the string. <code>M?</code> optionally matches a single <code>M</code> character. Since this is repeated three times, you&#8217;re matching anywhere from zero to three <code>M</code> characters in a row. And <code>$</code> matches the end of the string. When combined with the <code>^</code> character at the beginning, this means that the pattern must match the entire string, with no other characters before or after the <code>M</code> characters.
109 <li>The essence of the <code>re</code> module is the <code>search()</code> function, that takes a regular expression (<var>pattern</var>) and a string (<code>'M'</code>) to try to match against the regular expression. If a match is found, <code>search()</code> returns an object which has various methods to describe the match; if no match is found, <code>search()</code> returns <code>None</code>, the Python null value. All you care about at the moment is whether the pattern matches, which you can tell by just looking at the return value of <code>search()</code>. <code>'M'</code> matches this regular expression, because the first optional <code>M</code> matches and the second and third optional <code>M</code> characters are ignored.
110 <li><code>'MM'</code> matches because the first and second optional <code>M</code> characters match and the third <code>M</code> is ignored.
111 <li><code>'MMM'</code> matches because all three <code>M</code> characters match.
112 <li><code>'MMMM'</code> does not match. All three <code>M</code> characters match, but then the regular expression insists on the string ending (because of the <code>$</code> character), and the string doesn&#8217;t end yet (because of the fourth <code>M</code>). So <code>search()</code> returns <code>None</code>.
113 <li>Interestingly, an empty string also matches this regular expression, since all the <code>M</code> characters are optional.
114 </ol>
115 <h3 id=hundreds>Checking For Hundreds</h3>
116 <aside>? makes a pattern optional.</aside>
117 <p>The hundreds place is more difficult than the thousands, because there are several mutually exclusive ways it could be expressed, depending on its value.
118 <ul>
119 <li><code>100 = C</code>
120 <li><code>200 = CC</code>
121 <li><code>300 = CCC</code>
122 <li><code>400 = CD</code>
123 <li><code>500 = D</code>
124 <li><code>600 = DC</code>
125 <li><code>700 = DCC</code>
126 <li><code>800 = DCCC</code>
127 <li><code>900 = CM</code>
128 </ul>
129 <p>So there are four possible patterns:
130 <ul>
131 <li><code>CM</code>
132 <li><code>CD</code>
133 <li>Zero to three <code>C</code> characters (zero if the hundreds place is 0)
134 <li><code>D</code>, followed by zero to three <code>C</code> characters
135 </ul>
136 <p>The last two patterns can be combined:
137 <ul>
138 <li>an optional <code>D</code>, followed by zero to three <code>C</code> characters
139 </ul>
140 <p>This example shows how to validate the hundreds place of a Roman numeral.
141 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
142 <samp class=p>>>> </samp><kbd class=pp>import re</kbd>
143 <a><samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?(CM|CD|D?C?C?C?)$'</kbd> <span class=u>&#x2460;</span></a>
144 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCM')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
145 <samp>&lt;_sre.SRE_Match object at 01070390></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
146 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MD')</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
147 <samp>&lt;_sre.SRE_Match object at 01073A50></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
148 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMMCCC')</kbd> <span class=u>&#x2463;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
149 <samp>&lt;_sre.SRE_Match object at 010748A8></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
150 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMC')</kbd> <span class=u>&#x2464;</span></a>
151 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, '')</kbd> <span class=u>&#x2465;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
152 <samp>&lt;_sre.SRE_Match object at 01071D98></samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
153 <ol>
154 <li>This pattern starts out the same as the previous one, checking for the beginning of the string (<code>^</code>), then the thousands place (<code>M?M?M?</code>). Then it has the new part, in parentheses, which defines a set of three mutually exclusive patterns, separated by vertical bars: <code>CM</code>, <code>CD</code>, and <code>D?C?C?C?</code> (which is an optional <code>D</code> followed by zero to three optional <code>C</code> characters). The regular expression parser checks for each of these patterns in order (from left to right), takes the first one that matches, and ignores the rest.
155 <li><code>'MCM'</code> matches because the first <code>M</code> matches, the second and third <code>M</code> characters are ignored, and the <code>CM</code> matches (so the <code>CD</code> and <code>D?C?C?C?</code> patterns are never even considered). <code>MCM</code> is the Roman numeral representation of <code>1900</code>.
156 <li><code>'MD'</code> matches because the first <code>M</code> matches, the second and third <code>M</code> characters are ignored, and the <code>D?C?C?C?</code> pattern matches <code>D</code> (each of the three <code>C</code> characters are optional and are ignored). <code>MD</code> is the Roman numeral representation of <code>1500</code>.
157 <li><code>'MMMCCC'</code> matches because all three <code>M</code> characters match, and the <code>D?C?C?C?</code> pattern matches <code>CCC</code> (the <code>D</code> is optional and is ignored). <code>MMMCCC</code> is the Roman numeral representation of <code>3300</code>.
158 <li><code>'MCMC'</code> does not match. The first <code>M</code> matches, the second and third <code>M</code> characters are ignored, and the <code>CM</code> matches, but then the <code>$</code> does not match because you&#8217;re not at the end of the string yet (you still have an unmatched <code>C</code> character). The <code>C</code> does <em>not</em> match as part of the <code>D?C?C?C?</code> pattern, because the mutually exclusive <code>CM</code> pattern has already matched.
159 <li>Interestingly, an empty string still matches this pattern, because all the <code>M</code> characters are optional and ignored, and the empty string matches the <code>D?C?C?C?</code> pattern where all the characters are optional and ignored.
160 </ol>
161 <p>Whew! See how quickly regular expressions can get nasty? And you&#8217;ve only covered the thousands and hundreds places of Roman numerals. But if you followed all that, the tens and ones places are easy, because they&#8217;re exactly the same pattern. But let&#8217;s look at another way to express the pattern.
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
162 <p class=a>&#x2042;
163
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
164 <h2 id=nmsyntax>Using The <code>{n,m}</code> Syntax</h2>
165 <aside>{1,4} matches between 1 and 4 occurrences of a pattern.</aside>
166 <p>In the previous section, you were dealing with a pattern where the same character could be repeated up to three times. There is another way to express this in regular expressions, which some people find more readable. First look at the method we already used in the previous example.
167 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
168 <samp class=p>>>> </samp><kbd class=pp>import re</kbd>
169 <samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?$'</kbd>
170 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'M')</kbd> <span class=u>&#x2460;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
171 <samp>&lt;_sre.SRE_Match object at 0x008EE090></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
172 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MM')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
173 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
174 <a><samp>>>> </samp><kbd class=pp>re.search(pattern, 'MMM')</kbd> <span class=u>&#x2462;</span></a>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
175 <samp class=pp>&lt;_sre.SRE_Match object at 0x008EE090></samp>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
176 <a><samp>>>> </samp><kbd class=pp>re.search(pattern, 'MMMM')</kbd> <span class=u>&#x2463;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
177 <samp class=p>>>> </samp></pre>
178 <ol>
179 <li>This matches the start of the string, and then the first optional <code>M</code>, but not the second and third <code>M</code> (but that&#8217;s okay because they&#8217;re optional), and then the end of the string.
180 <li>This matches the start of the string, and then the first and second optional <code>M</code>, but not the third <code>M</code> (but that&#8217;s okay because it&#8217;s optional), and then the end of the string.
181 <li>This matches the start of the string, and then all three optional <code>M</code>, and then the end of the string.
8dbfd70b » Mark Pilgrim
2010-01-11 typo [thanks GG]
182 <li>This matches the start of the string, and then all three optional <code>M</code>, but then does not match the end of the string (because there is still one unmatched <code>M</code>), so the pattern does not match and returns <code>None</code>.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
183 </ol>
184 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
185 <a><samp class=p>>>> </samp><kbd class=pp>pattern = '^M{0,3}$'</kbd> <span class=u>&#x2460;</span></a>
186 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'M')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
187 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
188 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MM')</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
189 <samp>&lt;_sre.SRE_Match object at 0x008EE090></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
190 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMM')</kbd> <span class=u>&#x2463;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
191 <samp>&lt;_sre.SRE_Match object at 0x008EEDA8></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
192 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMMM')</kbd> <span class=u>&#x2464;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
193 <samp>>>> </samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
194 <ol>
195 <li>This pattern says: &#8220;Match the start of the string, then anywhere from zero to three <code>M</code> characters, then the end of the string.&#8221; The 0 and 3 can be any numbers; if you want to match at least one but no more than three <code>M</code> characters, you could say <code>M{1,3}</code>.
196 <li>This matches the start of the string, then one <code>M</code> out of a possible three, then the end of the string.
197 <li>This matches the start of the string, then two <code>M</code> out of a possible three, then the end of the string.
198 <li>This matches the start of the string, then three <code>M</code> out of a possible three, then the end of the string.
199 <li>This matches the start of the string, then three <code>M</code> out of a possible three, but then <em>does not match</em> the end of the string. The regular expression allows for up to only three <code>M</code> characters before the end of the string, but you have four, so the pattern does not match and returns <code>None</code>.
200 </ol>
201 <h3 id=tensandones>Checking For Tens And Ones</h3>
202 <p>Now let&#8217;s expand the Roman numeral regular expression to cover the tens and ones place. This example shows the check for tens.
203 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
204 <samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)$'</kbd>
205 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMXL')</kbd> <span class=u>&#x2460;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
206 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
207 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCML')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
208 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
209 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMLX')</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
210 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
211 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMLXXX')</kbd> <span class=u>&#x2463;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
212 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
213 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMLXXXX')</kbd> <span class=u>&#x2464;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
214 <samp class=p>>>> </samp></pre>
215 <ol>
4b7c4a54 » Mark Pilgrim
2009-09-11 typo
216 <li>This matches the start of the string, then the first optional <code>M</code>, then <code>CM</code>, then <code>XL</code>, then the end of the string. Remember, the <code>(A|B|C)</code> syntax means &#8220;match exactly one of A, B, or C&#8221;. You match <code>XL</code>, so you ignore the <code>XC</code> and <code>L?X?X?X?</code> choices, and then move on to the end of the string. <code>MCMXL</code> is the Roman numeral representation of <code>1940</code>.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
217 <li>This matches the start of the string, then the first optional <code>M</code>, then <code>CM</code>, then <code>L?X?X?X?</code>. Of the <code>L?X?X?X?</code>, it matches the <code>L</code> and skips all three optional <code>X</code> characters. Then you move to the end of the string. <code>MCML</code> is the Roman numeral representation of <code>1950</code>.
218 <li>This matches the start of the string, then the first optional <code>M</code>, then <code>CM</code>, then the optional <code>L</code> and the first optional <code>X</code>, skips the second and third optional <code>X</code>, then the end of the string. <code>MCMLX</code> is the Roman numeral representation of <code>1960</code>.
219 <li>This matches the start of the string, then the first optional <code>M</code>, then <code>CM</code>, then the optional <code>L</code> and all three optional <code>X</code> characters, then the end of the string. <code>MCMLXXX</code> is the Roman numeral representation of <code>1980</code>.
220 <li>This matches the start of the string, then the first optional <code>M</code>, then <code>CM</code>, then the optional <code>L</code> and all three optional <code>X</code> characters, then <em>fails to match</em> the end of the string because there is still one more <code>X</code> unaccounted for. So the entire pattern fails to match, and returns <code>None</code>. <code>MCMLXXXX</code> is not a valid Roman numeral.
221 </ol>
e24a73f7 » Mark Pilgrim
2010-05-12 be more precise about (a|b|c)
222 <aside>(A|B) matches either pattern A or pattern B, but not both.</aside>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
223 <p>The expression for the ones place follows the same pattern. I&#8217;ll spare you the details and show you the end result.
07ced98b » Mark Pilgrim
2009-07-14 class=nd fiddling
224 <pre class='nd screen'>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
225 <samp class=p>>>> </samp><kbd class=pp>pattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)(IX|IV|V?I?I?I?)$'</kbd>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
226 </pre><p>So what does that look like using this alternate <code>{n,m}</code> syntax? This example shows the new syntax.
227 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
228 <samp class=p>>>> </samp><kbd class=pp>pattern = '^M{0,3}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$'</kbd>
229 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MDLV')</kbd> <span class=u>&#x2460;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
230 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
231 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMDCLXVI')</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
232 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
233 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMMDCCCLXXXVIII')</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
234 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
235 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'I')</kbd> <span class=u>&#x2463;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
236 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
237 <ol>
238 <li>This matches the start of the string, then one of a possible three <code>M</code> characters, then <code>D?C{0,3}</code>. Of that, it matches the optional <code>D</code> and zero of three possible <code>C</code> characters. Moving on, it matches <code>L?X{0,3}</code> by matching the optional <code>L</code> and zero of three possible <code>X</code> characters. Then it matches <code>V?I{0,3}</code> by matching the optional <code>V</code> and zero of three possible <code>I</code> characters, and finally the end of the string. <code>MDLV</code> is the Roman numeral representation of <code>1555</code>.
239 <li>This matches the start of the string, then two of a possible three <code>M</code> characters, then the <code>D?C{0,3}</code> with a <code>D</code> and one of three possible <code>C</code> characters; then <code>L?X{0,3}</code> with an <code>L</code> and one of three possible <code>X</code> characters; then <code>V?I{0,3}</code> with a <code>V</code> and one of three possible <code>I</code> characters; then the end of the string. <code>MMDCLXVI</code> is the Roman numeral representation of <code>2666</code>.
240 <li>This matches the start of the string, then three out of three <code>M</code> characters, then <code>D?C{0,3}</code> with a <code>D</code> and three out of three <code>C</code> characters; then <code>L?X{0,3}</code> with an <code>L</code> and three out of three <code>X</code> characters; then <code>V?I{0,3}</code> with a <code>V</code> and three out of three <code>I</code> characters; then the end of the string. <code>MMMDCCCLXXXVIII</code> is the Roman numeral representation of <code>3888</code>, and it&#8217;s the longest Roman numeral you can write without extended syntax.
241 <li>Watch closely. (I feel like a magician. &#8220;Watch closely, kids, I&#8217;m going to pull a rabbit out of my hat.&#8221;) This matches the start of the string, then zero out of three <code>M</code>, then matches <code>D?C{0,3}</code> by skipping the optional <code>D</code> and matching zero out of three <code>C</code>, then matches <code>L?X{0,3}</code> by skipping the optional <code>L</code> and matching zero out of three <code>X</code>, then matches <code>V?I{0,3}</code> by skipping the optional <code>V</code> and matching one out of three <code>I</code>. Then the end of the string. Whoa.
242 </ol>
243 <p>If you followed all that and understood it on the first try, you&#8217;re doing better than I did. Now imagine trying to understand someone else&#8217;s regular expressions, in the middle of a critical function of a large program. Or even imagine coming back to your own regular expressions a few months later. I&#8217;ve done it, and it&#8217;s not a pretty sight.
244 <p>Now let&#8217;s explore an alternate syntax that can help keep your expressions maintainable.
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
245 <p class=a>&#x2042;
246
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
247 <h2 id=verbosere>Verbose Regular Expressions</h2>
248 <p>So far you&#8217;ve just been dealing with what I&#8217;ll call &#8220;compact&#8221; regular expressions. As you&#8217;ve seen, they are difficult to read, and even if you figure out what one does, that&#8217;s no guarantee that you&#8217;ll be able to understand it six months later. What you really need is inline documentation.
249 <p>Python allows you to do this with something called <i>verbose regular expressions</i>. A verbose regular expression is different from a compact regular expression in two ways:
250 <ul>
251 <li>Whitespace is ignored. Spaces, tabs, and carriage returns are not matched as spaces, tabs, and carriage returns. They&#8217;re not matched at all. (If you want to match a space in a verbose regular expression, you&#8217;ll need to escape it by putting a backslash in front of it.)
252 <li>Comments are ignored. A comment in a verbose regular expression is just like a comment in Python code: it starts with a <code>#</code> character and goes until the end of the line. In this case it&#8217;s a comment within a multi-line string instead of within your source code, but it works the same way.
253 </ul>
254 <p>This will be more clear with an example. Let&#8217;s revisit the compact regular expression you&#8217;ve been working with, and make it a verbose regular expression. This example shows how.
255 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
256 <samp class=p>>>> </samp><kbd class=pp>pattern = '''
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
257 ^ # beginning of string
e5b43fb4 » Mark Pilgrim
2009-07-25 wrote advanced-unit-testing chapter, decided to merge it into unit-te…
258 M{0,3} # thousands - 0 to 3 Ms
259 (CM|CD|D?C{0,3}) # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),
260 # or 500-800 (D, followed by 0 to 3 Cs)
261 (XC|XL|L?X{0,3}) # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),
262 # or 50-80 (L, followed by 0 to 3 Xs)
263 (IX|IV|V?I{0,3}) # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),
264 # or 5-8 (V, followed by 0 to 3 Is)
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
265 $ # end of string
654b102d » Mark Pilgrim
2009-06-05 you wouldn't believe me if I told you
266 '''</kbd>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
267 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'M', re.VERBOSE)</kbd> <span class=u>&#x2460;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
268 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
269 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MCMLXXXIX', re.VERBOSE)</kbd> <span class=u>&#x2461;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
270 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
49ec7ca6 » Mark Pilgrim
2009-06-09 finished httplib2-redirects section
271 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'MMMDCCCLXXXVIII', re.VERBOSE)</kbd> <span class=u>&#x2462;</span></a>
67bbe947 » Mark Pilgrim
2009-09-25 markup fiddling, fixed SRE_Match repr
272 <samp>&lt;_sre.SRE_Match object at 0x008EEB48></samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
273 <a><samp class=p>>>> </samp><kbd class=pp>re.search(pattern, 'M')</kbd> <span class=u>&#x2463;</span></a></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
274 <ol>
275 <li>The most important thing to remember when using verbose regular expressions is that you need to pass an extra argument when working with them: <code>re.VERBOSE</code> is a constant defined in the <code>re</code> module that signals that the pattern should be treated as a verbose regular expression. As you can see, this pattern has quite a bit of whitespace (all of which is ignored), and several comments (all of which are ignored). Once you ignore the whitespace and the comments, this is exactly the same regular expression as you saw in the previous section, but it&#8217;s a lot more readable.
276 <li>This matches the start of the string, then one of a possible three <code>M</code>, then <code>CM</code>, then <code>L</code> and three of a possible three <code>X</code>, then <code>IX</code>, then the end of the string.
277 <li>This matches the start of the string, then three of a possible three <code>M</code>, then <code>D</code> and three of a possible three <code>C</code>, then <code>L</code> and three of a possible three <code>X</code>, then <code>V</code> and three of a possible three <code>I</code>, then the end of the string.
278 <li>This does not match. Why? Because it doesn&#8217;t have the <code>re.VERBOSE</code> flag, so the <code>re.search</code> function is treating the pattern as a compact regular expression, with significant whitespace and literal hash marks. Python can&#8217;t auto-detect whether a regular expression is verbose or not. Python assumes every regular expression is compact unless you explicitly state that it is verbose.
279 </ol>
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
280 <p class=a>&#x2042;
281
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
282 <h2 id=phonenumbers>Case study: Parsing Phone Numbers</h2>
825254bd » Mark Pilgrim
2010-01-28 fix overzealous global replace
283 <aside>\d matches any numeric digit (0&ndash;9). \D matches anything but digits.</aside>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
284 <p>So far you&#8217;ve concentrated on matching whole patterns. Either the pattern matches, or it doesn&#8217;t. But regular expressions are much more powerful than that. When a regular expression <em>does</em> match, you can pick out specific pieces of it. You can find out what matched where.
285 <p>This example came from another real-world problem I encountered, again from a previous day job. The problem: parsing an American phone number. The client wanted to be able to enter the number free-form (in a single field), but then wanted to store the area code, trunk, number, and optionally an extension separately in the company&#8217;s database. I scoured the Web and found many examples of regular expressions that purported to do this, but none of them were permissive enough.
286 <p>Here are the phone numbers I needed to be able to accept:
287 <ul>
288 <li><code>800-555-1212</code>
289 <li><code>800 555 1212</code>
290 <li><code>800.555.1212</code>
291 <li><code>(800) 555-1212</code>
292 <li><code>1-800-555-1212</code>
293 <li><code>800-555-1212-1234</code>
294 <li><code>800-555-1212x1234</code>
295 <li><code>800-555-1212 ext. 1234</code>
296 <li><code>work 1-(800) 555.1212 #1234</code>
297 </ul>
298 <p>Quite a variety! In each of these cases, I need to know that the area code was <code>800</code>, the trunk was <code>555</code>, and the rest of the phone number was <code>1212</code>. For those with an extension, I need to know that the extension was <code>1234</code>.
299 <p>Let&#8217;s work through developing a solution for phone number parsing. This example shows the first step.
300 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
301 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})$')</kbd> <span class=u>&#x2460;</span></a>
302 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212').groups()</kbd> <span class=u>&#x2461;</span></a>
303 <samp class=pp>('800', '555', '1212')</samp>
304 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212-1234')</kbd> <span class=u>&#x2462;</span></a>
73d314cd » Mark Pilgrim
2009-07-08 typo
305 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212-1234').groups()</kbd> <span class=u>&#x2463;</span></a>
3b96c0f3 » Mark Pilgrim
2009-07-08 added None.groups() example
306 <samp class=traceback>Traceback (most recent call last):
307 File "&lt;stdin>", line 1, in &lt;module>
308 AttributeError: 'NoneType' object has no attribute 'groups'</samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
309 <ol>
43310ed0 » Mark Pilgrim
2009-08-14 markup fiddling
310 <li>Always read regular expressions from left to right. This one matches the beginning of the string, and then <code>(\d{3})</code>. What&#8217;s <code>\d{3}</code>? Well, <code>\d</code> means &#8220;any numeric digit&#8221; (0 through <code>9</code>). The <code>{3}</code> means &#8220;match exactly three numeric digits&#8221;; it&#8217;s a variation on the <a href=#nmsyntax><code>{n,m} syntax</code></a> you saw earlier. Putting it all in parentheses means &#8220;match exactly three numeric digits, <em>and then remember them as a group that I can ask for later</em>&#8221;. Then match a literal hyphen. Then match another group of exactly three digits. Then another literal hyphen. Then another group of exactly four digits. Then match the end of the string.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
311 <li>To get access to the groups that the regular expression parser remembered along the way, use the <code>groups()</code> method on the object that the <code>search()</code> method returns. It will return a tuple of however many groups were defined in the regular expression. In this case, you defined three groups, one with three digits, one with three digits, and one with four digits.
312 <li>This regular expression is not the final answer, because it doesn&#8217;t handle a phone number with an extension on the end. For that, you&#8217;ll need to expand the regular expression.
3b96c0f3 » Mark Pilgrim
2009-07-08 added None.groups() example
313 <li>And this is why you should never &#8220;chain&#8221; the <code>search()</code> and <code>groups()</code> methods in production code. If the <code>search()</code> method returns no matches, it returns <a href=native-datatypes.html#none><code>None</code></a>, not a regular expression match object. Calling <code>None.groups()</code> raises a perfectly obvious exception: <code>None</code> doesn&#8217;t have a <code>groups()</code> method. (Of course, it&#8217;s slightly less obvious when you get this exception from deep within your code. Yes, I speak from experience here.)
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
314 </ol>
315 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
316 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})-(\d+)$')</kbd> <span class=u>&#x2460;</span></a>
317 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212-1234').groups()</kbd> <span class=u>&#x2461;</span></a>
318 <samp class=pp>('800', '555', '1212', '1234')</samp>
319 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800 555 1212 1234')</kbd> <span class=u>&#x2462;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
320 <samp class=p>>>> </samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
321 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212')</kbd> <span class=u>&#x2463;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
322 <samp class=p>>>> </samp></pre>
323 <ol>
324 <li>This regular expression is almost identical to the previous one. Just as before, you match the beginning of the string, then a remembered group of three digits, then a hyphen, then a remembered group of three digits, then a hyphen, then a remembered group of four digits. What&#8217;s new is that you then match another hyphen, and a remembered group of one or more digits, then the end of the string.
325 <li>The <code>groups()</code> method now returns a tuple of four elements, since the regular expression now defines four groups to remember.
326 <li>Unfortunately, this regular expression is not the final answer either, because it assumes that the different parts of the phone number are separated by hyphens. What if they&#8217;re separated by spaces, or commas, or dots? You need a more general solution to match several different types of separators.
327 <li>Oops! Not only does this regular expression not do everything you want, it&#8217;s actually a step backwards, because now you can&#8217;t parse phone numbers <em>without</em> an extension. That&#8217;s not what you wanted at all; if the extension is there, you want to know what it is, but if it&#8217;s not there, you still want to know what the different parts of the main number are.
328 </ol>
329 <p>The next example shows the regular expression to handle separators between the different parts of the phone number.
330 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
331 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'^(\d{3})\D+(\d{3})\D+(\d{4})\D+(\d+)$')</kbd> <span class=u>&#x2460;</span></a>
332 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800 555 1212 1234').groups()</kbd> <span class=u>&#x2461;</span></a>
333 <samp class=pp>('800', '555', '1212', '1234')</samp>
334 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212-1234').groups()</kbd> <span class=u>&#x2462;</span></a>
335 <samp class=pp>('800', '555', '1212', '1234')</samp>
336 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('80055512121234')</kbd> <span class=u>&#x2463;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
337 <samp class=p>>>> </samp>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
338 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212')</kbd> <span class=u>&#x2464;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
339 <samp class=p>>>> </samp></pre>
340 <ol>
341 <li>Hang on to your hat. You&#8217;re matching the beginning of the string, then a group of three digits, then <code>\D+</code>. What the heck is that? Well, <code>\D</code> matches any character <em>except</em> a numeric digit, and <code>+</code> means &#8220;1 or more&#8221;. So <code>\D+</code> matches one or more characters that are not digits. This is what you&#8217;re using instead of a literal hyphen, to try to match different separators.
342 <li>Using <code>\D+</code> instead of <code>-</code> means you can now match phone numbers where the parts are separated by spaces instead of hyphens.
343 <li>Of course, phone numbers separated by hyphens still work too.
344 <li>Unfortunately, this is still not the final answer, because it assumes that there is a separator at all. What if the phone number is entered without any spaces or hyphens at all?
345 <li>Oops! This still hasn&#8217;t fixed the problem of requiring extensions. Now you have two problems, but you can solve both of them with the same technique.
346 </ol>
347 <p>The next example shows the regular expression for handling phone numbers <em>without</em> separators.
348 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
349 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'^(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')</kbd> <span class=u>&#x2460;</span></a>
350 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('80055512121234').groups()</kbd> <span class=u>&#x2461;</span></a>
351 <samp class=pp>('800', '555', '1212', '1234')</samp>
352 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800.555.1212 x1234').groups()</kbd> <span class=u>&#x2462;</span></a>
353 <samp class=pp>('800', '555', '1212', '1234')</samp>
354 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212').groups()</kbd> <span class=u>&#x2463;</span></a>
355 <samp class=pp>('800', '555', '1212', '')</samp>
356 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('(800)5551212 x1234')</kbd> <span class=u>&#x2464;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
357 <samp class=p>>>> </samp></pre>
358 <ol>
359 <li>The only change you&#8217;ve made since that last step is changing all the <code>+</code> to <code>*</code>. Instead of <code>\D+</code> between the parts of the phone number, you now match on <code>\D*</code>. Remember that <code>+</code> means &#8220;1 or more&#8221;? Well, <code>*</code> means &#8220;zero or more&#8221;. So now you should be able to parse phone numbers even when there is no separator character at all.
360 <li>Lo and behold, it actually works. Why? You matched the beginning of the string, then a remembered group of three digits (<code>800</code>), then zero non-numeric characters, then a remembered group of three digits (<code>555</code>), then zero non-numeric characters, then a remembered group of four digits (<code>1212</code>), then zero non-numeric characters, then a remembered group of an arbitrary number of digits (<code>1234</code>), then the end of the string.
361 <li>Other variations work now too: dots instead of hyphens, and both a space and an <code>x</code> before the extension.
362 <li>Finally, you&#8217;ve solved the other long-standing problem: extensions are optional again. If no extension is found, the <code>groups()</code> method still returns a tuple of four elements, but the fourth element is just an empty string.
363 <li>I hate to be the bearer of bad news, but you&#8217;re not finished yet. What&#8217;s the problem here? There&#8217;s an extra character before the area code, but the regular expression assumes that the area code is the first thing at the beginning of the string. No problem, you can use the same technique of &#8220;zero or more non-numeric characters&#8221; to skip over the leading characters before the area code.
364 </ol>
365 <p>The next example shows how to handle leading characters in phone numbers.
366 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
367 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'^\D*(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')</kbd> <span class=u>&#x2460;</span></a>
368 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('(800)5551212 ext. 1234').groups()</kbd> <span class=u>&#x2461;</span></a>
369 <samp class=pp>('800', '555', '1212', '1234')</samp>
370 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212').groups()</kbd> <span class=u>&#x2462;</span></a>
371 <samp class=pp>('800', '555', '1212', '')</samp>
372 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('work 1-(800) 555.1212 #1234')</kbd> <span class=u>&#x2463;</span></a>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
373 <samp class=p>>>> </samp></pre>
374 <ol>
375 <li>This is the same as in the previous example, except now you&#8217;re matching <code>\D*</code>, zero or more non-numeric characters, before the first remembered group (the area code). Notice that you&#8217;re not remembering these non-numeric characters (they&#8217;re not in parentheses). If you find them, you&#8217;ll just skip over them and then start remembering the area code whenever you get to it.
376 <li>You can successfully parse the phone number, even with the leading left parenthesis before the area code. (The right parenthesis after the area code is already handled; it&#8217;s treated as a non-numeric separator and matched by the <code>\D*</code> after the first remembered group.)
377 <li>Just a sanity check to make sure you haven&#8217;t broken anything that used to work. Since the leading characters are entirely optional, this matches the beginning of the string, then zero non-numeric characters, then a remembered group of three digits (<code>800</code>), then one non-numeric character (the hyphen), then a remembered group of three digits (<code>555</code>), then one non-numeric character (the hyphen), then a remembered group of four digits (<code>1212</code>), then zero non-numeric characters, then a remembered group of zero digits, then the end of the string.
378 <li>This is where regular expressions make me want to gouge my eyes out with a blunt object. Why doesn&#8217;t this phone number match? Because there&#8217;s a <code>1</code> before the area code, but you assumed that all the leading characters before the area code were non-numeric characters (<code>\D*</code>). Aargh.
379 </ol>
380 <p>Let&#8217;s back up for a second. So far the regular expressions have all matched from the beginning of the string. But now you see that there may be an indeterminate amount of stuff at the beginning of the string that you want to ignore. Rather than trying to match it all just so you can skip over it, let&#8217;s take a different approach: don&#8217;t explicitly match the beginning of the string at all. This approach is shown in the next example.
381 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
382 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')</kbd> <span class=u>&#x2460;</span></a>
383 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('work 1-(800) 555.1212 #1234').groups()</kbd> <span class=u>&#x2461;</span></a>
384 <samp class=pp>('800', '555', '1212', '1234')</samp>
317fc058 » Mark Pilgrim
2010-08-02 spacing fiddling
385 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212').groups()</kbd> <span class=u>&#x2462;</span></a>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
386 <samp class=pp>('800', '555', '1212', '')</samp>
317fc058 » Mark Pilgrim
2010-08-02 spacing fiddling
387 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('80055512121234').groups()</kbd> <span class=u>&#x2463;</span></a>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
388 <samp class=pp>('800', '555', '1212', '1234')</samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
389 <ol>
390 <li>Note the lack of <code>^</code> in this regular expression. You are not matching the beginning of the string anymore. There&#8217;s nothing that says you need to match the entire input with your regular expression. The regular expression engine will do the hard work of figuring out where the input string starts to match, and go from there.
391 <li>Now you can successfully parse a phone number that includes leading characters and a leading digit, plus any number of any kind of separators around each part of the phone number.
55af56f9 » Mark Pilgrim
2009-04-18 one more typo [h/t G.P.]
392 <li>Sanity check. This still works.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
393 <li>That still works too.
394 </ol>
395 <p>See how quickly a regular expression can get out of control? Take a quick glance at any of the previous iterations. Can you tell the difference between one and the next?
396 <p>While you still understand the final answer (and it is the final answer; if you&#8217;ve discovered a case it doesn&#8217;t handle, I don&#8217;t want to know about it), let&#8217;s write it out as a verbose regular expression, before you forget why you made the choices you made.
397 <pre class=screen>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
398 <samp class=p>>>> </samp><kbd class=pp>phonePattern = re.compile(r'''
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
399 # don't match beginning of string, number can start anywhere
400 (\d{3}) # area code is 3 digits (e.g. '800')
401 \D* # optional separator is any number of non-digits
402 (\d{3}) # trunk is 3 digits (e.g. '555')
403 \D* # optional separator
404 (\d{4}) # rest of number is 4 digits (e.g. '1212')
405 \D* # optional separator
406 (\d*) # extension is optional and can be any number of digits
407 $ # end of string
408 ''', re.VERBOSE)</kbd>
be2b7d35 » Mark Pilgrim
2009-06-08 colorize interactive shell examples
409 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('work 1-(800) 555.1212 #1234').groups()</kbd> <span class=u>&#x2460;</span></a>
410 <samp class=pp>('800', '555', '1212', '1234')</samp>
411 <a><samp class=p>>>> </samp><kbd class=pp>phonePattern.search('800-555-1212')</kbd> <span class=u>&#x2461;</span></a>
412 <samp class=pp>('800', '555', '1212', '')</samp></pre>
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
413 <ol>
414 <li>Other than being spread out over multiple lines, this is exactly the same regular expression as the last step, so it&#8217;s no surprise that it parses the same inputs.
415 <li>Final sanity check. Yes, this still works. You&#8217;re done.
416 </ol>
5b0405f6 » Mark Pilgrim
2009-05-29 asterisms for everyone!
417 <p class=a>&#x2042;
418
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
419 <h2 id=summary>Summary</h2>
420 <p>This is just the tiniest tip of the iceberg of what regular expressions can do. In other words, even though you&#8217;re completely overwhelmed by them now, believe me, you ain&#8217;t seen nothing yet.
421 <p>You should now be familiar with the following techniques:
422 <ul>
423 <li><code>^</code> matches the beginning of a string.
424 <li><code>$</code> matches the end of a string.
425 <li><code>\b</code> matches a word boundary.
426 <li><code>\d</code> matches any numeric digit.
427 <li><code>\D</code> matches any non-numeric character.
428 <li><code>x?</code> matches an optional <code>x</code> character (in other words, it matches an <code>x</code> zero or one times).
429 <li><code>x*</code> matches <code>x</code> zero or more times.
430 <li><code>x+</code> matches <code>x</code> one or more times.
431 <li><code>x{n,m}</code> matches an <code>x</code> character at least <code>n</code> times, but not more than <code>m</code> times.
e24a73f7 » Mark Pilgrim
2010-05-12 be more precise about (a|b|c)
432 <li><code>(a|b|c)</code> matches exactly one of <code>a</code>, <code>b</code> or <code>c</code>.
2489f7d9 » Mark Pilgrim
2009-04-18 fixed several irregularities in regular-expressions chapter (3 Ms ins…
433 <li><code>(x)</code> in general is a <em>remembered group</em>. You can get the value of what matched by using the <code>groups()</code> method of the object returned by <code>re.search</code>.
434 </ul>
435 <p>Regular expressions are extremely powerful, but they are not the correct solution for every problem. You should learn enough about them to know when they are appropriate, when they will solve your problems, and when they will cause more problems than they solve.
e5b43fb4 » Mark Pilgrim
2009-07-25 wrote advanced-unit-testing chapter, decided to merge it into unit-te…
436 <p class=v><a href=strings.html rel=prev title='back to &#8220;Strings&#8221;'><span class=u>&#x261C;</span></a> <a href=generators.html rel=next title='onward to &#8220;Closures &amp; Generators&#8221;'><span class=u>&#x261E;</span></a>
85be3148 » Mark Pilgrim
2011-05-18 switch to Git; update build files and feed links; update copyright year
437 <p class=c>&copy; 2001&ndash;11 <a href=about.html>Mark Pilgrim</a>
e39bffe2 » Mark Pilgrim
2009-06-05 build fiddling
438 <script src=j/jquery.js></script>
ae146df0 » Mark Pilgrim
2009-06-08 syntax highlighting for everyone!
439 <script src=j/prettify.js></script>
e39bffe2 » Mark Pilgrim
2009-06-05 build fiddling
440 <script src=j/dip3.js></script>
Something went wrong with that request. Please try again.