Skip to content
Newer
Older
100644 383 lines (326 sloc) 31.2 KB
71821cf stub
Mark Pilgrim authored Jul 25, 2009
1 <!DOCTYPE html>
2 <meta charset=utf-8>
81abf54 consistent capitalization
Mark Pilgrim authored Sep 18, 2009
3 <title>Comprehensions - Dive Into Python 3</title>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
4 <!--[if IE]><script src=j/html5.js></script><![endif]-->
5 <link rel=stylesheet href=dip3.css>
6 <style>
7 body{counter-reset:h1 3}
8 </style>
9 <link rel=stylesheet media='only screen and (max-device-width: 480px)' href=mobile.css>
10 <link rel=stylesheet media=print href=print.css>
11 <meta name=viewport content='initial-scale=1.0'>
b1b2410 placeholder text
Mark Pilgrim authored Sep 12, 2009
12 <form action=http://www.google.com/cse><div><input type=hidden name=cx value=014021643941856155761:l5eihuescdw><input type=hidden name=ie value=UTF-8>&nbsp;<input type=search name=q size=25 placeholder="powered by Google&trade;">&nbsp;<input type=submit name=root value=Search></div></form>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
13 <p>You are here: <a href=index.html>Home</a> <span class=u>&#8227;</span> <a href=table-of-contents.html#comprehensions>Dive Into Python 3</a> <span class=u>&#8227;</span>
14 <p id=level>Difficulty level: <span class=u title=beginner>&#x2666;&#x2666;&#x2662;&#x2662;&#x2662;</span>
15 <h1>Comprehensions</h1>
16 <blockquote class=q>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
17 <p><span class=u>&#x275D;</span> Our imagination is stretched to the utmost, not, as in fiction, to imagine things which are not really there, but just to comprehend those things which are. <span class=u>&#x275E;</span><br>&mdash; <a href=http://en.wikiquote.org/wiki/Richard_Feynman>Richard Feynman</a>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
18 </blockquote>
19 <p id=toc>&nbsp;
20 <h2 id=divingin>Diving In</h2>
66e867b this is all Philip's fault
Mark Pilgrim authored Feb 11, 2010
21 <p class=f>Every programming language has that one feature, a complicated thing intentionally made simple. If you&#8217;re coming from another language, you could easily miss it, because your old language didn&#8217;t make that thing simple (because it was busy making something else simple instead). This chapter will teach you about list comprehensions, dictionary comprehensions, and set comprehensions: three related concepts centered around one very powerful technique. But first, I want to take a little detour into two modules that will help you navigate your local file system.
71821cf stub
Mark Pilgrim authored Jul 25, 2009
22
e088b76 asterisms
Mark Pilgrim authored Oct 6, 2009
23 <p class=a>&#x2042;
24
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
25 <h2 id=os>Working With Files And Directories</h2>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
26
27 <p>Python 3 comes with a module called <code>os</code>, which stands for &#8220;operating system.&#8221; The <a href=http://docs.python.org/3.1/library/os.html><code>os</code> module</a> contains a plethora of functions to get information on&nbsp;&mdash;&nbsp;and in some cases, to manipulate&nbsp;&mdash;&nbsp;local directories, files, processes, and environment variables. Python does its best to offer a unified <abbr>API</abbr> across <a href=installing-python.html>all supported operating systems</a> so your programs can run on any computer with as little platform-specific code as possible.
28
29 <h3 id=getcwd>The Current Working Directory</h3>
30
31 <p>When you&#8217;re just getting started with Python, you&#8217;re going to spend a lot of time in <a href=installing-python.html#idle>the Python Shell</a>. Throughout this book, you will see examples that go like this:
32
33 <ol>
34 <li>Import one of the modules in the <a href=examples/><code>examples</code> folder</a>
35 <li>Call a function in that module
36 <li>Explain the result
37 </ol>
38
727c149 added a few asides
Mark Pilgrim authored Sep 27, 2009
39 <aside>There is always a current working directory.</aside>
40
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
41 <p>If you don&#8217;t know about the current working directory, step 1 will probably fail with an <code>ImportError</code>. Why? Because Python will look for the example module in <a href=your-first-python-program.html#importsearchpath>the import search path</a>, but it won&#8217;t find it because the <code>examples</code> folder isn&#8217;t one of the directories in the search path. To get past this, you can do one of two things:
42
43 <ol>
44 <li>Add the <code>examples</code> folder to the import search path
45 <li>Change the current working directory to the <code>examples</code> folder
46 </ol>
47
48 <p>The current working directory is an invisible property that Python holds in memory at all times. There is always a current working directory, whether you&#8217;re in the Python Shell, running your own Python script from the command line, or running a Python <abbr>CGI</abbr> script on a web server somewhere.
49
50 <p>The <code>os</code> module contains two functions to deal with the current working directory.
51
52 <pre class=screen>
53 <a><samp class=p>>>> </samp><kbd class=pp>import os</kbd> <span class=u>&#x2460;</span></a>
54 <a><samp class=p>>>> </samp><kbd class=pp>print(os.getcwd())</kbd> <span class=u>&#x2461;</span></a>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
55 <samp>C:\Python31</samp>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
56 <a><samp class=p>>>> </samp><kbd class=pp>os.chdir('/Users/pilgrim/diveintopython3/examples')</kbd> <span class=u>&#x2462;</span></a>
57 <a><samp class=p>>>> </samp><kbd class=pp>print(os.getcwd())</kbd> <span class=u>&#x2463;</span></a>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
58 <samp>C:\Users\pilgrim\diveintopython3\examples</samp></pre>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
59 <ol>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
60 <li>The <code>os</code> module comes with Python; you can import it anytime, anywhere.
61 <li>Use the <code>os.getcwd()</code> function to get the current working directory. When you run the graphical Python Shell, the current working directory starts as the directory where the Python Shell executable is. On Windows, this depends on where you installed Python; the default directory is <code>c:\Python31</code>. If you run the Python Shell from the command line, the current working directory starts as the directory you were in when you ran <code>python3</code>.
62 <li>Use the <code>os.chdir()</code> function to change the current working directory.
63 <li>When I called the <code>os.chdir()</code> function, I used a Linux-style pathname (forward slashes, no drive letter) even though I&#8217;m on Windows. This is one of the places where Python tries to paper over the differences between operating systems.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
64 </ol>
65
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
66 <h3 id=ospath>Working With Filenames and Directory Names</h3>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
67
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
68 <p>While we&#8217;re on the subject of directories, I want to point out the <code>os.path</code> module. <code>os.path</code> contains functions for manipulating filenames and directory names.
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
69
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
70 <pre class=screen>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
71 <samp class=p>>>> </samp><kbd class=pp>import os</kbd>
72 <a><samp class=p>>>> </samp><kbd class=pp>print(os.path.join('/Users/pilgrim/diveintopython3/examples/', 'humansize.py'))</kbd> <span class=u>&#x2460;</span></a>
5f50cf5 markup fiddling
Mark Pilgrim authored Aug 16, 2009
73 <samp>/Users/pilgrim/diveintopython3/examples/humansize.py</samp>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
74 <a><samp class=p>>>> </samp><kbd class=pp>print(os.path.join('/Users/pilgrim/diveintopython3/examples', 'humansize.py'))</kbd> <span class=u>&#x2461;</span></a>
5f50cf5 markup fiddling
Mark Pilgrim authored Aug 16, 2009
75 <samp>/Users/pilgrim/diveintopython3/examples\humansize.py</samp>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
76 <a><samp class=p>>>> </samp><kbd class=pp>print(os.path.expanduser('~'))</kbd> <span class=u>&#x2462;</span></a>
5f50cf5 markup fiddling
Mark Pilgrim authored Aug 16, 2009
77 <samp>c:\Users\pilgrim</samp>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
78 <a><samp class=p>>>> </samp><kbd class=pp>print(os.path.join(os.path.expanduser('~'), 'diveintopython3', 'examples', 'humansize.py'))</kbd> <span class=u>&#x2463;</span></a>
5f50cf5 markup fiddling
Mark Pilgrim authored Aug 16, 2009
79 <samp>c:\Users\pilgrim\diveintopython3\examples\humansize.py</samp></pre>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
80 <ol>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
81 <li>The <code>os.path.join()</code> function constructs a pathname out of one or more partial pathnames. In this case, it simply concatenates strings.
44ce5a1 more fiddling
Mark Pilgrim authored May 12, 2010
82 <li>In this slightly less trivial case, calling the <code>os.path.join()</code> function will add an extra slash to the pathname before joining it to the filename. It&#8217;s a backslash instead of a forward slash, because I constructed this example on Windows. If you replicate this example on Linux or Mac OS X, you&#8217;ll see a forward slash instead. Don&#8217;t fuss with slashes; always use <code>os.path.join()</code> and let Python do the right thing.
7db9ef3 clarify some stuff about trailing slashes
Mark Pilgrim authored Aug 16, 2009
83 <li>The <code>os.path.expanduser()</code> function will expand a pathname that uses <code>~</code> to represent the current user&#8217;s home directory. This works on any platform where users have a home directory, including Linux, Mac OS X, and Windows. The returned path does not have a trailing slash, but the <code>os.path.join()</code> function doesn&#8217;t mind.
d11dddf mentioned backslash/forward slash mixing
Mark Pilgrim authored Aug 16, 2009
84 <li>Combining these techniques, you can easily construct pathnames for directories and files in the user&#8217;s home directory. The <code>os.path.join()</code> function can take any number of arguments. I was overjoyed when I discovered this, since <code>addSlashIfNecessary()</code> is one of the stupid little functions I always need to write when building up my toolbox in a new language. <em>Do not</em> write this stupid little function in Python; smart people have already taken care of it for you.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
85 </ol>
86
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
87 <p><code>os.path</code> also contains functions to split full pathnames, directory names, and filenames into their constituent parts.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
88
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
89 <pre class=screen>
90 <samp class=p>>>> </samp><kbd class=pp>pathname = '/Users/pilgrim/diveintopython3/examples/humansize.py'</kbd>
91 <a><samp class=p>>>> </samp><kbd class=pp>os.path.split(pathname)</kbd> <span class=u>&#x2460;</span></a>
92 <samp class=pp>('/Users/pilgrim/diveintopython3/examples', 'humansize.py')</samp>
93 <a><samp class=p>>>> </samp><kbd class=pp>(dirname, filename) = os.path.split(pathname)</kbd> <span class=u>&#x2461;</span></a>
94 <a><samp class=p>>>> </samp><kbd class=pp>dirname</kbd> <span class=u>&#x2462;</span></a>
95 <samp class=pp>'/Users/pilgrim/diveintopython3/examples'</samp>
96 <a><samp class=p>>>> </samp><kbd class=pp>filename</kbd> <span class=u>&#x2463;</span></a>
97 <samp class=pp>'humansize.py'</samp>
98 <a><samp class=p>>>> </samp><kbd class=pp>(shortname, extension) = os.path.splitext(filename)</kbd> <span class=u>&#x2464;</span></a>
99 <samp class=p>>>> </samp><kbd class=pp>shortname</kbd>
100 <samp class=pp>'humansize'</samp>
101 <samp class=p>>>> </samp><kbd class=pp>extension</kbd>
102 <samp class=pp>'.py'</samp></pre>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
103 <ol>
7db9ef3 clarify some stuff about trailing slashes
Mark Pilgrim authored Aug 16, 2009
104 <li>The <code>split</code> function splits a full pathname and returns a tuple containing the path and filename.
105 <li>Remember when I said you could use <a href=native-datatypes.html#multivar>multi-variable assignment</a> to return multiple values from a function? The <code>os.path.split()</code> function does exactly that. You assign the return value of the <code>split</code> function into a tuple of two variables. Each variable receives the value of the corresponding element of the returned tuple.
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
106 <li>The first variable, <var>dirname</var>, receives the value of the first element of the tuple returned from the <code>os.path.split()</code> function, the file path.
107 <li>The second variable, <var>filename</var>, receives the value of the second element of the tuple returned from the <code>os.path.split()</code> function, the filename.
108 <li><code>os.path</code> also contains the <code>os.path.splitext()</code> function, which splits a filename and returns a tuple containing the filename and the file extension. You use the same technique to assign each of them to separate variables.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
109 </ol>
110
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
111 <h3 id=glob>Listing Directories</h3>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
112
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
113 <p>The <code>glob</code> module is another tool in the Python standard library. It&#8217;s an easy way to get the contents of a directory programmatically, and it uses the sort of wildcards that you may already be familiar with from working on the command line.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
114
727c149 added a few asides
Mark Pilgrim authored Sep 28, 2009
115 <aside>The <code>glob</code> module uses shell-like wildcards.</aside>
116
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
117 <pre class=screen>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
118 <samp class=p>>>> </samp><kbd class=pp>os.chdir('/Users/pilgrim/diveintopython3/')</kbd>
119 <samp class=p>>>> </samp><kbd class=pp>import glob</kbd>
120 <a><samp class=p>>>> </samp><kbd class=pp>glob.glob('examples/*.xml')</kbd> <span class=u>&#x2460;</span></a>
121 <samp class=pp>['examples\\feed-broken.xml',
122 'examples\\feed-ns0.xml',
123 'examples\\feed.xml']</samp>
124 <a><samp class=p>>>> </samp><kbd class=pp>os.chdir('examples/')</kbd> <span class=u>&#x2461;</span></a>
125 <a><samp class=p>>>> </samp><kbd class=pp>glob.glob('*test*.py')</kbd> <span class=u>&#x2462;</span></a>
126 <samp class=pp>['alphameticstest.py',
127 'pluraltest1.py',
128 'pluraltest2.py',
129 'pluraltest3.py',
130 'pluraltest4.py',
131 'pluraltest5.py',
132 'pluraltest6.py',
133 'romantest1.py',
134 'romantest10.py',
135 'romantest2.py',
136 'romantest3.py',
137 'romantest4.py',
138 'romantest5.py',
139 'romantest6.py',
140 'romantest7.py',
141 'romantest8.py',
142 'romantest9.py']</samp></pre>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
143 <ol>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
144 <li>The <code>glob</code> module takes a wildcard and returns the path of all files and directories matching the wildcard. In this example, the wildcard is a directory path plus &#8220;<code>*.xml</code>&#8221;, which will match all <code>.xml</code> files in the <code>examples</code> subdirectory.
145 <li>Now change the current working directory to the <code>examples</code> subdirectory. The <code>os.chdir()</code> function can take relative pathnames.
146 <li>You can include multiple wildcards in your glob pattern. This example finds all the files in the current working directory that end in a <code>.py</code> extension and contain the word <code>test</code> anywhere in their filename.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
147 </ol>
148
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
149 <h3 id=osstat>Getting File Metadata</h3>
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
150
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
151 <p>Every modern file system stores metadata about each file: creation date, last-modified date, file size, and so on. Python provides a single <abbr>API</abbr> to access this metadata. You don&#8217;t need to open the file; all you need is the filename.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
152
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
153 <pre class=screen>
154 <samp class=p>>>> </samp><kbd class=pp>import os</kbd>
155 <a><samp class=p>>>> </samp><kbd class=pp>print(os.getcwd())</kbd> <span class=u>&#x2460;</span></a>
156 <samp class=pp>c:\Users\pilgrim\diveintopython3\examples</samp>
157 <a><samp class=p>>>> </samp><kbd class=pp>metadata = os.stat('feed.xml')</kbd> <span class=u>&#x2461;</span></a>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
158 <a><samp class=p>>>> </samp><kbd class=pp>metadata.st_mtime</kbd> <span class=u>&#x2462;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
159 <samp class=pp>1247520344.9537716</samp>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
160 <a><samp class=p>>>> </samp><kbd class=pp>import time</kbd> <span class=u>&#x2463;</span></a>
161 <a><samp class=p>>>> </samp><kbd class=pp>time.localtime(metadata.st_mtime)</kbd> <span class=u>&#x2464;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
162 <samp class=pp>time.struct_time(tm_year=2009, tm_mon=7, tm_mday=13, tm_hour=17,
163 tm_min=25, tm_sec=44, tm_wday=0, tm_yday=194, tm_isdst=1)</samp>
164 </pre>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
165 <ol>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
166 <li>The current working directory is the <code>examples</code> folder.
167 <li><code>feed.xml</code> is a file in the <code>examples</code> folder. Calling the <code>os.stat()</code> function returns an object that contains several different types of metadata about the file.
168 <li><code>st_mtime</code> is the modification time, but it&#8217;s in a format that isn&#8217;t terribly useful. (Technically, it&#8217;s the number of seconds since the Epoch, which is defined as the first second of January 1st, 1970. Seriously.)
169 <li>The <code>time</code> module is part of the Python standard library. It contains functions to convert between different time representations, format time values into strings, and fiddle with timezones.
170 <li>The <code>time.localtime()</code> function converts a time value from seconds-since-the-Epoch (from the <code>st_mtime</code> property returned from the <code>os.stat()</code> function) into a more useful structure of year, month, day, hour, minute, second, and so on. This file was last modified on July 13, 2009, at around 5:25 PM.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
171 </ol>
172
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
173 <pre class=screen>
174 # continued from the previous example
175 <a><samp class=p>>>> </samp><kbd class=pp>metadata.st_size</kbd> <span class=u>&#x2460;</span></a>
176 <samp class=pp>3070</samp>
177 <samp class=p>>>> </samp><kbd class=pp>import humansize</kbd>
8b04ec5 typo
Mark Pilgrim authored Jul 27, 2009
178 <a><samp class=p>>>> </samp><kbd class=pp>humansize.approximate_size(metadata.st_size)</kbd> <span class=u>&#x2461;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
179 <samp class=pp>'3.0 KiB'</samp></pre>
180 <ol>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
181 <li>The <code>os.stat()</code> function also returns the size of a file, in the <code>st_size</code> property. The file <code>feed.xml</code> is <code>3070</code> bytes.
182 <li>You can pass the <code>st_size</code> property to the <a href=your-first-python-program.html#divingin><code>approximate_size()</code> function</a>.
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
183 </ol>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
184
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
185 <h3 id=abspath>Constructing Absolute Pathnames</h3>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
186
6027e4e fixed ref
Mark Pilgrim authored Aug 16, 2009
187 <p>In <a href=#osstat>the previous section</a>, the <code>glob.glob()</code> function returned a list of relative pathnames. The first example had pathnames like <code>'examples\feed.xml'</code>, and the second example had even shorter relative pathnames like <code>'romantest1.py'</code>. As long as you stay in the same current working directory, these relative pathnames will work for opening files or getting file metadata. But if you want to construct an absolute pathname&nbsp;&mdash;&nbsp;<i>i.e.</i> one that includes all the directory names back to the root directory or drive letter&nbsp;&mdash;&nbsp;then you&#8217;ll need the <code>os.path.realpath()</code> function.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
188
189 <pre class=screen>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
190 <samp class=p>>>> </samp><kbd class=pp>import os</kbd>
191 <samp class=p>>>> </samp><kbd class=pp>print(os.getcwd())</kbd>
192 <samp class=pp>c:\Users\pilgrim\diveintopython3\examples</samp>
eab3da2 use realpath instead of abspath
Mark Pilgrim authored Aug 16, 2009
193 <samp class=p>>>> </samp><kbd class=pp>print(os.path.realpath('feed.xml'))</kbd>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
194 <samp class=pp>c:\Users\pilgrim\diveintopython3\examples\feed.xml</samp></pre>
195
e088b76 asterisms
Mark Pilgrim authored Oct 6, 2009
196 <p class=a>&#x2042;
197
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
198 <h2 id=listcomprehension>List Comprehensions</h2>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
199
727c149 added a few asides
Mark Pilgrim authored Sep 28, 2009
200 <aside>You can use any Python expression in a list comprehension.</aside>
201
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
202 <p>A <dfn>list comprehension</dfn> provides a compact way of mapping a list into another list by applying a function to each of the elements of the list.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
203
204 <pre class=screen>
97f2e06 markup fiddling
Mark Pilgrim authored Sep 25, 2009
205 <samp class=p>>>> </samp><kbd class=pp>a_list = [1, 9, 8, 4]</kbd>
206 <a><samp class=p>>>> </samp><kbd class=pp>[elem * 2 for elem in a_list]</kbd> <span class=u>&#x2460;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
207 <samp class=pp>[2, 18, 16, 8]</samp>
97f2e06 markup fiddling
Mark Pilgrim authored Sep 26, 2009
208 <a><samp class=p>>>> </samp><kbd class=pp>a_list</kbd> <span class=u>&#x2461;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
209 <samp class=pp>[1, 9, 8, 4]</samp>
97f2e06 markup fiddling
Mark Pilgrim authored Sep 26, 2009
210 <a><samp class=p>>>> </samp><kbd class=pp>a_list = [elem * 2 for elem in a_list]</kbd> <span class=u>&#x2462;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
211 <samp class=p>>>> </samp><kbd>a_list</kbd>
212 <samp class=pp>[2, 18, 16, 8]</samp></pre>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
213 <ol>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
214 <li>To make sense of this, look at it from right to left. <var>a_list</var> is the list you&#8217;re mapping. The Python interpreter loops through <var>a_list</var> one element at a time, temporarily assigning the value of each element to the variable <var>elem</var>. Python then applies the function <code><var>elem</var> * 2</code> and appends that result to the returned list.
215 <li>A list comprehension creates a new list; it does not change the original list.
216 <li>It is safe to assign the result of a list comprehension to the variable that you&#8217;re mapping. Python constructs the new list in memory, and when the list comprehension is complete, it assigns the result to the original variable.
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
217 </ol>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
218
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
219 <p>You can use any Python expression in a list comprehension, including the functions in the <code>os</code> module for manipulating files and directories.
b964a57 finished #os.path and glob sections
Mark Pilgrim authored Jul 26, 2009
220
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
221 <pre class=screen>
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
222 <samp class=p>>>> </samp><kbd class=pp>import os, glob</kbd>
eab3da2 use realpath instead of abspath
Mark Pilgrim authored Aug 16, 2009
223 <a><samp class=p>>>> </samp><kbd class=pp>glob.glob('*.xml')</kbd> <span class=u>&#x2460;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
224 <samp class=pp>['feed-broken.xml', 'feed-ns0.xml', 'feed.xml']</samp>
eab3da2 use realpath instead of abspath
Mark Pilgrim authored Aug 16, 2009
225 <a><samp class=p>>>> </samp><kbd class=pp>[os.path.realpath(f) for f in glob.glob('*.xml')]</kbd> <span class=u>&#x2461;</span></a>
ceb302f more in comprehensions chapter
Mark Pilgrim authored Jul 26, 2009
226 <samp class=pp>['c:\\Users\\pilgrim\\diveintopython3\\examples\\feed-broken.xml',
227 'c:\\Users\\pilgrim\\diveintopython3\\examples\\feed-ns0.xml',
228 'c:\\Users\\pilgrim\\diveintopython3\\examples\\feed.xml']</samp>
229 </pre>
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
230 <ol>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
231 <li>This returns a list of all the <code>.xml</code> files in the current working directory.
232 <li>This list comprehension takes that list of <code>.xml</code> files and transforms it into a list of full pathnames.
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
233 </ol>
234
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
235 <p>List comprehensions can also filter items, producing a result that can be smaller than the original list.
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
236
237 <pre class=screen>
238 <samp class=p>>>> </samp><kbd class=pp>import os, glob</kbd>
239 <a><samp class=p>>>> </samp><kbd class=pp>[f for f in glob.glob('*.py') if os.stat(f).st_size > 6000]</kbd> <span class=u>&#x2460;</span></a>
240 <samp class=pp>['pluraltest6.py',
241 'romantest10.py',
242 'romantest6.py',
243 'romantest7.py',
244 'romantest8.py',
245 'romantest9.py']</samp>
246 </pre>
247 <ol>
248 <li>To filter a list, you can include an <code>if</code> clause at the end of the list comprehension. The expression after the <code>if</code> keyword will be evaluated for each item in the list. If the expression evaluates to <code>True</code>, the item will be included in the output. This list comprehension looks at the list of all <code>.py</code> files in the current directory, and the <code>if</code> expression filters that list by testing whether the size of each file is greater than <code>6000</code> bytes. There are six such files, so the list comprehension returns a list of six filenames.
249 </ol>
250
251 <p>All the examples of list comprehensions so far have featured simple expressions&nbsp;&mdash;&nbsp;multiply a number by a constant, call a single function, or simply return the original list item (after filtering). But there&#8217;s no limit to how complex a list comprehension can be.
252
253 <pre class=screen>
254 <samp class=p>>>> </samp><kbd class=pp>import os, glob</kbd>
eab3da2 use realpath instead of abspath
Mark Pilgrim authored Aug 16, 2009
255 <a><samp class=p>>>> </samp><kbd class=pp>[(os.stat(f).st_size, os.path.realpath(f)) for f in glob.glob('*.xml')]</kbd> <span class=u>&#x2460;</span></a>
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
256 <samp class=pp>[(3074, 'c:\\Users\\pilgrim\\diveintopython3\\examples\\feed-broken.xml'),
257 (3386, 'c:\\Users\\pilgrim\\diveintopython3\\examples\\feed-ns0.xml'),
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
258 (3070, 'c:\\Users\\pilgrim\\diveintopython3\\examples\\feed.xml')]</samp>
259 <samp class=p>>>> </samp><kbd class=pp>import humansize</kbd>
260 <a><samp class=p>>>> </samp><kbd class=pp>[(humansize.approximate_size(os.stat(f).st_size), f) for f in glob.glob('*.xml')]</kbd> <span class=u>&#x2461;</span></a>
261 <samp class=pp>[('3.0 KiB', 'feed-broken.xml'),
262 ('3.3 KiB', 'feed-ns0.xml'),
263 ('3.0 KiB', 'feed.xml')]</samp></pre>
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
264 <ol>
eab3da2 use realpath instead of abspath
Mark Pilgrim authored Aug 16, 2009
265 <li>This list comprehension finds all the <code>.xml</code> files in the current working directory, gets the size of each file (by calling the <code>os.stat()</code> function), and constructs a tuple of the file size and the absolute path of each file (by calling the <code>os.path.realpath()</code> function).
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
266 <li>This comprehension builds on the previous one to call the <a href=your-first-python-program.html#divingin><code>approximate_size()</code> function</a> with the file size of each <code>.xml</code> file.
4a273cf started list comprehensions
Mark Pilgrim authored Jul 26, 2009
267 </ol>
268
71821cf stub
Mark Pilgrim authored Jul 25, 2009
269 <p class=a>&#x2042;
270
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
271 <h2 id=dictionarycomprehension>Dictionary Comprehensions</h2>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
272
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
273 <p>A <dfn>dictionary comprehension</dfn> is like a list comprehension, but it constructs a dictionary instead of a list.
274
275 <pre class=screen>
276 <samp class=p>>>> </samp><kbd class=pp>import os, glob</kbd>
277 <a><samp class=p>>>> </samp><kbd class=pp>metadata = [(f, os.stat(f)) for f in glob.glob('*test*.py')]</kbd> <span class=u>&#x2460;</span></a>
278 <a><samp class=p>>>> </samp><kbd class=pp>metadata[0]</kbd> <span class=u>&#x2461;</span></a>
279 <samp class=pp>('alphameticstest.py', nt.stat_result(st_mode=33206, st_ino=0, st_dev=0,
280 st_nlink=0, st_uid=0, st_gid=0, st_size=2509, st_atime=1247520344,
281 st_mtime=1247520344, st_ctime=1247520344))</samp>
282 <a><samp class=p>>>> </samp><kbd class=pp>metadata_dict = {f:os.stat(f) for f in glob.glob('*test*.py')}</kbd> <span class=u>&#x2462;</span></a>
283 <a><samp class=p>>>> </samp><kbd class=pp>type(metadata_dict)</kbd> <span class=u>&#x2463;</span></a>
284 <samp>&lt;class 'dict'></samp>
285 <a><samp class=p>>>> </samp><kbd class=pp>list(metadata_dict.keys())</kbd> <span class=u>&#x2464;</span></a>
286 <samp class=pp>['romantest8.py', 'pluraltest1.py', 'pluraltest2.py', 'pluraltest5.py',
287 'pluraltest6.py', 'romantest7.py', 'romantest10.py', 'romantest4.py',
288 'romantest9.py', 'pluraltest3.py', 'romantest1.py', 'romantest2.py',
289 'romantest3.py', 'romantest5.py', 'romantest6.py', 'alphameticstest.py',
290 'pluraltest4.py']</samp>
291 <a><samp class=p>>>> </samp><kbd class=pp>metadata_dict['alphameticstest.py'].st_size</kbd> <span class=u>&#x2465;</span></a>
292 <samp class=pp>2509</samp></pre>
293 <ol>
294 <li>This is not a dictionary comprehension; it&#8217;s a <a href=#listcomprehension>list comprehension</a>. It finds all <code>.py</code> files with <code>test</code> in their name, then constructs a tuple of the filename and the file metadata (from calling the <code>os.stat()</code> function).
295 <li>Each item of the resulting list is a tuple.
296 <li>This is a dictionary comprehension. The syntax is similar to a list comprehension, with two differences. First, it is enclosed in curly braces instead of square brackets. Second, instead of a single expression for each item, it contains two expressions separated by a colon. The expression before the colon (<code>f</code> in this example) is the dictionary key; the expression after the colon (<code>os.stat(f)</code> in this example) is the value.
297 <li>A dictionary comprehension returns a dictionary.
298 <li>The keys of this particular dictionary are simply the filenames returned from the call to <code>glob.glob('*test*.py')</code>.
299 <li>The value associated with each key is the return value from the <code>os.stat()</code> function. That means we can &#8220;look up&#8221; a file by name in this dictionary to get its file metadata. One of the pieces of metadata is <code>st_size</code>, the file size. The file <code>alphameticstest.py</code> is <code>2509</code> bytes long.
300 </ol>
301
302 <p>Like list comprehensions, you can include an <code>if</code> clause in a dictionary comprehension to filter the input sequence based on an expression which is evaluated with each item.
303
304 <pre class=screen>
305 <samp class=p>>>> </samp><kbd class=pp>import os, glob, humansize</kbd>
3937e1f split final dictionary comprehension example into two steps to make t…
Mark Pilgrim authored Sep 17, 2009
306 <a><samp class=p>>>> </samp><kbd class=pp>metadata_dict = {f:os.stat(f) for f in glob.glob('*')}</kbd> <span class=u>&#x2460;</span></a>
307 <a><samp class=p>>>> </samp><kbd class=pp>humansize_dict = {os.path.splitext(f)[0]:humansize.approximate_size(meta.st_size) \ </kbd>
308 <samp class=p>... </samp><kbd class=pp> for f, meta in metadata_dict.items() if meta.st_size > 6000}</kbd> <span class=u>&#x2461;</span></a>
309 <a><samp class=p>>>> </samp><kbd class=pp>list(humansize_dict.keys())</kbd> <span class=u>&#x2462;</span></a>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
310 <samp class=pp>['romantest9', 'romantest8', 'romantest7', 'romantest6', 'romantest10', 'pluraltest6']</samp>
3937e1f split final dictionary comprehension example into two steps to make t…
Mark Pilgrim authored Sep 17, 2009
311 <a><samp class=p>>>> </samp><kbd class=pp>humansize_dict['romantest9']</kbd> <span class=u>&#x2463;</span></a>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
312 <samp class=pp>'6.5 KiB'</samp></pre>
313 <ol>
3937e1f split final dictionary comprehension example into two steps to make t…
Mark Pilgrim authored Sep 17, 2009
314 <li>This dictionary comprehension constructs a list of all the files in the current working directory (<code>glob.glob('*')</code>), gets the file metadata for each file (<code>os.stat(f)</code>), and constructs a dictionary whose keys are filenames and whose values are the metadata for each file.
315 <li>This dictionary comprehension builds on the previous comprehension, filters out files smaller than <code>6000</code> bytes (<code>if meta.st_size > 6000</code>), and uses that filtered list to construct a dictionary whose keys are the filename minus the extension (<code>os.path.splitext(f)[0]</code>) and whose values are the approximate size of each file (<code>humansize.approximate_size(meta.st_size)</code>).
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
316 <li>As you saw in a previous example, there are six such files, thus there are six items in this dictionary.
317 <li>The value of each key is the string returned from the <code>approximate_size()</code> function.
318 </ol>
319
320 <h3 id=stupiddicttricks>Other Fun Stuff To Do With Dictionary Comprehensions</h3>
321
322 <p>Here&#8217;s a trick with dictionary comprehensions that might be useful someday: swapping the keys and values of a dictionary.
323
324 <pre class=screen>
325 <samp class=p>>>> </samp><kbd class=pp>a_dict = {'a': 1, 'b': 2, 'c': 3}</kbd>
326 <samp class=p>>>> </samp><kbd class=pp>{value:key for key, value in a_dict.items()}</kbd>
327 <samp class=pp>{1: 'a', 2: 'b', 3: 'c'}</samp></pre>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
328
f02cd3e add example about trying to swap keys and values in a dictionary with…
Mark Pilgrim authored Nov 6, 2009
329 <p>Of course, this only works if the values of the dictionary are immutable, like strings or tuples. If you try this with a dictionary that contains lists, it will fail most spectacularly.
330
331 <pre class=screen>
332 <samp class=p>>>> </samp><kbd class=pp>a_dict = {'a': [1, 2, 3], 'b': 4, 'c': 5}</kbd>
333 <samp class=p>>>> </samp><kbd class=pp>{value:key for key, value in a_dict.items()}</kbd>
334 <samp class=traceback>Traceback (most recent call last):
335 File "&lt;stdin>", line 1, in &lt;module>
336 File "&lt;stdin>", line 1, in &lt;dictcomp>
337 TypeError: unhashable type: 'list'</samp></pre>
338
71821cf stub
Mark Pilgrim authored Jul 25, 2009
339 <p class=a>&#x2042;
340
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
341 <h2 id=setcomprehension>Set Comprehensions</h2>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
342
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
343 <p>Not to be left out, sets have their own comprehension syntax as well. It is remarkably similar to the syntax for dictionary comprehensions. The only difference is that sets just have values instead of key:value pairs.
344
345 <pre class=screen>
346 <samp class=p>>>> </samp><kbd class=pp>a_set = set(range(10))</kbd>
347 <samp class=p>>>> </samp><kbd class=pp>a_set</kbd>
348 <samp class=pp>{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}</samp>
349 <a><samp class=p>>>> </samp><kbd class=pp>{x ** 2 for x in a_set}</kbd> <span class=u>&#x2460;</span></a>
350 <samp class=pp>{0, 1, 4, 81, 64, 9, 16, 49, 25, 36}</samp>
351 <a><samp class=p>>>> </samp><kbd class=pp>{x for x in a_set if x % 2 == 0}</kbd> <span class=u>&#x2461;</span></a>
352 <samp class=pp>{0, 8, 2, 4, 6}</samp>
353 <a><samp class=p>>>> </samp><kbd class=pp>{2**x for x in range(10)}</kbd> <span class=u>&#x2462;</span></a>
354 <samp class=pp>{32, 1, 2, 4, 8, 64, 128, 256, 16, 512}</samp>
355 </pre>
356 <ol>
43310ed markup fiddling
Mark Pilgrim authored Aug 14, 2009
357 <li>Set comprehensions can take a set as input. This set comprehension calculates the squares of the set of numbers from 0 to <code>9</code>.
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
358 <li>Like list comprehensions and dictionary comprehensions, set comprehensions can contain an <code>if</code> clause to filter each item before returning it in the result set.
359 <li>Set comprehensions do not need to take a set as input; they can take any sequence.
360 </ol>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
361
362 <p class=a>&#x2042;
363
364 <h2 id=furtherreading>Further Reading</h2>
365 <ul>
19a5b7e more stubbing
Mark Pilgrim authored Jul 25, 2009
366 <li><a href=http://docs.python.org/3.1/library/os.html><code>os</code> module</a>
10a56a4 links to PyMOTW
Mark Pilgrim authored Jul 31, 2009
367 <li><a href=http://www.doughellmann.com/PyMOTW/os/><code>os</code>&nbsp;&mdash;&nbsp;Portable access to operating system specific features</a>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
368 <li><a href=http://docs.python.org/3.1/library/os.path.html><code>os.path</code> module</a>
10a56a4 links to PyMOTW
Mark Pilgrim authored Jul 31, 2009
369 <li><a href=http://www.doughellmann.com/PyMOTW/ospath/><code>os.path</code>&nbsp;&mdash;&nbsp;Platform-independent manipulation of file names</a>
ea656b4 typo
Mark Pilgrim authored Aug 5, 2009
370 <li><a href=http://docs.python.org/3.1/library/glob.html><code>glob</code> module</a>
10a56a4 links to PyMOTW
Mark Pilgrim authored Jul 31, 2009
371 <li><a href=http://www.doughellmann.com/PyMOTW/glob/><code>glob</code>&nbsp;&mdash;&nbsp;Filename pattern matching</a>
beeb8c8 link to time module
Mark Pilgrim authored Jul 27, 2009
372 <li><a href=http://docs.python.org/3.1/library/time.html><code>time</code> module</a>
10a56a4 links to PyMOTW
Mark Pilgrim authored Jul 31, 2009
373 <li><a href=http://www.doughellmann.com/PyMOTW/time/><code>time</code>&nbsp;&mdash;&nbsp;Functions for manipulating clock time</a>
463266f finished comprehensions chapter
Mark Pilgrim authored Jul 27, 2009
374 <li><a href=http://docs.python.org/3.1/tutorial/datastructures.html#list-comprehensions>List comprehensions</a>
375 <li><a href=http://docs.python.org/3.1/tutorial/datastructures.html#nested-list-comprehensions>Nested list comprehensions</a>
376 <li><a href=http://docs.python.org/3.1/tutorial/datastructures.html#looping-techniques>Looping techniques</a>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
377 </ul>
378 <p class=v><a href=native-datatypes.html rel=prev title='back to &#8220;Native Datatypes&#8221;'><span class=u>&#x261C;</span></a> <a href=strings.html rel=next title='onward to &#8220;Strings&#8221;'><span class=u>&#x261E;</span></a>
85be314 switch to Git; update build files and feed links; update copyright year
Mark Pilgrim authored May 18, 2011
379 <p class=c>&copy; 2001&ndash;11 <a href=about.html>Mark Pilgrim</a>
71821cf stub
Mark Pilgrim authored Jul 25, 2009
380 <script src=j/jquery.js></script>
381 <script src=j/prettify.js></script>
382 <script src=j/dip3.js></script>
Something went wrong with that request. Please try again.