-
Notifications
You must be signed in to change notification settings - Fork 16
/
Copy pathindex.html
354 lines (262 loc) · 18.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="IPython Cookbook, ">
<!-- FAVICON -->
<link rel="apple-touch-icon" sizes="57x57" href="/apple-touch-icon-57x57.png">
<link rel="apple-touch-icon" sizes="114x114" href="/apple-touch-icon-114x114.png">
<link rel="apple-touch-icon" sizes="72x72" href="/apple-touch-icon-72x72.png">
<link rel="apple-touch-icon" sizes="144x144" href="/apple-touch-icon-144x144.png">
<link rel="apple-touch-icon" sizes="60x60" href="/apple-touch-icon-60x60.png">
<link rel="apple-touch-icon" sizes="120x120" href="/apple-touch-icon-120x120.png">
<link rel="apple-touch-icon" sizes="76x76" href="/apple-touch-icon-76x76.png">
<link rel="apple-touch-icon" sizes="152x152" href="/apple-touch-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon-180x180.png">
<link rel="icon" type="image/png" href="/favicon-192x192.png" sizes="192x192">
<link rel="icon" type="image/png" href="/favicon-160x160.png" sizes="160x160">
<link rel="icon" type="image/png" href="/favicon-96x96.png" sizes="96x96">
<link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16">
<link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="msapplication-TileImage" content="/mstile-144x144.png">
<link rel="alternate" href="https://ipython-books.github.io/feeds/all.atom.xml" type="application/atom+xml" title="IPython Cookbook Full Atom Feed"/>
<title>IPython Cookbook - 2.1. Learning the basics of the Unix shell</title>
<link href="//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.2.0/css/font-awesome.min.css" rel="stylesheet">
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/pure/0.3.0/pure-min.css">
<!--[if lte IE 8]>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/pure/0.5.0/pure-min.css">
<![endif]-->
<!--[if gt IE 8]><!-->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/pure/0.5.0/pure-min.css">
<!--<![endif]-->
<link rel="stylesheet" href="https://ipython-books.github.io/theme/css/styles.css">
<link rel="stylesheet" href="https://ipython-books.github.io/theme/css/pygments.css">
<!-- <link href='https://fonts.googleapis.com/css?family=Lato:300,400,700' rel='stylesheet' type='text/css'> -->
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,500" rel="stylesheet" type="text/css">
<link href='https://fonts.googleapis.com/css?family=Ubuntu+Mono' rel='stylesheet' type='text/css'>
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
</head>
<body>
<header id="header" class="pure-g">
<div class="pure-u-1 pure-u-md-3-4">
<div id="menu">
<div class="pure-menu pure-menu-open pure-menu-horizontal">
<ul>
<li><a href="/">home</a></li>
<li><a href="https://github.com/ipython-books/cookbook-2nd-code">Jupyter notebooks</a></li>
<li><a href="https://github.com/ipython-books/minibook-2nd-code">minibook</a></li>
<li><a href="https://cyrille.rossant.net">author</a></li>
</ul> </div>
</div>
</div>
<div class="pure-u-1 pure-u-md-1-4">
<div id="social">
<div class="pure-menu pure-menu-open pure-menu-horizontal">
<ul>
<li><a href="https://twitter.com/cyrillerossant"><i class="fa fa-twitter"></i></a></li>
<li><a href="https://github.com/ipython-books/cookbook-2nd"><i class="fa fa-github"></i></a></li>
</ul> </div>
</div>
</div>
</header>
<div id="layout" class="pure-g">
<section id="content" class="pure-u-1 pure-u-md-4-4">
<div class="l-box">
<header id="page-header">
<h1>2.1. Learning the basics of the Unix shell</h1>
</header>
<section id="page">
<p><a href="/"><img src="https://raw.githubusercontent.com/ipython-books/cookbook-2nd/master/cover-cookbook-2nd.png" align="left" alt="IPython Cookbook, Second Edition" height="130" style="margin-right: 20px; margin-bottom: 10px;" /></a> <em>This is one of the 100+ free recipes of the <a href="/">IPython Cookbook, Second Edition</a>, by <a href="http://cyrille.rossant.net">Cyrille Rossant</a>, a guide to numerical computing and data science in the Jupyter Notebook. The ebook and printed book are available for purchase at <a href="https://www.packtpub.com/big-data-and-business-intelligence/ipython-interactive-computing-and-visualization-cookbook-second-e">Packt Publishing</a>.</em></p>
<p>▶ <em><a href="https://github.com/ipython-books/cookbook-2nd">Text on GitHub</a> with a <a href="https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode">CC-BY-NC-ND license</a></em><br />
▶ <em><a href="https://github.com/ipython-books/cookbook-2nd-code">Code on GitHub</a> with a <a href="https://opensource.org/licenses/MIT">MIT license</a></em></p>
<p>▶ <a href="https://ipython-books.github.io/chapter-2-best-practices-in-interactive-computing/"><strong><em>Go to</em></strong> <em>Chapter 2 : Best practices in Interactive Computing</em></a><br />
▶ <a href="https://github.com/ipython-books/cookbook-2nd-code/blob/master/chapter02_best_practices/01_shell.ipynb"><em><strong>Get</strong> the Jupyter notebook</em></a> </p>
<p>Learning how to interact with the operating system using a command-line interface (or terminal) is a required skill in interactive computing and data analysis. We will use a command-line interface in most recipes of this book. IPython and the Jupyter Notebook are typically launched from a terminal. Installing Python packages is typically done from a terminal.</p>
<p>In this recipe, we will show the very basics of the Unix shell, which is natively available in Linux distributions (such as Debian, Ubuntu, and so on) and macOS. On Windows 10, one can install the <strong>Windows Subsystem for Linux</strong>, a command-line interface to a Unix subsystem integrated with the Windows operating system (see <a href="https://docs.microsoft.com/windows/wsl/about">https://docs.microsoft.com/windows/wsl/about</a>).</p>
<h2>Getting ready</h2>
<p>Here are the instructions to open a Unix shell on macOS, Linux, and Windows. <strong>bash</strong> is the most common Unix shell and this is what we will use in this recipe.</p>
<p>On macOS, bring up the Spotlight Search, type <code>terminal</code>, and press Enter.</p>
<p>On Windows, follow the instructions at <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10.">https://docs.microsoft.com/en-us/windows/wsl/install-win10.</a> Then, open the Windows menu, type <code>bash</code>, and press Enter.</p>
<p>On Linux, open the Dash by clicking on the top-left icon on the desktop, type <code>terminal</code>, and open the <code>Terminal</code> application.</p>
<p>If you want to run this notebook in Jupyter, you need to install <code>bash_kernel</code>, available at <a href="https://github.com/takluyver/bash_kernel.">https://github.com/takluyver/bash_kernel.</a> Open a terminal and type <code>pip install bash_kernel && python -m bash_kernel.install</code>.</p>
<p>This will install a bash kernel in Jupyter, and it will allow you to run this recipe's code directly in the Notebook.</p>
<h2>How to do it...</h2>
<p>The Unix shell comes with hundreds of commands. We will see the most common ones in this recipe.</p>
<p><strong>1. </strong> The terminal lets us write text commands with the keyboard. We execute them by pressing Enter, and the output is displayed below the command. The <strong>working directory</strong> is the directory of our file system that is currently "active" in the terminal. We can get the absolute path of the working directory as follows:</p>
<div class="highlight"><pre><span></span><span class="nb">pwd</span>
</pre></div>
<div class="highlight"><pre><span></span>~/git/cookbook-2nd/chapter02_best_practices
</pre></div>
<blockquote>
<p>the dollar <code>$</code> sign must not be typed: it is typically used by the shell to indicate where the user can start typing. The information written before it may show the user name, the computer name, and part of the working directory. Here, only the three characters <code>pwd</code> should be typed before pressing Enter.</p>
</blockquote>
<p><strong>2. </strong> We can list all files and subdirectories in the working directory as follows:</p>
<div class="highlight"><pre><span></span>ls
</pre></div>
<div class="highlight"><pre><span></span>00_intro.md 03_git.md 07_high_quality.md
01_shell.md 04_git_advanced.md 08_test.md
02_py3 05_workflows.md 09_debugging.md
02_py3.md 06_tips.md images
</pre></div>
<div class="highlight"><pre><span></span>ls -l
</pre></div>
<div class="highlight"><pre><span></span>total 100
-rw-rw-r-- 1 owner 769 Dec 12 10:23 00_intro.md
-rw-rw-r-- 1 owner 2473 Dec 12 14:21 01_shell.md
...
-rw-rw-r-- 1 owner 9390 Dec 12 11:46 08_test.md
-rw-rw-r-- 1 owner 5032 Dec 12 10:23 09_debugging.md
drwxrwxr-x 2 owner 4096 Aug 1 16:49 images
</pre></div>
<p>The <code>-l</code> option displays the directory contents as a detailed list, showing the permissions and owner of the files, the file sizes, and the last modified dates. Most shell commands come with many options that alter their behavior and that can be arbitrarily combined.</p>
<p><strong>3. </strong> We use the <code>cd</code> command to navigate between subdirectories. The current directory is named <code>.</code> (single dot), and the parent directory is named <code>..</code> (double dot):</p>
<div class="highlight"><pre><span></span><span class="nb">cd</span> images
</pre></div>
<div class="highlight"><pre><span></span><span class="nb">pwd</span>
</pre></div>
<div class="highlight"><pre><span></span>~/git/cookbook-2nd/chapter02_best_practices/images
</pre></div>
<div class="highlight"><pre><span></span>ls
</pre></div>
<div class="highlight"><pre><span></span>folder.png github_new.png
</pre></div>
<div class="highlight"><pre><span></span><span class="nb">cd</span> ..
</pre></div>
<div class="highlight"><pre><span></span><span class="nb">pwd</span>
</pre></div>
<div class="highlight"><pre><span></span>~/git/cookbook-2nd/chapter02_best_practices
</pre></div>
<p><strong>4. </strong> Paths can be specified as relative (depending on a reference directory, generally the working directory) or absolute. The <strong>home directory</strong>, specified as <code>~</code>, contains the user's personal files. Configuration files are often stored in a directory like <code>~/.program_name</code>. For example, <code>~/.ipython</code> contains configuration files of IPython:</p>
<div class="highlight"><pre><span></span>ls -la ~/.ipython
</pre></div>
<div class="highlight"><pre><span></span>total 20
drwxr-xr-x 5 cyrille 4096 Nov 14 16:16 .
drwxr-xr-x 93 cyrille 4096 Dec 12 10:50 ..
drwxr-xr-x 2 cyrille 4096 Nov 14 16:16 extensions
drwxr-xr-x 2 cyrille 4096 Nov 14 16:16 nbextensions
drwxr-xr-x 7 cyrille 4096 Dec 12 14:18 profile_default
</pre></div>
<blockquote>
<p>in most terminals, we can use the arrow keys on the keyboard to navigate in the history of past commands. Also, the Tab key enables tab completion, which automatically completes the first characters of a command or a file. For example, typing <code>ls -la ~/.ipy</code> and pressing Tab would automatically complete to <code>ls -la ~/.ipython</code>, or it would present the list of possible options if there are several files or directories that begin with <code>~/.ipy</code>.</p>
</blockquote>
<p><strong>5. </strong> We can create, move, rename, copy, delete files and directories from the terminal:</p>
<div class="highlight"><pre><span></span><span class="c1"># We create an empty directory:</span>
mkdir md_files
<span class="c1"># We copy all Markdown files into the new directory:</span>
cp *.md md_files
<span class="c1"># We rename the directory:</span>
mv md_files markdown_files
ls markdown_files
</pre></div>
<div class="highlight"><pre><span></span>00_intro.md 05_workflows.md
01_shell.md 06_tips.md
02_py3.md 07_high_quality.md
03_git.md 08_test.md
04_git_advanced.md 09_debugging.md
</pre></div>
<div class="highlight"><pre><span></span>rmdir markdown_files
</pre></div>
<div class="highlight"><pre><span></span>rmdir: failed to remove 'markdown_files':
Directory not empty
</pre></div>
<div class="highlight"><pre><span></span>rm markdown_files/*
</pre></div>
<div class="highlight"><pre><span></span>rmdir markdown_files
</pre></div>
<blockquote>
<p>The <code>rm</code> command lets us delete files and directories. The <code>rm -rf path</code> deletes the given path recursively, even if subdirectories are not empty. It is an extremely dangerous command as it cannot be undone: the files are immediately and permanently deleted, they do not go into a trash directory first. See <a href="https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md">https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md</a> for more details.</p>
</blockquote>
<p><strong>6. </strong> There are several useful commands to deal with text files:</p>
<div class="highlight"><pre><span></span><span class="c1"># Show the first three lines of a text file:</span>
head -n <span class="m">3</span> 01_shell.md
</pre></div>
<div class="highlight"><pre><span></span># Learning the basics of the Unix shell
Learning how to interact with the operating system (...)
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># Show the last line of a text file:</span>
tail -n <span class="m">1</span> 00_intro.md
</pre></div>
<div class="highlight"><pre><span></span>We will also cover more general topics (...)
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># We display some text:</span>
<span class="nb">echo</span> <span class="s2">"Hello world!"</span>
</pre></div>
<div class="highlight"><pre><span></span>Hello world!
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># We redirect the output of a command to</span>
<span class="c1"># a text file with `>`:</span>
<span class="nb">echo</span> <span class="s2">"Hello world!"</span> > myfile.txt
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># We display the entire contents of the file:</span>
cat myfile.txt
</pre></div>
<div class="highlight"><pre><span></span>Hello world!
</pre></div>
<p>Several command-line text editors are available, such as <code>pico</code>, <code>nano</code>, or <code>vi</code>. Learning these text editors requires time and effort, especially vi.</p>
<p><strong>7. </strong> The <code>grep</code> command lets us search substrings in text. In the following example, we find all instances of <code>Unix</code> followed by a word (using regular expressions):</p>
<div class="highlight"><pre><span></span>grep -Eo <span class="s2">"Unix \w+"</span> 01_shell.md
</pre></div>
<div class="highlight"><pre><span></span>Unix shell
Unix shell
Unix subsystem
Unix shell
(...)
Unix shell
Unix shell
</pre></div>
<p><strong>8. </strong> A major strength of the Unix shell is that commands can be combined with <strong>pipes</strong>: the output of one command can be directly transferred to the input of another command:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span> <span class="s2">"This is a Unix shell"</span> <span class="p">|</span> grep -Eo <span class="s2">"Unix \w+"</span>
</pre></div>
<div class="highlight"><pre><span></span>Unix shell
</pre></div>
<h2>There's more...</h2>
<p>We only scratched the surface of the Unix shell in this recipe. There are many other commands that can be combined in an infinite number of ways. Many repetitive tasks that would take hours of manual work can be done in a few minutes by writing the appropriate commands. Mastering the Unix shell may take a lot of effort, but it leads to dramatic time gains in the long term.</p>
<p>Here are a few references:</p>
<ul>
<li>Linux tutorial at <a href="https://ryanstutorials.net/linuxtutorial/">https://ryanstutorials.net/linuxtutorial/</a></li>
<li>Bash commands at <a href="https://ss64.com/bash/">https://ss64.com/bash/</a></li>
<li>Learn Bash in Y minutes, at <a href="https://learnxinyminutes.com/docs/bash/">https://learnxinyminutes.com/docs/bash/</a></li>
<li>Learn the shell interactively, at <a href="http://www.learnshell.org/">http://www.learnshell.org/</a></li>
<li>The fish shell, at <a href="https://fishshell.com/">https://fishshell.com/</a></li>
<li>xonsh, a Python-powered shell, at <a href="http://xon.sh/">http://xon.sh/</a></li>
<li>Windows Subsystem for Linux, at <a href="https://docs.microsoft.com/windows/wsl/about">https://docs.microsoft.com/windows/wsl/about</a></li>
</ul>
<h2>See also</h2>
<ul>
<li>Ten tips for conducting reproducible interactive computing experiments</li>
</ul>
</section>
</div>
</section>
<footer id="footer" class="pure-u-1 pure-u-md-4-4">
<div class="l-box">
<div>
<p>© <a href="https://cyrille.rossant.net">Cyrille Rossant</a> –
Built with <a href="https://github.com/PurePelicanTheme/pure-single">Pure Theme</a>
for <a href="https://blog.getpelican.com/">Pelican</a>
</p>
</div>
</div>
</footer>
</div>
<!-- Start of StatCounter Code for Default Guide -->
<script type="text/javascript">
var sc_project=9752080;
var sc_invisible=1;
var sc_security="c177b501";
var scJsHost = (("https:" == document.location.protocol) ?
"https://secure." : "http://www.");
</script>
<script type="text/javascript"
src="https://www.statcounter.com/counter/counter.js"
async></script>
<noscript><div class="statcounter"><a title="Web Analytics"
href="https://statcounter.com/" target="_blank"><img
class="statcounter"
src="//c.statcounter.com/9752080/0/c177b501/1/" alt="Web
Analytics"></a></div></noscript>
<!-- End of StatCounter Code for Default Guide -->
</body>
</html>