Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 292 lines (189 sloc) 6.645 kb
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
1 =====
604c915 @whiteinge Added preliminary xargs presentation
authored
2 xargs
3 =====
4
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
5 :Organization: Ogden Area Linux User Group
6 :Author: Seth House <seth@eseth.com>
7 :Date: 2011-02-22
604c915 @whiteinge Added preliminary xargs presentation
authored
8
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
9 xargs
10 =====
604c915 @whiteinge Added preliminary xargs presentation
authored
11
12 srs? A whole presentation just on xargs?
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
13 ----------------------------------------
14
15 .. container:: r2b-note
16
17 “build and execute command lines from standard input”
18
19 — xargs(1)
20
21 “The traditional Unix operating system kernel has a fixed limit of memory
22 available for program environment and argument list combined.”
23
24 — http://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Argument-list-too-long
25
26 The getconf command may be used to print the currently implemented limit::
27
28 % getconf ARG_MAX
29 2097152
30 % touch {1..314759}
31
32 “The Linux kernel has removed the classic ARG_MAX limitation. See the
33 changeset on git.kernel.org for the change. This was released with the
34 linux-2.6.23 kernel. It will eventually propagate into the release software
35 distributions which include it. Note that glibc's getconf ARG_MAX hasn't
36 caught up yet and still reports a fixed value.”
37
38 — http://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Argument-list-too-long
604c915 @whiteinge Added preliminary xargs presentation
authored
39
40 That’s right.
41
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
42 Getting started
43 ===============
44
604c915 @whiteinge Added preliminary xargs presentation
authored
45 Baby steps
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
46 ----------
604c915 @whiteinge Added preliminary xargs presentation
authored
47
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
48 .. container:: r2b-note
604c915 @whiteinge Added preliminary xargs presentation
authored
49
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
50 ...
604c915 @whiteinge Added preliminary xargs presentation
authored
51
52 If not given a command, ``xargs`` assumes ``echo``::
53
54 ls *.foo | xargs
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
55
56 Output the command to be run::
57
604c915 @whiteinge Added preliminary xargs presentation
authored
58 ls *.foo | xargs -t
59
60 Prompt before running the command::
61
62 ls *.foo | xargs -p
63
64 Why is that useful?
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
65 -------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
66
67 Get a list of process IDs::
68
69 pgrep ssh
70
71 Get info on a list of process IDs::
72
73 pgrep ssh | xargs ps
74
75 Handle arguments in groups
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
76 --------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
77
78 All at once::
79
80 echo 1 2 3 4 | xargs
81
82 One at a time::
83
84 echo 1 2 3 4 | xargs -n1
85
86 In groups of two::
87
88 echo 1 2 3 4 | xargs -n2
89
90 Call programs that take multiple arguments
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
91 ------------------------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
92
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
93 .. container:: r2b-note
604c915 @whiteinge Added preliminary xargs presentation
authored
94
95 ``-I``
96 replace-string
97
98 ::
99
100 ls *.foo | xargs -n1 -I@ cp @ ~/mybackups
101
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
102 Advanced
103 ========
604c915 @whiteinge Added preliminary xargs presentation
authored
104
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
105 On-the-fly shell scripts (that take arguments!)
106 -----------------------------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
107
108 Use ``xargs`` to invoke ``sh -c`` and have access to positional parameters::
109
110 ls *.foo | xargs -n1 sh -c 'echo $1' -
111
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
112 ``-``
113 Pass stdin as an argument to the script. This gives access to
114 shell-script positional parameters like ``$@`` and ``$*``.
115 ``-n1``
116 Handle input one-at-a-time.
117
604c915 @whiteinge Added preliminary xargs presentation
authored
118 Parallel execution
119 ==================
120
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
121 Get the load for several machines
122 ---------------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
123
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
124 ::
604c915 @whiteinge Added preliminary xargs presentation
authored
125
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
126 for ip in 192.168.0.{1..5}; \
127 do ssh $ip uptime; done
604c915 @whiteinge Added preliminary xargs presentation
authored
128
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
129 Get the load for several machines in parallel
130 ---------------------------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
131
132 Get the load for several machines::
133
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
134 echo -n 192.168.0.{1..5} \
135 | xargs -P5 -n1 -d" " -I"HOST" \
136 sh -c 'ssh -T HOST uptime'
604c915 @whiteinge Added preliminary xargs presentation
authored
137
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
138 ``-P``
139 Specify the number of processes to start.
140 Usage ``nproc`` (in coreutil) to get the number of processors.
141 ``-d``
142 Specify the delimeter to look for in the input.
604c915 @whiteinge Added preliminary xargs presentation
authored
143
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
144 Processing a large number of files
145 ----------------------------------
604c915 @whiteinge Added preliminary xargs presentation
authored
146
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
147 Sequentially::
604c915 @whiteinge Added preliminary xargs presentation
authored
148
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
149 for file in songs*.wav ; do lame -f $file $file.mp3
604c915 @whiteinge Added preliminary xargs presentation
authored
150
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
151 In parallel::
604c915 @whiteinge Added preliminary xargs presentation
authored
152
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
153 find . -name "songs*.wav" \
154 | xargs -I@ -P 5 lame -f @ @.mp3
155
156 Parallelize grep
157 ----------------
604c915 @whiteinge Added preliminary xargs presentation
authored
158
159 If you need to grep though a large directory structure it can be parallelized!
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
160 Create one process per CPU, looking at 10 files each process::
604c915 @whiteinge Added preliminary xargs presentation
authored
161
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
162 find /some/path -type f \
163 | xargs -P$(nproc) -n 10 \
164 grep -H 'string-to-search'
604c915 @whiteinge Added preliminary xargs presentation
authored
165
166 Convert several files to pdf simultaneously
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
167 -------------------------------------------
168
169 Sequentially::
170
171 find ./ -name "*.pdf" \
172 | xargs -Istr pdf2ps str
173
174 In parallel::
175
176 find ./ -name "*.pdf" \
177 | xargs -n 8 -Istr pdf2ps str
178
179 Problems with xargs
180 ===================
181
182 Output from multiple processes is not buffered
183 ----------------------------------------------
184
185 Post-process with ``sed``::
604c915 @whiteinge Added preliminary xargs presentation
authored
186
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
187 echo -n 192.168.0.{1..5} \
188 | xargs -P5 -n1 -d" " -I"HOST" \
189 sh -c 'ssh -T HOST uptime \
190 | sed -e "s/^/HOST: /"'
604c915 @whiteinge Added preliminary xargs presentation
authored
191
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
192 Or::
604c915 @whiteinge Added preliminary xargs presentation
authored
193
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
194 echo 1 2 3 \
195 | xargs -P3 -n1 sh -c '$PWD/count.sh $* \
196 | sed "s/^/$$:/"' – | sort
197
198 annotate
199 --------
200
201 ``annotate`` from the ``devscripts`` package::
202
203 #!/bin/sh
204 OUT=`mktemp /tmp/atomic.XXXXXX` || exit 1
205 ERR=`mktemp /tmp/atomic.XXXXXX` || exit 1
206 "$@" >> $OUT 2>> $ERR ; EXIT=$?
207 cat $ERR
208 cat $OUT
209 rm -f $OUT $ERR
210 exit $EXIT
211
212 xargs is not line-oriented
213 --------------------------
214
215 .. container:: r2b-note
216
217 “xargs reads items from the standard input, delimited by blanks (which can
218 be protected with double or single quotes or a backslash) or newlines
219 […] Because Unix filenames can contain blanks and newlines, this default
220 behaviour is often problematic”
221
222 — xargs(1)
223
224 “Some of the UNIX utilities can use NULL as record separator [perl, locate,
225 find, grep, sort]. […] Many UNIX utilities are line oriented. These may
226 work with xargs as long as the lines do not contain ', " or space. […]
227 [M]any UNIX utilities cannot use NULL as separator (e.g. head, tail, ls,
228 echo, sed, tar -v, wc, which).”
229
230 — http://en.wikipedia.org/wiki/Xargs#The_separator_problem
231
232 FIXME: doesn't --delimiter="\n" solve this?
233
234 the first tr is here to terminate lines by null bytes for xargs --null
235 option so that quotes redirection etc are not expansed wrongly::
236
237 tr '\012' '\000' <mycommands.sh | xargs --null --max-procs=$X bash -c
238
239 — http://stackoverflow.com/questions/832253/minimal-task-queue-with-stock-linux-tools-to-leverage-multicore-cpu
240
241 Handle file names with spaces, quotes, or backslashes with the ``-O`` switch.
242
243 ::
244
245 touch important_file
246 touch 'not important_file'
247 ls not* | xargs rm
248 mkdir -p '12" records'
249 ls | xargs rmdir
250
251 GNU parallel
252 ============
253
254 Run jobs in parallel
255 --------------------
256
257 ::
258
259 cat somelist \
260 | parallel do_something \
261 | process_output
262
263 * http://www.gnu.org/software/parallel/
264 * https://build.opensuse.org/project/repositories?project=home:tange
265 * ``man parallel``
266
267 Lightweight job queue
268 ---------------------
269
270 ::
271
272 echo > jobqueue; tail -f jobqueue | parallel
273 echo my_command my_args >> jobqueue
274
275 Run tasks remotely via ssh
276 --------------------------
277
278 ::
279
280 seq 10 | parallel \
281 --sshlogin server1,server2 echo
282
283 Distribute across many machines
284 -------------------------------
285
286 ::
604c915 @whiteinge Added preliminary xargs presentation
authored
287
8b50105 @whiteinge Updated xargs presentation for rst2beamer; added stuff & helper scripts
authored
288 parallel --trc {.}.ogg -j+0 \
289 -S server1,server2,: \
290 'mpg321 -w - {} \
291 | oggenc -q0 - -o {.}.ogg' ::: *.mp3
Something went wrong with that request. Please try again.