Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 190 lines (189 sloc) 5.816 kb
4cd60b9 Jan Schaumann Initial import of sources from Yahoo!
authored
1 .\" This manual page was originally written by Jan Schaumann
2 .\" <jschauma@yahoo-inc.com> in May 2009.
3 .Dd January 29, 2011
4 .Dt SCANMASTER 1
5 .Os
6 .Sh NAME
7 .Nm scanmaster
8 .Nd farm scanning jobs out to hosts running scanslave
9 .Sh SYNOPSIS
10 .Nm
11 .Op Fl Hh
12 .Op Fl A Ar authmode
13 .Op Fl a Ar aggregator
14 .Op Fl c Ar post-chunk
15 .Op Fl d Ar dir
16 .Op Fl j Ar jobs
17 .Op Fl p Ar pre
18 .Op Fl w Ar workers
19 .Fl i Ar input
20 .Fl n Ar name
21 .Fl r Ar remote
22 .Op -- scanslave-passthrough-args
23 .Sh DESCRIPTION
24 The
25 .Nm
26 utility is part of the "scanmaster" suite of scripts.
27 It is the primary invoking command and hence named the same.
28 It represents the portion of the process that farms a scan job out to a
29 number of scanning nodes, which in turn invoke
30 .Xr scanslave 1 .
31 .Pp
32 .Nm
33 supports the following command-line options:
34 .Bl -tag -width _aggregator__
35 .It Fl A Ar authmode
36 Specify the authentication mode.
37 Valid options are "all", "password" and "pubkey".
38 This is passed on to the
39 .Xr scanslave 1
40 command.
41 .It Fl H
42 Do not use the headless user for ssh connections.
43 .It Fl a Ar aggregator
44 The path to a script to aggregate the results from all slaves.
45 This script is run after all slaves have finished and it is passed as
46 the first argument the name of the scan, as the second argument the
47 path to the original input file and as the third argument the date
48 (YYYYMMDD) that the scan was started.
49 If you require more flexibility, you may wish to use a wrapper script that
50 invokes
51 .Nm
52 without an aggregator script and the runs whatever other tools you have to
53 post-process the data.
54 .It Fl c Ar post-chunk
55 The script each slave should execute after it's finished.
56 As this flag is passed on to the
57 .Xr scanslave 1
58 invocations, the path needs to be one that all slave nodes can resolve.
59 Using a script on an NFS share is a good solution to this problem.
60 .It Fl d Ar dir
61 The directory under which
62 .Nm
63 should create all output.
64 Preferably this is on an NFS share amongst all scanning nodes, though this
65 is not a technical requirement (it just makes postprocessing of the scan
66 results an order of magnitude easier).
67 Please note that if no NFS share is used, the location still needs to be
68 the same for all slave nodes.
69 If not specified, defaults to "/mnt/scanmaster".
70 .It Fl h
71 Print a short usage and exit.
72 .It Fl i Ar input
73 Specify the file containing the input list of hosts to scan.
74 .It Fl j Ar jobs
75 The number of jobs
76 .Xr scanhosts 1
77 should execute.
78 This flag is passed on to
79 .Xr scanslave 1 ;
80 if not specified, use that tool's defaults.
81 .It Fl n Ar name
82 Specify the name of this scan.
83 This name is used in a number of places during output file creation.
84 .It Fl w Ar workers
85 Specify the file containing the list of worker hosts to farm this job out
86 to.
87 If not specified, defaults to "/usr/local/share/scanmaster/slavenodes".
88 .It Fl p Ar pre
89 Specify the script to run before doing anything else.
90 This script is invoked with the scan name as the single argument.
91 .It Fl r Ar remote
92 Specify the script to execute on all target hosts.
93 As this flag is passed on to the
94 .Xr scanslave 1
95 invocations, the path needs to be one that all slave nodes can resolve.
96 Using a script on an NFS share is a good solution to this problem.
97 .El
98 .Pp
99 Any arguments given after a
100 .Fl -
101 will be treated as "scanslave pass-through" arguments.
102 That is, they will be passed on to
103 .Xr scanslave 1 ,
104 with the
105 .Fl -
106 intact, meaning
107 .Xr scanslave 1
108 will further pass them through to other tools it may invoke.
109 .Sh DETAILS
110 Upon invocation,
111 .Nm
112 will first run the script given to the
113 .Fl p
114 flag (if any), passing it the name of the scan as the single argument.
115 It then removes any files found under the shared directory for a scan by
116 this name, creates a new sub directory (by date), and tries to set up the
117 environment for the currently running (if any)
118 .Xr ssh-agent 1 .
119 Next, it splits the input file into N chunks, where N is the number of
120 hosts found in the worker hosts file.
121 After that, it will ssh to each worker host, create a screenrc file that
122 will invoke
123 .Xr scanslave 1
124 with the appropriate options; it then invokes
125 .Xr screen 1
126 with a key-binding of
127 .Ar ^Ww .
128 The user needs to detach from each host's screen session in order to let
129 .Nm
130 move on to the next host.
131 .Pp
132 After all hosts have started a screen session,
133 .Nm
134 will wait for the scanning nodes to complete their respective jobs.
135 Once every minute,
136 .Nm
137 will look for the existence of a file called ".done" which is created in
138 the well-defined shared location by
139 .Xr scanslave 1 .
140 When it finds all ".done" files, it will execute the script given via the
141 .Ar a
142 flag (if any), passing it the name of the scan as the first
143 argument and the location of the original input file as the second
144 argument.
145 .Sh ENVIRONMENT
146 .Nm
147 uses the following environment variables:
148 .Bl -tag -width SCAN_ENV_
149 .It SCAN_ENV
150 .Nm
151 invokes
152 .Xr scanslave 1
153 by calling
154 .Bd -literal -offset indent
155 env ${SCAN_ENV} scanslave other-args
156 .Ed
157 This allows the user to pass anything into the environment of the invoked
158 processes on the scanslave hosts.
159 Obviously, SCAN_ENV should then contain key=value pairs.
160 .El
161 .Sh EXAMPLES
162 The following examples illustrate common usage of this tool.
163 .Pp
164 .Bd -literal -offset indent
165 SCAN=info1
166 PREFIX=/mnt/scanmaster/jschauma/${SCAN}
167 scanmaster \\
168 -a ${PREFIX}/aggregator.sh \\
169 -c ${PREFIX}/post.chunk.sh \\
170 -i ${PREFIX}/files/${SCAN}-input \\
171 -p ${PREFIX}/pre.sh \\
172 -r ${PREFIX}/remote.sh \\
173 -n ${SCAN}
174 .Ed
175 .Sh SEE ALSO
176 .Xr autopw 1 ,
177 .Xr checkhosts 1 ,
178 .Xr scanhosts 1 ,
179 .Xr scanslave 1 ,
180 .Xr tkill 1
181 .Sh HISTORY
182 The
183 .Nm
184 utility was originally written by
185 .An Jan Schaumann
186 .Aq jschauma@yahoo-inc.com
187 in July 2007 as "cmd.chunker".
188 .Sh BUGS
189 Please report bugs and feature requests to the author.
Something went wrong with that request. Please try again.