Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Initial import of sources from Yahoo!

  • Loading branch information...
commit 4cd60b93b20706abaceb6b09ac2bdd0382c5f530 0 parents
Jan Schaumann authored
81 CHANGES
@@ -0,0 +1,81 @@
+Version 3.1.4 (2011-01-29)
+ * aggregator scripts should be passed date the scan was
+ started
+ * don't use possibly PREFIX variable in the generic aggregator -- it is
+ set in the scanmaster already
+
+Version 3.1.3 (2011-01-28)
+ * remove hardcoded references users, keys, hostnames,
+ pathnames etc.
+
+Version 3.1.2 (2011-01-22)
+ * ensure all output is collected in aggregator
+
+Version 3.1.1 (2011-01-19)
+ * let the aggregator also collect hosts_exit_* files
+
+Version 3.1.0 (2010-12-11)
+ * allow use to specify password-only, pubkey-only or
+ either-or via -A authmode; allow user to set SSHOPTS in environment
+ * allow user to set SSH_ADD_OPTS to influence ssh-add(1) behaviour
+ * allow user to set SCAN_ENV to influence all scanslave(1) processes
+
+Version 3.0.0 (2010-11-23)
+ * use a headless user by default
+
+Version 2.3.3 (2010-08-23)
+ * add generic wrapper scripts
+
+Version 2.3.2 (2010-08-12)
+ * if input has too few lines, don't try to split
+
+Version 2.3.1 (2010-08-04)
+ * invoke tkill every minute; otherwise a hung job will be terminated only
+ after 10 minutes (worst case)
+ * try to kill a possibly running ssh-agent when all is said and done
+ * tkill all processes running for longer than 120 seconds
+
+Version 2.3.0 (2010-07-21)
+ * let scanslave start an ssh-agent;
+ change protocol order to 2,1: not all hosts have an sshv1 key, so
+ attempting sshv1 first with a key and failing will end the attempts
+ * explicitly disable agent forwarding when connecting to the target hosts
+
+Version 2.2.2 (2009-11-07)
+ * remove slave status file after it finished
+
+Version 2.2.1 (2009-11-07)
+ * fix syntax error introduced in 2.2
+
+Version 2.2 (2009-11-07)
+ * add suggestion for a wrapper if aggregator args aren't
+ sufficiently flexible
+ * offer a way to show what slaves are still working (done
+ by way of SIGINFO)
+ * correct use of ssh agent forwarding
+
+Version 2.1 (2009-09-14)
+ * explicitly set StrictHostKeyChecking=no
+ * don't interpret '*' as a filename if not enough hosts
+ are given on input to actually create a split
+ * don't require optional files/arguments
+ * clarify that a number of scripts passed to scanmaster
+ are passed through to all scanslaves
+
+Version 2.0 (2009-08-02)
+ * use , as OFS
+
+Version 1.1 (2009-08-02)
+ * make remote exit codes useful
+ * add examples to scanslave and scanmaster manpages
+ * let scanmaster pass additional flags to scanslave
+
+Version 1.0 (2009-05-18)
+ * run the pre-script before checking for existence of other given files,
+ since the pre-script might generate some of them
+ * don't invoke an ssh-agent; if users want to use an
+ agent, they need to start it on their desktop and forward it
+ (change in behaviour, so bumping major)
+
+Version 0.1 (2009-05-07)
+ * Initial packaged version
29 LICENSE
@@ -0,0 +1,29 @@
+Redistribution and use of this software in source and binary forms,
+with or without modification, are permitted provided that the following
+conditions are met:
+
+* Redistributions of source code must retain the above
+ copyright notice, this list of conditions and the
+ following disclaimer.
+
+* Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the
+ following disclaimer in the documentation and/or other
+ materials provided with the distribution.
+
+* Neither the name of Yahoo! Inc. nor the names of its
+ contributors may be used to endorse or promote products
+ derived from this software without specific prior
+ written permission of Yahoo! Inc.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
12 README
@@ -0,0 +1,12 @@
+scanmaster is a set of scripts that allow you to ssh to a very large
+number of hosts in parallel to run an arbitrary script on the remote side.
+It works best if the scans can take advantage of a set of scanslave nodes
+with access to shared NFS space, though that is not a hard requirement.
+
+scanmaster was initially written January 2007 by Jan Schaumann while
+working at Yahoo! Inc. Yahoo! open sourced the tool in the hopes that it
+will be useful to other people -- unless otherwise noted, all files are
+released under the terms of a 3-clause BSD license as noted in the file
+LICENSE.
+
+http://www.netmeister.org/apps/scanmaster/
241 doc/checkhosts.1
@@ -0,0 +1,241 @@
+.\" This manual page was originally written by Jan Schaumann
+.\" <jschauma@yahoo-inc.com> in February 2007.
+.Dd January 28, 2011
+.Dt CHECKHOSTS 1
+.Os
+.Sh NAME
+.Nm checkhosts
+.Nd scan a list of hosts and run a given script on them
+.Sh SYNOPSIS
+.Nm
+.Op Fl HIPSbh
+.Op Fl A Ar authmode
+.Op Fl o Ar dir
+.Op Fl r Ar script
+.Fl f Ar file
+.Sh DESCRIPTION
+For every host in the input, the
+.Nm
+utility will check whether or not the host pings, check if sshd is
+listening on port 22 of the remote host, and finally tries to run the
+specified script on the remote host.
+The script is piped into the ssh command, and executed on the remote side
+via
+.Xr bash 1 .
+The output from the script is collected in the output file, prepended with
+the hostname (and IP address) that
+.Nm
+used to connect to this host.
+Any and all other output formatting is the responsibility of the remote
+script or the post-processing script.
+.Pp
+The following options are supported:
+.Bl -tag -width A_authmode_
+.It Fl A Ar authmode
+Specify the authentication mode.
+Valid options are "all", "password" and "pubkey".
+Specifying this option will cause
+.Xr ssh 1
+to be instructed to only use the given authentication mechanism with no
+others.
+.It Fl H
+Do not use a headless user.
+Per default,
+.Nm
+will assume that the ssh connections it establishes are for a headless and
+hence disables password authentication explicitly (it relies on the
+invoking user having access to the required ssh key for this user).
+.It Fl I
+Do not use IP addresses to ssh to.
+Per default,
+.Nm
+will use the IP address (or rather the second field of the input line) to
+connect to.
+.It Fl P
+Do not try to ping the hosts.
+Per default,
+.Nm
+will try to ping each host before trying to ssh to it.
+If a host fails to ping, it is skipped.
+.It Fl S
+Do not check if sshd is listening on port 22 on the hosts.
+Per default,
+.Nm
+will try to connect to port 22 on the remote host and checks if it
+gets an SSH version string back.
+If not, the host is skipped.
+.It Fl b
+The script is run in the background.
+This slightly changes the default behaviour and should normally only be
+set by the invoking
+.Xr scanhosts 1
+script.
+.It Fl f Ar file
+Read the list of hosts from
+.Ar file .
+Multiple
+.Fl f Ar file
+are allowed.
+.It Fl h
+Print a usage message and exit.
+.It Fl o Ar dir
+Put the output files into
+.Ar dir .
+Per default, output is generated in TMPDIR.
+.It Fl r Ar script
+Run this script on the remote host.
+The script is executed on the remote host via
+.Xr bash 1 .
+Per default, the file './remote.sh' is used.
+See the section
+.Xr SCRIPT
+for more details.
+.El
+.Sh SCRIPT
+The script executed on the remote side is executed with
+.Xr bash 1
+(either explicitly or via the headless user's shell).
+Hence, it needs to be a valid bash script, though due to different
+versions of bash found on different hosts, it is recommended to have the
+script be /bin/sh compatible.
+The script can be as simple or complex as you like, and in theory you
+should be able to let it perform any possible task you can script
+(including generating code in another language, storing it in another
+file, compiling it if needed and executing it).
+.Pp
+Please note that if the script contains
+.Xr sudo 8
+invocations, the password will actually be echoed back on the terminal!
+This is due to the way
+.Xr sudo 8
+and
+.Xr ssh 1
+interact when called from a script.
+Please take care to not let anybody shoulder-surf and do not store the
+output in any files readable by anybody else.
+.Sh INPUT
+Input can be provided via the
+.Fl f Ar file
+flag.
+Multiple
+.Fl f Ar filename
+are allowed.
+The input format of the file is two fields per line, the first field being
+the hostname, the second the IP address of the host.
+Lines starting with a # are treated as comments and ignored.
+.Sh OUTPUT
+Output is generated into the following distinct files:
+.Bl -tag -width hosts_checked
+.It hosts_checked
+The list of hosts that
+.Nm
+attempted to connect to.
+.It hosts_exit_N
+The list of hosts (including output of the script) on which the given
+script was executed, but returned a non-zero exit status.
+The output of the script will be prepended with the hostname, but otherwise
+the format of the output file is determined by the output generated by the
+script.
+.It hosts_noping
+The list of hosts that did not respond to
+.Xr ping 1
+(unless
+.Fl P
+was given).
+.It hosts_nossh
+The list of hosts that were unable to be reached via
+.Xr ssh 1 .
+There are numerous reasons why a host may fail.
+Among them are:
+the user does not have an account on the host;
+the machine can not be reached due to firewall (ACL) issues;
+the host refuses logins via ssh by way of /etc/hosts.allow;
+.It hosts_nosshd
+The list of hosts that were found not to have an ssh daemon listening on
+port 22 (unless
+.Fl S
+was given).
+.It hosts_ok
+The list of hosts on which the given script was successfully executed,
+together with the output of the script.
+The output of the script will be prepended with the hostname, but otherwise
+the format of the output file is determined by the output generated by the
+script.
+.El
+.Sh SSH PROCESSES
+.Nm
+spawns a large number of ssh processes, and sometimes it can be necessary to
+monitor them to ensure they are not hung.
+.Xr ssh 1
+is invoked by
+.Nm
+via a wrapper named ".checkhosts-ssh" in the output directory to ensure
+that its connections can easily be distinguished from normal ssh
+connections.
+.Pp
+The use of the
+.Xr tkill 1
+utility is recommended to periodically prune any possibly hung ssh
+processes.
+.Sh CAVEATS
+If you run
+.Nm
+with password authentication enabled, then it will, not very surprisingly,
+prompt you for your password.
+It will do that for every ssh connection it makes.
+To get around this, use your favorite password injector expect script
+(let's call it 'autopw') to invoke
+.Nm .
+This tool specifically does not attempt to 'cache' your passwords for you.
+.Sh EXAMPLES
+The following examples illustrate common usage of this tool.
+.Pp
+This runs the scans without checking if a host pings, reads input from the
+file /tmp/input and puts the output into the directory ./output/20070227 under
+the current working directory:
+.Pp
+.Bd -literal -offset indent
+$ autopw checkhosts -P -f /tmp/input -o ./output/20070227
+.Ed
+.Pp
+This runs the script ~/bin/myscript.sh on the hosts given on the command-line.
+Output is placed into TMPDIR (or /tmp, if that variable is not set):
+.Pp
+.Bd -literal -offset indent
+$ autopw checkhosts -r ~/bin/myscript.sh host1 host2 host3 ... hostN
+.Ed
+.Pp
+Assuming a headless user named 'headless' exists and uses the ssh key
+found in ~/.ssh/headless.privkey use the SSHOPTS environment variable as
+follows:
+.Pp
+.Bd -literal -offset indent
+$ autopw env SSHOPTS="-l headless -i ~/.ssh/headkess.privkey" \\
+ checkhosts <other options>
+.Ed
+.Sh EXIT STATUS
+.Ex -std
+.Sh ENVIRONMENT
+.Nm
+will pass the value of the variable SSHOPTS on to
+.Xr ssh 1
+after any options it passes itself.
+This allows the user to overwrite any options normally used by
+.Nm
+and is particularly useful to specify the username and location of ssh
+keys different from the defaults.
+.Sh SEE ALSO
+.Xr scanhosts 1 ,
+.Xr sigsh 1 ,
+.Xr tkill 1
+.Sh HISTORY
+The
+.Nm
+utility was originally written by
+.An Jan Schaumann
+.Aq jschauma@yahoo-inc.com
+in January 2007.
+It's original task was to check hosts if they were able to deal with the new
+Daylight Saving Time instituted in the USA in 2007.
+.Sh BUGS
+Please reports bugs or feature requests to the author.
140 doc/scanhosts.1
@@ -0,0 +1,140 @@
+.\" This manual page was originally written by Jan Schaumann
+.\" <jschauma@yahoo-inc.com> in February 2007.
+.Dd December 11, 2010
+.Dt SCANHOSTS 1
+.Os
+.Sh NAME
+.Nm scanhosts
+.Nd run checkhosts against a large number of hosts
+.Sh SYNOPSIS
+.Nm
+.Op Fl BHIPSh
+.Op Fl A Ar authmode
+.Op Fl n Ar num
+.Op Fl o Ar dir
+.Op Fl p Ar script
+.Op Fl s Ar suffix
+.Fl f Ar file
+.Fl r Ar script
+.Sh DESCRIPTION
+The
+.Nm
+utility is used to run
+.Xr checkhosts 1
+against a large number of hosts.
+It is normally invoked via
+.Xr autopw 1
+by
+.Xr scanslave 1 .
+It splits the input and then invokes
+.Xr checkhosts 1
+on the individual files in the background, eventually waiting for them all to
+complete.
+It then invokes the optionally provided post-processing command, passing it the
+name of the output directory as the single argument.
+.Pp
+The following options are supported:
+.Bl -tag -width A_authmode_
+.It Fl A Ar authmode
+Specify the authentication mode.
+Valid options are "all", "password" and "pubkey".
+This is passed on to the
+.Xr checkhosts 1
+command.
+.It Fl B
+Do not background
+.Xr checkhosts 1 Ns .
+This option is required if the command run on the remote host invokes
+.Xr sudo 8
+(or does any other interactive prompting) and will necessarily result in a
+significant performance decrease, since jobs no longer are executed in
+parallel but sequentially.
+.It Fl H
+Do not use the headless user for ssh connections.
+This is passed on to the
+.Xr checkhosts 1
+command.
+.It Fl I
+Do not use IP addresses to ssh to.
+This is passed on to the
+.Xr checkhosts 1
+command.
+.It Fl P
+Do not try to ping the hosts.
+This is passed on to the
+.Xr checkhosts 1
+command.
+.It Fl S
+Do not check if sshd is listening on port 22 on the hosts.
+This is passed on to the
+.Xr checkhosts 1
+command.
+.It Fl f Ar file
+Take input from this file.
+Multiple
+.Fl f Ar file
+may be given.
+The format of the file is exactly one host per line.
+Lines starting with a # are treated as comments and ignored until the end of
+the line.
+.It Fl h
+Print a usage message and exit.
+.It Fl n Ar num
+Split the input into this many files and thus run
+.Ar num
+instances of
+.Xr checkhosts 1
+in parallel.
+.It Fl o Ar dir
+Put the output files into
+.Ar dir .
+Per default, output is generated in TMPDIR.
+.It Fl p Ar cmd
+Invoke the specified command for post-processing of the output.
+This command is passed as a single argument the directory where the output was
+collected in.
+.It Fl r Ar script
+Specify the file that contains the script to run on the remote side.
+Passed through to
+.Xr checkhosts 1 .
+.It Fl s Ar suffix
+Use given suffix instead of default date for all output.
+.El
+.Sh OUTPUT
+Output is generated in the directory "output" under the directory specified
+via
+.Fl o Ar dir .
+In that directory, N subdirectories will be created (where N is the number of
+files the input was split into), into which the
+.Xr checkhosts 1
+instance will generate output.
+.Sh EXAMPLES
+The following examples illustrate common usage of this tool.
+.Pp
+This runs the scans without checking if a host pings, reads input from the
+file /tmp/input, splits the input into files of 20 lines each and puts the
+output into the directory ./output/20070227 under the current working
+directory:
+.Pp
+.Bd -literal -offset indent
+$ autopw scanhosts -P -n 20 -f /tmp/input -o ./output/20070227 \\
+ -r /tmp/remote.sh
+.Ed
+.Sh EXIT STATUS
+.Ex -std
+.Sh SEE ALSO
+.Xr autopw 1 ,
+.Xr checkhosts 1 ,
+.Xr scanslave 1 ,
+.Xr scanmaster 1
+.Sh HISTORY
+The
+.Nm
+utility was originally written by
+.An Jan Schaumann
+.Aq jschauma@yahoo-inc.com
+in January 2007.
+It's original task was to check hosts if they were able to deal with the new
+Daylight Saving Time instituted in the USA in 2007.
+.Sh BUGS
+Please reports bugs or feature requests to the author.
189 doc/scanmaster.1
@@ -0,0 +1,189 @@
+.\" This manual page was originally written by Jan Schaumann
+.\" <jschauma@yahoo-inc.com> in May 2009.
+.Dd January 29, 2011
+.Dt SCANMASTER 1
+.Os
+.Sh NAME
+.Nm scanmaster
+.Nd farm scanning jobs out to hosts running scanslave
+.Sh SYNOPSIS
+.Nm
+.Op Fl Hh
+.Op Fl A Ar authmode
+.Op Fl a Ar aggregator
+.Op Fl c Ar post-chunk
+.Op Fl d Ar dir
+.Op Fl j Ar jobs
+.Op Fl p Ar pre
+.Op Fl w Ar workers
+.Fl i Ar input
+.Fl n Ar name
+.Fl r Ar remote
+.Op -- scanslave-passthrough-args
+.Sh DESCRIPTION
+The
+.Nm
+utility is part of the "scanmaster" suite of scripts.
+It is the primary invoking command and hence named the same.
+It represents the portion of the process that farms a scan job out to a
+number of scanning nodes, which in turn invoke
+.Xr scanslave 1 .
+.Pp
+.Nm
+supports the following command-line options:
+.Bl -tag -width _aggregator__
+.It Fl A Ar authmode
+Specify the authentication mode.
+Valid options are "all", "password" and "pubkey".
+This is passed on to the
+.Xr scanslave 1
+command.
+.It Fl H
+Do not use the headless user for ssh connections.
+.It Fl a Ar aggregator
+The path to a script to aggregate the results from all slaves.
+This script is run after all slaves have finished and it is passed as
+the first argument the name of the scan, as the second argument the
+path to the original input file and as the third argument the date
+(YYYYMMDD) that the scan was started.
+If you require more flexibility, you may wish to use a wrapper script that
+invokes
+.Nm
+without an aggregator script and the runs whatever other tools you have to
+post-process the data.
+.It Fl c Ar post-chunk
+The script each slave should execute after it's finished.
+As this flag is passed on to the
+.Xr scanslave 1
+invocations, the path needs to be one that all slave nodes can resolve.
+Using a script on an NFS share is a good solution to this problem.
+.It Fl d Ar dir
+The directory under which
+.Nm
+should create all output.
+Preferably this is on an NFS share amongst all scanning nodes, though this
+is not a technical requirement (it just makes postprocessing of the scan
+results an order of magnitude easier).
+Please note that if no NFS share is used, the location still needs to be
+the same for all slave nodes.
+If not specified, defaults to "/mnt/scanmaster".
+.It Fl h
+Print a short usage and exit.
+.It Fl i Ar input
+Specify the file containing the input list of hosts to scan.
+.It Fl j Ar jobs
+The number of jobs
+.Xr scanhosts 1
+should execute.
+This flag is passed on to
+.Xr scanslave 1 ;
+if not specified, use that tool's defaults.
+.It Fl n Ar name
+Specify the name of this scan.
+This name is used in a number of places during output file creation.
+.It Fl w Ar workers
+Specify the file containing the list of worker hosts to farm this job out
+to.
+If not specified, defaults to "/usr/local/share/scanmaster/slavenodes".
+.It Fl p Ar pre
+Specify the script to run before doing anything else.
+This script is invoked with the scan name as the single argument.
+.It Fl r Ar remote
+Specify the script to execute on all target hosts.
+As this flag is passed on to the
+.Xr scanslave 1
+invocations, the path needs to be one that all slave nodes can resolve.
+Using a script on an NFS share is a good solution to this problem.
+.El
+.Pp
+Any arguments given after a
+.Fl -
+will be treated as "scanslave pass-through" arguments.
+That is, they will be passed on to
+.Xr scanslave 1 ,
+with the
+.Fl -
+intact, meaning
+.Xr scanslave 1
+will further pass them through to other tools it may invoke.
+.Sh DETAILS
+Upon invocation,
+.Nm
+will first run the script given to the
+.Fl p
+flag (if any), passing it the name of the scan as the single argument.
+It then removes any files found under the shared directory for a scan by
+this name, creates a new sub directory (by date), and tries to set up the
+environment for the currently running (if any)
+.Xr ssh-agent 1 .
+Next, it splits the input file into N chunks, where N is the number of
+hosts found in the worker hosts file.
+After that, it will ssh to each worker host, create a screenrc file that
+will invoke
+.Xr scanslave 1
+with the appropriate options; it then invokes
+.Xr screen 1
+with a key-binding of
+.Ar ^Ww .
+The user needs to detach from each host's screen session in order to let
+.Nm
+move on to the next host.
+.Pp
+After all hosts have started a screen session,
+.Nm
+will wait for the scanning nodes to complete their respective jobs.
+Once every minute,
+.Nm
+will look for the existence of a file called ".done" which is created in
+the well-defined shared location by
+.Xr scanslave 1 .
+When it finds all ".done" files, it will execute the script given via the
+.Ar a
+flag (if any), passing it the name of the scan as the first
+argument and the location of the original input file as the second
+argument.
+.Sh ENVIRONMENT
+.Nm
+uses the following environment variables:
+.Bl -tag -width SCAN_ENV_
+.It SCAN_ENV
+.Nm
+invokes
+.Xr scanslave 1
+by calling
+.Bd -literal -offset indent
+env ${SCAN_ENV} scanslave other-args
+.Ed
+This allows the user to pass anything into the environment of the invoked
+processes on the scanslave hosts.
+Obviously, SCAN_ENV should then contain key=value pairs.
+.El
+.Sh EXAMPLES
+The following examples illustrate common usage of this tool.
+.Pp
+.Bd -literal -offset indent
+SCAN=info1
+PREFIX=/mnt/scanmaster/jschauma/${SCAN}
+scanmaster \\
+ -a ${PREFIX}/aggregator.sh \\
+ -c ${PREFIX}/post.chunk.sh \\
+ -i ${PREFIX}/files/${SCAN}-input \\
+ -p ${PREFIX}/pre.sh \\
+ -r ${PREFIX}/remote.sh \\
+ -n ${SCAN}
+.Ed
+.Sh SEE ALSO
+.Xr autopw 1 ,
+.Xr checkhosts 1 ,
+.Xr scanhosts 1 ,
+.Xr scanslave 1 ,
+.Xr tkill 1
+.Sh HISTORY
+The
+.Nm
+utility was originally written by
+.An Jan Schaumann
+.Aq jschauma@yahoo-inc.com
+in July 2007 as "cmd.chunker".
+.Sh BUGS
+Please report bugs and feature requests to the author.
155 doc/scanslave.1
@@ -0,0 +1,155 @@
+.\" This manual page was originally written by Jan Schaumann
+.\" <jschauma@yahoo-inc.com> in May 2009.
+.Dd January 28, 2011
+.Dt SCANSLAVE 1
+.Os
+.Sh NAME
+.Nm scanslave
+.Nd invoke scanhosts on a scanning node
+.Sh SYNOPSIS
+.Nm
+.Op Fl Hh
+.Op Fl A Ar authmode
+.Op Fl d Ar dir
+.Op Fl n Ar N
+.Op Fl p Ar post
+.Op Fl r Ar remote
+.Ar scan
+.Ar chunkfile
+.Op Ar -- scanhost-args
+.Sh DESCRIPTION
+The
+.Nm
+utility is part of the "scanmaster" suite of scripts.
+It represents the portion of the process that runs on each scanning node
+and invokes
+.Xr scanhosts 1 .
+It takes as input the name of a scan and the so-called chunkfile
+containing the subset of hosts this scanning node should work on.
+.Pp
+.Nm
+supports the following command-line options:
+.Bl -tag -width A_authmode_
+.It Fl A Ar authmode
+Specify the authentication mode.
+Valid options are "all", "password" and "pubkey".
+This is passed on to the
+.Xr scanshosts 1
+command.
+.It Fl H
+Do not use the headless user for ssh connections.
+.It Fl d Ar dir
+The directory under which
+.Nm
+should create all output.
+Preferably this is on an NFS share amongst all scanning nodes, though this
+is not a technical requirement (it just makes postprocessing of the scan
+results an order of magnitude easier).
+If not specified, defaults to "/mnt/scanmaster".
+.It Fl n Ar N
+The number of jobs
+.Xr scanhosts 1
+should execute via its
+.Fl n
+flag.
+If not specified, defaults to 175.
+.It Fl p Ar post
+The location of the script to be passed to
+.Xr scanhosts 1
+as the post-processing script.
+If not specified, defaults to "./post.chunk.sh".
+.It Fl r Ar remote
+The location of the script to be passed to
+.Xr scanhosts 1
+as the script to execute on the remote hosts.
+If not specified, defaults to "./remote.sh".
+.El
+.Pp
+It furthermore takes two mandatory arguments:
+.Bl -tag -width chunkfile_
+.It scan
+The name of the scan.
+This name is used in a number of places to generate output directories and
+files.
+.It chunkfile
+The path to the file containing the hostnames to be scanned.
+.El
+.Pp
+Any additional arguments are passed on to
+.Xr scanhosts 1 .
+However, since
+.Nm
+already uses the following flags, they should not be provided as this may
+lead to undefined results:
+.Fl n ,
+.Fl f ,
+.Fl o ,
+.Fl s ,
+.Fl p ,
+.Fl r .
+.Sh DETAILS
+Upon invocation,
+.Nm
+sets a few variables based on the environment (see below).
+It then starts an
+.Xr ssh-agent 1
+and invokes
+.Xr ssh-add 1 ,
+passing the value of the environment variable SSH_ADD_FLAGS.
+Next, it creates the output directories in the preferably shared location,
+adds a crontab entry for
+.Xr tkill 1
+and eventually kicks off
+.Xr scanhosts 1
+via
+.Xr autopw 1 .
+When the scanhosts process has terminated, the ssh-agent is killed and the
+crontab entries removed.
+.Sh EXAMPLES
+The following examples illustrate common usage of this tool.
+.Pp
+To run a scan named "scan1", reading a list of hosts to scan from the file
+"/tmp/input.list" with the script "remote.sh" from the current working
+directory to be executed and the script "../post.chunk.sh" used as a
+per-chunk post script:
+.Bd -literal -offset indent
+scanslave -r ./remote.sh -p ../post.chunk.sh \\
+ scan1 /tmp/input.list
+.Ed
+.Pp
+To run the same scan but pass the flags "-I" and "-S" to
+.Xr scanhosts 1
+(which is useful when connecting to hosts that are reached via a tunnel,
+such as vault hosts):
+.Bd -literal -offset indent
+scanslave -r ./remote.sh -p ../post.chunk.sh \\
+ scan1 /tmp/input.list -- -I -S
+.Ed
+.Sh ENVIRONMENT
+.Nm
+honors the following environment variables:
+.Bl -tag -width SSH_ADD_FLAGS_
+.It SSH_ADD_FLAGS
+Flags to be passed to
+.Xr ssh-add 1 .
+This allows the user to specify custom keys to be added.
+.It TMPDIR
+The location of a temporary directory.
+If not specified, defaults to "/tmp".
+.El
+.Sh SEE ALSO
+.Xr autopw 1 ,
+.Xr checkhosts 1 ,
+.Xr scanhosts 1 ,
+.Xr ssh-agent 1 ,
+.Xr ssh-add 1 ,
+.Xr tkill 1
+.Sh HISTORY
+The
+.Nm
+utility was originally written by
+.An Jan Schaumann
+.Aq jschauma@yahoo-inc.com
+in July 2007 as "cmd.chunk".
+.Sh BUGS
+Please report bugs and feature requests to the author.
67 doc/tkill.1
@@ -0,0 +1,67 @@
+.\" This manual page was written by Jan Schaumann <jschauma@yahoo-inc.com>
+.Dd February 02, 2011
+.Dt TKILL 1
+.Os
+.Sh NAME
+.Nm tkill
+.Nd kill a process if it has been running for too long
+.Sh SYNOPSIS
+.Nm
+.Op Fl Phl
+.Op Fl t Ar timeout
+.Op Fl s Ar signal
+.Ns ( Ns Ar pid Ar ... | Fl C Ar cmd | Fl c Ar cmd Ns )
+.Sh DESRIPTION
+The
+.Nm
+utility kills a given process or process group if it has been running for
+longer than the specified time.
+.Pp
+The following options are supported:
+.Bl -tag -width t_timeout_
+.It Fl C Ar cmd
+Kill processes by name, not by process ID.
+If the given command is found to be a substring of any command in the
+process table, then those processes are killed.
+.It Fl P
+Kill all processes whose parent is the given pid.
+.It Fl c Ar cmd
+Kill processes by name, not by process ID.
+The given command must match exactly the command in the process table.
+.It Fl h
+Print a usage statement and exit.
+.It Fl l
+Print a list of valid signals and exit.
+.It Fl t Ar timeout
+Specify the timeout in seconds.
+If not specified, default to 300.
+.It Fl s Ar signal
+Kill with the given signal, which may be either numeric or the symbolic
+name of hte signal.
+If not specified, kill with the TERM signal.
+.El
+.Sh EXAMPLES
+The following examples illustrate common usage of this tool.
+.Pp
+To kill the processes with the process IDs 1234 1235 and 1236 if they have
+been running for longer than 500 seconds:
+.Bd -literal -offset indent
+tkill 1234 1235 1236
+.Ed
+.Pp
+To kill all the children of process ID 1234 if they have been running for
+longer than one hour:
+.Bd -literal -offset indent
+tkill -t 3600 -P 1234
+.Ed
+.Sh SEE ALSO
+.Xr kill 1
+.Sh HISTORY
+The
+.Nm
+utility was originally written by
+.An Jan Schaumann
+.Aq jschauma@yahoo-inc.com
+in July 2007.
+.Sh BUGS
+Please submit bug reports and feature requests to the author.
61 helpers/generic-aggregator
@@ -0,0 +1,61 @@
+#!/bin/sh
+#
+# $Id: generic-aggregator 72 2011-02-02 23:31:11Z jans $
+# $URL: svn+ssh://svn.corp.yahoo.com/yahoo/tools/scanmaster/scanner/branches/public/helpers/generic-aggregator $
+#
+# Copyright (c) 2008 Yahoo! Inc.
+#
+# Originally written by Jan Schaumann <jans@yahoo-inc.com> in July 2007.
+#
+# This script aggregates the data from a full scan into a directory under
+# the webservers docroot. It takes two arguments, the one-word name of
+# the scan and the original input file.
+
+name=${1}
+input=${2}
+date=${3}
+suffix=$(echo ${input} | sed -e "s|.*/${name}-||")
+prefix=${BASEDIR:-/mnt/scanmaster}/chunked/${name}
+outdir=${BASEDIR:-/mnt/scanmaster}/${USER}/${name}-not-yet/${date}/${suffix}
+
+(
+ echo "Aggregating data..."
+ mkdir -p ${prefix}/final
+ cp ${input} ${prefix}/final/input
+
+ # mutli-line output in 'output', hostnames etc. in 'ok'
+ find ${prefix}/${date}* -name hosts_ok -exec cat {} \; | tee ${prefix}/final/output | grep , | sort -u > ${prefix}/final/ok
+
+ # cat */*/*/hosts_checked might be too many files for the shell,
+ # so find(1) them instead.
+ find ${prefix}/${date}* \( ! -name 'hosts_ok' -and -name 'hosts_*' \) | while read filename; do
+ f="${filename##*/hosts_}"
+ cat ${filename} >> ${prefix}/final/${f}
+ done
+
+ mkdir -p ${outdir}
+ cp ${prefix}/final/* ${outdir}/
+
+ cd ${outdir}
+
+ wc -l * | egrep -v "(count|total|start|end)" > count
+
+ chmod -R a+rX,g+rw ${outdir}
+
+ mkdir -p ${prefix}/.old/${date}
+ mv ${prefix}/* ${prefix}/.old/${date}/
+)
+
+ended=$(date)
+host=$(hostname)
+
+echo "Sending mail..."
+
+/usr/sbin/sendmail -t <<EOM
+To: ${USER}
+From: ${USER}
+Subject: scanhosts ${name} results from ${date}
+
+${outdir}
+
+EOM
64 helpers/generic-post
@@ -0,0 +1,64 @@
+#! /bin/sh
+#
+# $Id: generic-post 1 2010-08-23 18:19:52Z jans $
+# $URL: svn+ssh://svn.corp.yahoo.com/yahoo/tools/scanmaster/scanner/branches/public/helpers/generic-post $
+#
+# Copyright (c) 2007 Yahoo! Inc.
+#
+# Originally written by Jan Schaumann <jans@yahoo-inc.com> in February 2009.
+#
+# Takes as argument the name of the output directory into which scanhosts
+# dumped its files.
+
+readonly DATE=$(date +%Y%m%d)
+
+DATE_DIR=
+OUTPUTDIR=
+
+# function : collect_output
+# purpose : collects the output of all runs into a separate directory
+# inputs : none, run_checkhosts has completed
+# returns : none, output in 'full' directory
+
+collect_output () {
+ local readonly output_subdir="${OUTPUTDIR}/output/${DATE_DIR}"
+ local readonly input_files="${OUTPUTDIR}/output/input"
+ local readonly checked_files="${output_subdir}/*/hosts_checked"
+ local readonly ok_files="${output_subdir}/*/hosts_ok"
+ local readonly noping_files="${output_subdir}/*/hosts_noping"
+ local readonly nossh_files="${output_subdir}/*/hosts_nossh"
+ local readonly nosshd_files="${output_subdir}/*/hosts_nosshd"
+
+ local readonly outdir="${OUTPUTDIR}/final"
+
+ local readonly checked="${outdir}/checked"
+ local readonly input="${outdir}/input"
+ local readonly noping="${outdir}/noping"
+ local readonly nossh="${outdir}/nossh"
+ local readonly nosshd="${outdir}/nosshd"
+ local readonly ok="${outdir}/ok"
+
+ mkdir -p ${outdir}
+
+ cat ${ok_files} > ${ok}
+ sort -u ${input_files} > ${input}
+ sort -u ${checked_files} > ${checked}
+ sort -u ${noping_files} > ${noping}
+ sort -u ${nossh_files} > ${nossh}
+ sort -u ${nosshd_files} > ${nosshd}
+}
+
+###
+### Main
+###
+
+if [ $# -ne 1 -o ! -d ${1} ]; then
+ echo "Usage: ${0##*/} <directory>" >&2
+ exit 1
+ # NOTREACHED
+fi
+
+OUTPUTDIR="${1}"
+DATE_DIR="$(cd ${OUTPUTDIR}/output && ls -rt | tail -1)"
+
+collect_output
359 src/checkhosts
@@ -0,0 +1,359 @@
+#!/bin/sh
+#
+# Copyright (c) 2007,2008,2009,2010,2011 Yahoo! Inc.
+# All rights reserved.
+#
+# Originally written by Jan Schaumann <jschauma@yahoo-inc.com> in January 2007.
+#
+# Redistribution and use of this software in source and binary forms,
+# with or without modification, are permitted provided that the following
+# conditions are met:
+#
+# * Redistributions of source code must retain the above
+# copyright notice, this list of conditions and the
+# following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the
+# following disclaimer in the documentation and/or other
+# materials provided with the distribution.
+#
+# * Neither the name of Yahoo! Inc. nor the names of its
+# contributors may be used to endorse or promote products
+# derived from this software without specific prior
+# written permission of Yahoo! Inc.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+#
+# This script takes as input a list of hosts and tries to ssh to them.
+# It generates as output a file containing the list of hosts that did not ping
+# in one file, a list of hosts that do not appear to be listening on port 22
+# in a second file, the list of hosts it was unable to ssh to in a third file
+# and the outputs of the command it was told to run in the fourth file.
+
+###
+### Globals
+###
+
+TDIR="${TMPDIR:-/tmp}"
+
+AUTHMODE="pubkey"
+BACKGROUND='no'
+HEADLESS='yes'
+INPUT_FILES=
+OUTPUT_DIR=${TDIR}
+SCRIPT="remote.sh"
+SSH_WRAPPER=".checkhosts-ssh"
+SCP_WRAPPER=".checkhosts-scp"
+
+CHECK_PING='yes'
+CHECK_SSHD='yes'
+HOSTNAMES_ONLY='no'
+
+###
+### Subroutines
+###
+
+# function : usage
+# purpose : print a help message
+# usage : usage
+# inputs : none
+# outputs : help message
+
+usage () {
+ eat <<EOF
+Usage: ${0##*/} [-HIPSbh] [-A authmode] [-o dir] [-r script] -f file
+ -A Specify ssh authentication mode.
+ -H Do not use a headless account.
+ -I Do not use IP addresses.
+ -P Do not try to ping the hosts.
+ -S Do not check if sshd is running on the hosts.
+ -b The script is run in the background.
+ -f file Read list of hosts from file.
+ -h Print this message and exit.
+ -o dir Put output files into dir (default: TMPDIR).
+ -r script Specify the file that contains the script to run on the remote side.
+EOF
+}
+
+# function : checkhost
+# purpose : checks an individual host
+# as such it performs:
+# - a ping check
+# - a check whether sshd is listening on port 22
+# - tries to ssh to it and run uname on it
+# usage : checkhost host
+# inputs : a line consisting of a hostname and an optional second field
+# assumed to be an IP address
+# outputs : appends to the three output files
+
+checkhost () {
+ local input="${1}"
+ local host="${1%% *}"
+ local ip="${1##* }"
+ local target="${ip}"
+
+ local host_output substr rval
+ local outfile=${OUTFILE_OK}
+
+ local readonly remotefile="${TDIR}/scanhosts.${USER}.$$"
+ local readonly rcmd="bash ${remotefile} && rm -f ${remotefile}"
+
+ local remote_shell="bash -s"
+ local sshopts="-q
+ -o ForwardAgent=no
+ -o ClearAllForwardings=yes
+ -o StrictHostKeyChecking=no
+ -o UserKnownHostsFile=/dev/null
+ -o GlobalKnownHostsFile=/dev/null
+ -o Protocol=2,1"
+
+ case x"${AUTHMODE}" in
+ xall)
+ sshopts="${sshopts} -o PasswordAuthentication=yes
+ -o PubkeyAuthentication=yes"
+ ;;
+ xpassword)
+ sshopts="${sshopts} -o PasswordAuthentication=yes
+ -o KbdInteractiveAuthentication=no
+ -o PubkeyAuthentication=no"
+ ;;
+ xpubkey)
+ sshopts="${sshopts} -o PasswordAuthentication=no
+ -o KbdInteractiveAuthentication=no
+ -o PubkeyAuthentication=yes"
+ ;;
+ esac
+
+ if [ "${HEADLESS}" = "yes" ]; then
+ sshopts="${sshopts} -o IdentitiesOnly=yes"
+ remote_shell=""
+ else
+ sshopts="${sshopts} -o NumberOfPasswordPrompts=1"
+ fi
+
+ echo "${input}" >> ${OUTFILE_CHECKED}
+
+ if [ "${HOSTNAMES_ONLY}" = "yes" -o "${host}" = "${ip}" ]; then
+ target="${host}"
+ ip=""
+ fi
+
+ if [ "${CHECK_PING}" = "yes" ]; then
+ ping -c 1 -i 1 -t 3 -q ${target} >/dev/null 2>&1 || {
+ echo ${input} >> ${OUTFILE_NOPING}
+ return 1
+ # NOTREACHED
+ }
+ fi
+
+ if [ "${CHECK_SSHD}" = "yes" ]; then
+ check_sshd ${target} || {
+ echo ${input} >> ${OUTFILE_NOSSHD}
+ return 1
+ # NOTREACHED
+ }
+ fi
+
+ if [ "${BACKGROUND}" = "no" ] && [ "${HEADLESS}" = "no" ]; then
+
+ ${SCP_WRAPPER} ${sshopts} ${SSHOPTS} ${SCRIPT} ${target}:${remotefile}
+ if [ $? -gt 0 ]; then
+ echo ${input} >> ${OUTFILE_NOSSH}
+ return 1
+ # NOTREACHED
+ fi
+
+ host_output=$(${SSH_WRAPPER} ${sshopts} ${SSHOPTS} ${target} "${rcmd}")
+ else
+ host_output=$(cat ${SCRIPT} | ${SSH_WRAPPER} ${sshopts} ${SSHOPTS} ${target} "${remote_shell}")
+ fi
+
+ rval=$?
+
+ if [ ${rval} -eq 255 ]; then
+ echo ${input} >> ${OUTFILE_NOSSH}
+ return 1
+ # NOTREACHED
+ elif [ ${rval} -gt 0 ]; then
+ outfile="${OUTFILE_EXITCODE}_${rval}"
+ fi
+
+ echo "${host},${ip},${host_output}" | tr '
+' '\n' >> ${outfile}
+}
+
+# function : process_input_files
+# purpose : work of the lines in all input files
+# usage : process_input_files
+# inputs : none, operates on global INPUT_FILES
+# outputs : none
+
+process_input_files () {
+
+ local file
+
+ for file in ${INPUT_FILES}; do
+ if [ -r "${file}" ]; then
+ local readonly tmpf=$(mktemp ${TDIR}/checkhosts.$$.XXXXXX)
+ grep -v "^#" ${file} > ${tmpf}
+ oIFS=${IFS}
+ IFS='
+'
+ for line in $(cat ${tmpf}); do
+ IFS=${oIFS}
+ checkhost "${line}"
+ done
+ rm ${tmpf}
+ else
+ echo "Cannot read ${file}." >&2
+ fi
+ done
+}
+
+# function : check_sshd
+# purpose : verify that ssh is listening on port 22 of the given host
+# inputs : a hostname
+# returns : 0 if the host is in fact listening on port 22, >0 otherwise
+
+check_sshd() {
+ local readonly host=${1}
+
+ local i
+ local readonly tmpfile=$(mktemp ${TDIR}/$$.XXXXXX)
+
+ ( echo | nc $host 22 | grep -i ssh >${tmpfile}; ) &
+ pid=$!
+ i=0
+ while [ ${i} -lt 60 ]; do
+ kill -0 ${pid} >/dev/null 2>&1
+ if [ $? -ne 0 ]; then
+ break
+ fi
+ i=$(( ${i} + 1 ))
+ sleep 1
+ done
+ kill -9 ${pid} >/dev/null 2>&1
+ wait ${pid} >/dev/null 2>&1
+ if [ $? -eq 0 ]; then
+ local ssh="$(cat ${tmpfile})"
+ rm -f ${tmpfile}
+ if [ -n "${ssh}" ]; then
+ return 0
+ fi
+ fi
+
+ rm -f ${tmpfile}
+ return 1
+}
+
+# function : wrap_ssh
+# purpose : create a near no-op ssh wrapper so that the name of the
+# wrapper appears in the process table and we can more easily
+# identify the ssh processes related to this program
+# inputs : none
+# returns : nothing, creates a symlink pointing to ssh in the output
+# directory, iff it doesn't already exist
+
+wrap_ssh() {
+
+ export PATH=${OUTPUT_DIR}:${PATH}
+ if [ -h "${OUTPUT_DIR}/${SSH_WRAPPER}" -a -h "${OUTPUT_DIR}/${SCP_WRAPPER}" ]; then
+ return
+ fi
+
+ ln -sf $(which ssh) ${OUTPUT_DIR}/${SSH_WRAPPER}
+ ln -sf $(which scp) ${OUTPUT_DIR}/${SCP_WRAPPER}
+}
+
+
+###
+### Main
+###
+
+main() {
+ while getopts 'A:HIPSbf:ho:r:' opt; do
+ case ${opt} in
+ A)
+ AUTHMODE="${OPTARG}"
+ ;;
+ H)
+ HEADLESS='no'
+ ;;
+ I)
+ HOSTNAMES_ONLY='yes'
+ ;;
+ P)
+ CHECK_PING='no'
+ ;;
+ S)
+ CHECK_SSHD='no'
+ ;;
+ b)
+ BACKGROUND='yes'
+ ;;
+ f)
+ INPUT_FILES="${INPUT_FILES} ${OPTARG}"
+ ;;
+ h|\?)
+ usage
+ exit 0
+ # NOTREACHED
+ ;;
+ o)
+ if [ ! -d ${OPTARG} ]; then
+ echo "Not a directory: ${OPTARG}."
+ exit 1
+ # NOTREACHED
+ fi
+ OUTPUT_DIR="${OPTARG}"
+ ;;
+ r)
+ SCRIPT="${OPTARG}"
+ ;;
+ *)
+ usage
+ exit 1
+ # NOTREACHED
+ ;;
+ esac
+ done
+ shift $(($OPTIND - 1))
+
+ if [ -z "${INPUT_FILES}" -a $# -lt 1 ]; then
+ usage
+ exit 1
+ # NOTREACHED
+ fi
+
+ if [ ! -r ${SCRIPT} ]; then
+ echo "Unable to read script \"${SCRIPT}\"." >&2
+ exit 1
+ # NOTREACHED
+ fi
+
+ OUTFILE_CHECKED="${OUTPUT_DIR}/hosts_checked"
+ OUTFILE_EXITCODE="${OUTPUT_DIR}/hosts_exit"
+ OUTFILE_NOPING="${OUTPUT_DIR}/hosts_noping"
+ OUTFILE_NOSSH="${OUTPUT_DIR}/hosts_nossh"
+ OUTFILE_NOSSHD="${OUTPUT_DIR}/hosts_nosshd"
+ OUTFILE_OK="${OUTPUT_DIR}/hosts_ok"
+
+ wrap_ssh
+
+ process_input_files
+
+ return 0
+}
+
287 src/scanhosts
@@ -0,0 +1,287 @@
+#!/bin/sh
+#
+# Copyright (c) 2007,2008,2009,2010,2011 Yahoo! Inc.
+# All rights reserved.
+#
+# Originally written by Jan Schaumann <jschauma@yahoo-inc.com> in January
+# 2007.
+#
+# Redistribution and use of this software in source and binary forms,
+# with or without modification, are permitted provided that the following
+# conditions are met:
+#
+# * Redistributions of source code must retain the above
+# copyright notice, this list of conditions and the
+# following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the
+# following disclaimer in the documentation and/or other
+# materials provided with the distribution.
+#
+# * Neither the name of Yahoo! Inc. nor the names of its
+# contributors may be used to endorse or promote products
+# derived from this software without specific prior
+# written permission of Yahoo! Inc.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# A program used to run 'checkhosts' against a large number of hosts.
+
+###
+### Globals
+###
+
+# set in init() or by getopts
+ALL_OUTPUT="${TMPDIR:-/tmp}"
+AUTHMODE="pubkey"
+BACKGROUND="&"
+BACKGROUND_FLAG="-b"
+NO_HEADLESS_FLAG=""
+IP_FLAG=""
+PING_FLAG=""
+SSHD_FLAG=""
+INPUTDIR=""
+INPUT_FILES=""
+NUM=40
+SCRIPT=""
+SUFFIX=""
+POST_CMD=""
+
+###
+### Subroutines
+###
+
+# function : usage
+# purpose : print help message
+# inputs : none
+# returns : none
+
+usage () {
+ cat <<EOH
+Usage: autopw ${0##*/} [-BHIPSh] [-A authmode] [-n num] [-o dir]
+ [-p cmd] [-s suffix] -f file -r script
+ -A authmode specify ssh authentication mode
+ -B do not put checkhosts in the background
+ -H do not let checkhosts use the headless user
+ -I do not use IP address to ssh to
+ -P do not try to ping hosts first
+ -S do not check if sshd is listening on the hosts
+ -f file take input from this file
+ -h print this message and exit
+ -n num split input into this many files
+ -o dir collect output in <dir>
+ -p cmd invoke the specified cmd for post-processing of the output
+ this command is passed as a single argument the directory where
+ the output was collected in
+ -r script specify the file that contains the script to run on the remote side
+ -s suffix use given suffix instead of default date for all output
+EOH
+}
+
+# function : init
+# purpose : initialize variables, make directories etc.
+# inputs : none
+# returns : none
+
+init () {
+ local nada
+
+ if [ x"${NO_HEADLESS_FLAG}" = x"-H" ] && [ x"${AUTHMODE}" != x"pubkey" ]; then
+ # prompt for password early on to avoid waiting for user input
+ # down the line
+ if [ -t 0 ]; then
+ stty -echo
+ read -p "Password: " nada
+ echo
+ stty echo
+ fi
+ fi
+
+ if [ -z "${SUFFIX}" ]; then
+ SUFFIX="$(date +%Y%m%d)"
+ fi
+ INPUTDIR=${ALL_OUTPUT}/input/${SUFFIX}
+ OUTPUTDIR=${ALL_OUTPUT}/output
+
+ # remove any old directories in case a scan was interrupted and
+ # restarted so that we don't get duplicate input or append dupes
+ # to the output
+ rm -fr ${OUTPUTDIR} ${INPUTDIR}
+ mkdir -p ${OUTPUTDIR} ${INPUTDIR}
+}
+
+# function : prep_input_files
+# purpose : collect input and split it into individual files
+# inputs : none
+# returns : nothing; input split into smaller files
+
+prep_input_files () {
+ local readonly input=${OUTPUTDIR}/input
+
+ local skip
+ local num_in lines
+
+ cat ${INPUT_FILES} > ${input}
+
+ num_in=$(wc -l <${input})
+ lines=$(( ${num_in}/${NUM} ))
+
+ if [ ${lines} -eq 0 ]; then
+ cat ${input} >> ${INPUTDIR}/xaa
+ return
+ fi
+
+ split -l ${lines} ${input} ${INPUTDIR}/ || exit 1
+
+ # Ugly, but since we can't tell split to split into exactly N
+ # files by line, we need to account for surplus. If
+ # we're splitting into files with a small number of lines, then
+ # the first file might be significantly larger than the others,
+ # but so be it. If we're splitting into files with a large number
+ # of lines, then the spillover should be small.
+
+ (
+ local first last
+ cd ${INPUTDIR}
+ spillover=$(ls | awk -v N="${NUM}" '{ if (FNR > N) { print }}')
+ last=$(ls | awk -v N="${NUM}" '{ if (FNR == N) { print }}')
+ if [ -n "${spillover}" ]; then
+ cat ${spillover} >> ${last} && rm ${spillover}
+ fi
+ )
+}
+
+# function : run_checkhosts
+# purpose : invokes checkhosts in the background for each file found in
+# the inputdir, then waits for all jobs
+# inputs : none, input was split into separate files
+# returns : none, checkhosts was run for all input
+
+run_checkhosts () {
+
+ cd ${ALL_OUTPUT} || exit 1
+
+ for file in ${INPUTDIR}/*; do
+ local readonly sufx=${file##*/}
+ local readonly dir=${OUTPUTDIR}/${SUFFIX}/${sufx}
+
+ if [ -s ${file} -a "${file##*/}" != "input" ]; then
+ mkdir -p ${dir}
+ eval checkhosts -A ${AUTHMODE} ${BACKGROUND_FLAG} ${IP_FLAG} \
+ ${SSHD_FLAG} ${PING_FLAG} ${NO_HEADLESS_FLAG} \
+ -r ${SCRIPT} -o ${dir} -f ${file} >/dev/null ${BACKGROUND}
+ fi
+ done
+
+ # wait for _all_ jobs to finish
+ wait
+}
+
+# function : post_process
+# purpose : invoke the provided script for post-processing
+# inputs : the directory where output was generated
+# returns : exit status of the invoked script
+
+post_process () {
+ if [ -n "${POST_CMD}" ]; then
+ ${POST_CMD} "${ALL_OUTPUT}"
+ fi
+}
+
+# function : fullname
+# purpose : determine the full absolute path for a given binary
+# inputs : any pathname
+# returns : the absolute path
+
+fullname () {
+ local readonly arg="${1}"
+ echo $(cd $(dirname ${arg}) && pwd -P)/$(basename ${arg})
+}
+
+###
+### Main
+###
+
+while getopts 'A:BHIPS?f:hn:o:p:r:s:' opt; do
+ case ${opt} in
+ A)
+ AUTHMODE="${OPTARG}"
+ ;;
+ B)
+ BACKGROUND=""
+ BACKGROUND_FLAG=""
+ ;;
+ H)
+ NO_HEADLESS_FLAG="-H"
+ ;;
+ I)
+ IP_FLAG="-I"
+ ;;
+ P)
+ PING_FLAG="-P"
+ ;;
+ S)
+ SSHD_FLAG="-S"
+ ;;
+ f)
+ INPUT_FILES="$(fullname ${OPTARG}) ${INPUT_FILES}"
+ ;;
+ h|\?)
+ usage
+ exit 0
+ # NOTREACHED
+ ;;
+ n)
+ NUM=${OPTARG}
+ ;;
+ o)
+ if [ ! -d "${OPTARG}" ]; then
+ echo "Not a directory: ${OPTARG}"
+ exit 1
+ # NOTREACHED
+ fi
+ ALL_OUTPUT="$(fullname ${OPTARG})"
+ ;;
+ p)
+ POST_CMD="$(fullname ${OPTARG})"
+ ;;
+ r)
+ SCRIPT="$(fullname ${OPTARG})"
+ ;;
+ s)
+ SUFFIX="${OPTARG}"
+ ;;
+ *)
+ usage
+ exit 1
+ # NOTREACHED
+ ;;
+ esac
+done
+shift $(($OPTIND - 1))
+
+if [ $# -gt 0 -o -z "${INPUT_FILES}" -o -z "${SCRIPT}" ]; then
+ usage
+ exit 1
+ # NOTREACHED
+fi
+
+date > "${ALL_OUTPUT}/started"
+
+init
+prep_input_files
+run_checkhosts
+post_process
+
+date > "${ALL_OUTPUT}/ended"
342 src/scanmaster
@@ -0,0 +1,342 @@
+#!/bin/sh
+#
+# Copyright (c) 2007,2008,2009,2010,2011 Yahoo! Inc.
+# All rights reserved.
+#
+# Originally written by Jan Schaumann <jschauma@yahoo-inc.com> in March 2007.
+#
+# Redistribution and use of this software in source and binary forms,
+# with or without modification, are permitted provided that the following
+# conditions are met:
+#
+# * Redistributions of source code must retain the above
+# copyright notice, this list of conditions and the
+# following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the
+# following disclaimer in the documentation and/or other
+# materials provided with the distribution.
+#
+# * Neither the name of Yahoo! Inc. nor the names of its
+# contributors may be used to endorse or promote products
+# derived from this software without specific prior
+# written permission of Yahoo! Inc.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# This script is part of the "scanmaster" suite of scripts. It is the
+# primary invoking command and hence named the same. It represents the
+# portion of the process that farms a scan job out to a number of scanning
+# nodes.
+#
+# See scanmaster(1) for details.
+
+#set -eu
+
+###
+### Globals
+###
+
+AGGREGATOR=""
+AUTHMODE="pubkey"
+FULLDATE=$(date +%Y%m%d%H%M)
+DATE=$(date +%Y%m%d)
+DIR="/mnt/scanmaster"
+INPUT=""
+JOBS=""
+NAME=""
+NO_HEADLESS_FLAG=""
+NUM_HOSTS=
+PASSTHROUGH=""
+POST=""
+POSTFILE=""
+PROGNAME="${0##*/}"
+PRE=""
+REMOTE_SCRIPT=""
+WHOSTS="/usr/local/share/scanmaster/slavenodes"
+
+# set in prescanTasks
+PREFIX=""
+
+###
+### Subroutines
+###
+
+# function : farmOutJobs
+# purpose : set up a screenrc on each of our worker hosts, then invoke it
+# note: user needs to detach from each host's screen session
+# inputs : none
+
+farmOutJobs() {
+ local count=1
+
+ local goners="${HOME}/.screenrc.scanslave"
+ local scanslave="scanslave -A ${AUTHMODE} ${NO_HEADLESS_FLAG} -d ${DIR} -r ${REMOTE_SCRIPT} ${POST} ${JOBS}"
+ local cmd detach host file chunkfile
+
+ if [ x"${NO_HEADLESS_FLAG}" = x"-H" ]; then
+ detach=""
+ else
+ detach="-dm"
+ fi
+
+ cd ${PREFIX}/input/${FULLDATE}
+
+ for file in *; do
+ if [ x"${file}"x = 'x*x' ]; then
+ # nothing to do, no files found in directory
+ break
+ fi
+ host=$( echo ${count}p | ed -s ${WHOSTS})
+ chunkfile="${PREFIX}/input/${FULLDATE}/${file}"
+ cmd="${scanslave} ${NAME} ${chunkfile} ${PASSTHROUGH} && rm -f ${goners}"
+ ssh ${host} "echo \"screen sh -c \\\"${cmd}\\\"\" > ~/.screenrc.scanslave"
+ ssh -t ${host} "env ${SCAN_ENV} screen ${detach} -c ~/.screenrc.scanslave -e^Ww && sleep 1"
+ count=$(( ${count} + 1 ))
+ done
+}
+
+
+# function : finishTasks
+# purpose : clean up, invoke any optional aggregator scripts
+# inputs : none
+
+finishTasks() {
+
+ date > ${PREFIX}/ended
+
+ if [ -n "${AGGREGATOR}" ]; then
+ ${AGGREGATOR} ${NAME} ${INPUT} ${DATE}
+ fi
+ echo "Done."
+}
+
+# function : prepChunks
+# purpose : split the input into chunks as needed
+# inputs : none
+
+prepChunks() {
+
+ local num_in
+
+ local first last lines
+
+ local chunkdir="${PREFIX}/input/${FULLDATE}"
+
+ if [ ${NUM_HOSTS} -lt 2 ]; then
+ echo "Need more than ${num_in} hosts. Bailing out..." >&2
+ exit
+ fi
+
+ cd ${chunkdir}
+
+ num_in=$(wc -l <${INPUT})
+ lines=$(( ${num_in}/${NUM_HOSTS} ))
+
+ if [ ${lines} -gt 0 ]; then
+ split -l ${lines} ${INPUT}
+ last=$(ls | awk -v N="${NUM_HOSTS}" '{ if (FNR > N) { print }}')
+ first=$(ls | head -1)
+ if [ -n "${last}" ]; then
+ cat ${last} >> ${first} && rm ${last}
+ fi
+ else
+ # too little input to split, just copy the file into place
+ cp ${INPUT} .
+ fi
+}
+
+
+# function : prescanTasks
+# purpose : run any pre- script, verify all given files exist,
+# clean up old gunk, create directories as needed
+# inputs : none
+
+prescanTasks() {
+ local file var
+
+ if [ -n "${PRE}" ]; then
+ ${PRE} ${NAME}
+ fi
+
+ for file in "${INPUT}" "${REMOTE_SCRIPT}" "${WHOSTS}"; do
+ if [ ! -f "${file}" ]; then
+ echo "${PROGNAME}: No such file: ${file}" >&2
+ exit 1;
+ # NOTREACHED
+ fi
+ done
+
+ # optional arguments
+ for var in "AGGREGATOR" "POSTFILE"; do
+ file=$(eval echo \$${var})
+ if [ -n "${file}" ] && [ ! -f "${file}" ]; then
+ echo "${PROGNAME}: No such file: ${file}" >&2
+ exit 1;
+ # NOTREACHED
+ fi
+ done
+
+ PREFIX="${DIR}/chunked/${NAME}"
+
+ rm -fr ${PREFIX}/*
+ mkdir -p ${PREFIX}/input/${FULLDATE} || exit 1
+
+ date > ${PREFIX}/started
+
+ NUM_HOSTS=$(wc -l <${WHOSTS})
+}
+
+# function : status
+# purpose : print out the names of the slaves still working
+# inputs : none; called when SIGINFO is received
+# returns : nothing; prints to stdout the hostnames of all slaves still
+# working
+
+status() {
+ find ${PREFIX}/* -maxdepth 2 -name '.slave' -exec cat {} \; 2>/dev/null
+}
+
+
+# function : usage
+# purpose : print usage
+# inputs : none
+
+usage() {
+ cat <<EOH
+Usage: ${PROGNAME} [-Hh] [-A authmode] [-a aggregator] [-c post-chunk] [-d dir]
+ [-j jobs] [-p pre] [-w workers] -i input -n name -r remote
+ [-- scanslave-passthrough-args]
+ -A authmode specify ssh authentication mode
+ -H do not use the headless user for ssh connections
+ -a aggregator specify a script to aggregate all results
+ -d dir specify a shared directory
+ -h print this message and exit
+ -i input specify the file containing the hostlist
+ -j jobs specify the number of jobs scanhosts(1) should run
+ (passed on to scanslave(1))
+ -n name specify the name of this scan
+ -w workers specify a file containing a list of worker hosts
+ -p pre specify a script to run before doing anything else
+ -r remote specify the script to execute on all target hosts
+EOH
+}
+
+
+# function : waitForAll
+# purpose : wait for all screen sessions on all hosts to complete by way
+# of counting "done" files that each finished job drops into
+# the well-known directory
+# inputs : none
+
+waitForAll() {
+ local count time
+
+ while [ 1 ]; do
+ count=$(find ${PREFIX}/* -maxdepth 2 -name '.done' -print 2>/dev/null | wc -l)
+ time=$(date +%T)
+ echo -n ${time}: ${count} /${NUM_HOSTS} done
+ if [ ${count} -eq ${NUM_HOSTS} ]; then
+ echo
+ break
+ fi
+ echo " - waiting..."
+ # sleep in intervals to give SIGINFO a chance to be caught
+ for n in 1 2 3 4 5 6; do
+ sleep 10
+ done
+ done
+}
+
+
+###
+### Main
+###
+
+trap status INFO
+
+while getopts 'A:Ha:c:d:hi:j:n:p:r:w:' opt; do
+ case ${opt} in
+ A)
+ AUTHMODE="${OPTARG}"
+ ;;
+ H)
+ NO_HEADLESS_FLAG="-H"
+ ;;
+ a)
+ AGGREGATOR="${OPTARG}"
+ ;;
+ c)
+ POSTFILE="${OPTARG}"
+ POST="-p ${POSTFILE}"
+ ;;
+ d)
+ DIR="${OPTARG}"
+ ;;
+ h|\?)
+ usage
+ exit 0
+ # NOTREACHED
+ ;;
+ i)
+ INPUT="${OPTARG}"
+ ;;
+ j)
+ JOBS="-n ${OPTARG}"
+ ;;
+ n)
+ NAME="${OPTARG}"
+ ;;
+ p)
+ if [ ! -f "${OPTARG}" ]; then
+ echo "${PROGNAME}: No such file: ${OPTARG}" >&2
+ exit 1;
+ # NOTREACHED
+ fi
+ PRE="${OPTARG}"
+ ;;
+ r)
+ REMOTE_SCRIPT="${OPTARG}"
+ ;;
+ w)
+ WHOSTS="${OPTARG}"
+ ;;
+ *)
+ usage
+ exit 1
+ # NOTREACHED
+ ;;
+ esac
+done
+shift $(($OPTIND - 1))
+
+if [ -z "${INPUT}" -o -z "${REMOTE_SCRIPT}" -o -z "${NAME}" ]; then
+ usage
+ exit 1
+ # NOTREACHED
+fi
+
+if [ x"${1}" = x"--" ]; then
+ shift
+fi
+
+if [ $# -gt 0 ]; then
+ PASSTHROUGH="-- $@"
+fi
+
+prescanTasks
+prepChunks
+farmOutJobs
+waitForAll
+finishTasks
207 src/scanslave
@@ -0,0 +1,207 @@
+#! /bin/sh
+#
+# Copyright (c) 2008,2009,2010,2011 Yahoo! Inc.
+# All rights reserved.
+#
+# Originally written by Jan Schaumann <jschauma@yahoo-inc.com> in July 2007.
+#
+# Redistribution and use of this software in source and binary forms,
+# with or without modification, are permitted provided that the following
+# conditions are met:
+#
+# * Redistributions of source code must retain the above
+# copyright notice, this list of conditions and the
+# following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the
+# following disclaimer in the documentation and/or other
+# materials provided with the distribution.
+#
+# * Neither the name of Yahoo! Inc. nor the names of its
+# contributors may be used to endorse or promote products
+# derived from this software without specific prior
+# written permission of Yahoo! Inc.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# This script is part of the "scanmaster" suite of scripts. It represents
+# the portion of the process that runs on each scanning node and invokes
+# the scanhosts command. It takes as input the name of a scan and the
+# so-called chunkfile containing the subset of hosts this scanning node
+# should work on.
+#
+# See scanslave(1) for details.
+
+#set -eu
+
+AUTHMODE="pubkey"
+NUM=175
+DIR="/mnt/scanmaster"
+NO_HEADLESS_FLAG=""
+PROGNAME="${0##*/}"
+POST="./post.chunk.sh"
+REMOTE="./remote.sh"
+TDIR="${TMPDIR:-"/tmp"}"
+
+DATE=$(date +%Y%m%d%H%M)
+OUTDIR=""
+
+# set further below
+PREFIX=""
+
+###
+### Subroutines
+###
+
+# function : postScan
+# purpose : run anything needed after a scan has completed
+# inputs : none
+
+postScan() {
+ local tmpct=$(mktemp ${TDIR}/cr.$$.XXXX)
+
+ date > ${OUTDIR}/.done
+ rm ${OUTDIR}/.slave
+
+ crontab -l | sed -e "/tkill -C '.checkhosts-s/d" >${tmpct}
+ crontab ${tmpct}
+ rm ${tmpct}
+
+ if [ -n "${SSH_AGENT_PID}" ]; then
+ kill "${SSH_AGENT_PID}"
+ fi
+}
+
+
+# function : preScan
+# purpose : run anything needed before we kick off the scan
+# inputs : none
+
+preScan() {
+ local tmpct=$(mktemp ${TDIR}/cr.$$.XXXX)
+
+ OUTDIR="${PREFIX}/${SCAN}/${DATE}.${CHUNKFILE##*/}"
+
+ mkdir -p ${OUTDIR} || exit 1
+ hostname > ${OUTDIR}/.slave
+
+ crontab -l >${tmpct}
+ echo "* * * * * tkill -C '.checkhosts-ssh' -t 60" >> ${tmpct}
+ echo "* * * * * tkill -C '.checkhosts-scp' -t 60" >> ${tmpct}
+ crontab ${tmpct}
+ rm ${tmpct}
+
+ if [ x"${NO_HEADLESS_FLAG}" = x"-H" ] && [ x"${AUTHMODE}" != x"password" ]; then
+ eval $(ssh-agent)
+ ssh-add ${SSH_ADD_FLAGS}
+ fi
+}
+
+# function : usage
+# purpose : print usage
+# inputs : none
+
+usage() {
+ cat <<EOH
+Usage: ${PROGNAME} [-Hh] [-A authmode] [-d dir] [-n N] [-p post] [-r remote] scan chunkfile
+ -A authmode specify ssh authentication mode
+ -H do not use the headless user for ssh connections
+ -d dir the directory under which to create all output
+ -h print this help and exit
+ -n N the number of jobs scanhosts(1) should execute
+ -p post the location of the per-scan post-processing script
+ -r remote the location of the remote script
+EOH
+}
+
+
+###
+### Main
+###
+
+while getopts 'A:Hhd:n:p:r:' opt; do
+ case ${opt} in
+ A)
+ AUTHMODE="${OPTARG}"
+ ;;
+ H)
+ NO_HEADLESS_FLAG="-H"
+ ;;
+ d)
+ DIR=${OPTARG}
+ ;;
+ h|\?)
+ usage
+ exit 0
+ # NOTREACHED
+ ;;
+ n)
+ NUM=${OPTARG}
+ ;;
+ p)
+ POST="${OPTARG}"
+ if [ ! -f "${POST}" ]; then
+ echo "${PROGNAME}: No such file: ${POST}" >&2
+ exit 1;
+ # NOTREACHED
+ fi
+ ;;
+ r)
+ REMOTE="${OPTARG}"
+ if [ ! -f "${REMOTE}" ]; then
+ echo "${PROGNAME}: No such file: ${REMOTE}" >&2
+ exit 1;
+ # NOTREACHED
+ fi
+ ;;
+ *)
+ usage
+ exit 1
+ # NOTREACHED
+ ;;
+ esac
+done
+shift $(($OPTIND - 1))
+
+if [ $# -lt 2 ]; then
+ usage
+ exit 1
+ # NOTREACHED
+fi
+
+SCAN=${1}
+CHUNKFILE=${2}
+shift
+shift
+
+if [ x"${1}" = x"--" ]; then
+ shift
+fi
+
+PREFIX="${DIR}/chunked"
+
+preScan
+
+AUTOPW=""
+if [ x"${NO_HEADLESS_FLAG}" = x"-H" ]; then
+ AUTOPW="autopw"
+fi
+
+${AUTOPW} scanhosts \
+ $@ \
+ -A ${AUTHMODE} -n ${NUM} \
+ -f ${CHUNKFILE} -o ${OUTDIR} -s ${DATE} -p ${POST} \
+ -r ${REMOTE} ${NO_HEADLESS_FLAG}
+
+postScan
247 src/tkill.py
@@ -0,0 +1,247 @@
+#! /usr/bin/env python
+#
+# Copyright (c) 2007,2011 Yahoo! Inc.
+# All rights reserved.
+#
+# Originally written by Jan Schaumann <jschauma@yahoo-inc.com> in July 2007.
+#
+# Redistribution and use of this software in source and binary forms,
+# with or without modification, are permitted provided that the following
+# conditions are met:
+#
+# * Redistributions of source code must retain the above
+# copyright notice, this list of conditions and the
+# following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the
+# following disclaimer in the documentation and/or other
+# materials provided with the distribution.
+#
+# * Neither the name of Yahoo! Inc. nor the names of its
+# contributors may be used to endorse or promote products
+# derived from this software without specific prior
+# written permission of Yahoo! Inc.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+#
+# This program kills a given process or process group if it has been running
+# for longer than a specified time.
+#
+# See http://mail-index.netbsd.org/tech-userlevel/2007/07/26/0003.html for
+# a patch to NetBSD's pkill(1) sources.
+
+import getopt
+import os
+import re
+import string
+import sys
+import time
+
+###
+### Globals
+###
+
+EXIT_ERROR = 1
+EXIT_SUCCESS = 0
+
+PS_CMD = "ps -awwx -o lstart,pid,ppid,command"
+
+SIGNALS = [ "HUP", "INT", "QUIT", "ILL", "TRAP", "ABRT", "EMT", "FPE",
+ "KILL", "BUS", "SEGV", "SYS", "PIPE", "ALRM", "TERM", "URG",
+ "STOP", "TSTP", "CONT", "CHLD", "TTIN", "TTOU", "IO", "XCPU",
+ "XFSZ", "VTALRM","PROF","WINCH","INFO", "USR1", "USR2", "PWR" ]
+
+CMD = ""
+LOOSE_MATCH = False
+PARENT = False
+PIDS = []
+SIG = "TERM"
+TIMEOUT = 300
+
+###
+### Subroutines
+###
+
+def usage():
+ """print short usage"""
+
+ print 'Usage: %s [-P] [-h] [-l] [-t timeout] [-s signal] (pid [...] | -c cmd)' \
+ % sys.argv[0]
+ print '\t-C cmd kill processes containing the given command string'
+ print '\t-P kill all processes whose parent is pid'
+ print '\t-c cmd kill processes matching given command exactly'
+ print '\t-h print this help and exit'
+ print '\t-l print list of valid signals'
+ print '\t-t timeout specify timeout in seconds (default: 300)'
+ print '\t-s signal kill with given signal (default: TERM)'
+
+
+
+def isValidSignal(sig):
+ """validate the given signal by checking if it is a valid numerical signal
+ or as a string in the array of signals we know"""
+
+ try:
+ s = string.atoi(sig)
+ if (s < 1) or (s > 32):
+ raise ValueError("illegal numerical signal %d" % sig)
+ except:
+ for s in SIGNALS:
+ if sig == s:
+ return sig
+ raise ValueError("illegal numerical signal %s" % sig)
+
+ return sig
+
+
+
+def parseOpts():
+ """parse command-line options and act on them"""
+
+ global CMD, LOOSE_MATCH, PARENT, SIG, TIMEOUT
+
+ try:
+ opts, args = getopt.getopt(sys.argv[1:], "C:Pc:hlt:s:")
+ except getopt.GetoptError:
+ usage()
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+
+ for o, a in opts:
+ if o in ("-C"):
+ CMD = a
+ LOOSE_MATCH = True
+ if o in ("-P"):
+ PARENT = True
+ if o in ("-c"):
+ CMD = a
+ if o in ("-h"):
+ usage()
+ sys.exit(EXIT_SUCCESS)
+ # NOTREACHED
+ if o in ("-l"):
+ for s in SIGNALS:
+ sys.stdout.write("%s " % s)
+ sys.stdout.write("\n")
+ sys.exit(EXIT_SUCCESS)
+ # NOTREACHED
+ if o in ("-t"):
+ try:
+ TIMEOUT = string.atoi(a)
+ except:
+ sys.stderr.write('%s: -t: not a number: "%s"\n' % (sys.argv[0], a))
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+ if o in ("-s"):
+ try:
+ SIG = isValidSignal(a)
+ except ValueError:
+ sys.stderr.write('%s: -s: bad signal `%s\'\n' % (sys.argv[0], a))
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+
+ if args:
+ if CMD:
+ usage()
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+ else:
+ try:
+ for p in args:
+ PIDS.append(string.atoi(p))
+ except ValueError:
+ sys.stderr.write('%s: bad pid `%s\'\n' % (sys.argv[0], p))
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+ else:
+ if not CMD:
+ usage()
+ sys.exit(EXIT_ERROR)
+ # NOTREACHED
+
+
+
+def isStalePid(start):
+ """return true if the given processes running time has been longer than
+ the limit"""
+
+ global TIMEOUT
+
+ now = time.time()
+ if start < (now - TIMEOUT):
+ return True
+
+ return False
+
+
+
+def killAllStalePids():
+ """open a pipe to the PS command and parse the output"""
+
+ global CMD, LOOSE_MATCH, PARENT, PIDS, PS_CMD
+
+ ps = os.popen(PS_CMD)
+ while 1:
+ line = ps.readline()
+ if not line:
+ break
+ p = re.compile("^(\S+ \S+ +\d+ [0-9:]+ \d+)\s+(\d+)\s+(\d+)\s+(.*)$")
+ m = p.match(line)
+
+ if not m:
+ continue
+
+ (start, pid, ppid, comm) = m.groups()
+
+ t = time.strptime(start)
+ start = time.mktime(t)
+
+ pid = string.atoi(pid)
+ ppid = string.atoi(ppid)
+
+ if ( ((CMD == comm.strip()) or \
+ (LOOSE_MATCH and (comm.find(CMD) >= 0))) \
+ and isStalePid(start) ):
+ killPid(pid)
+
+ for p in PIDS:
+ if not isStalePid(start):
+ continue
+ if PARENT:
+ if p == ppid:
+ killPid(p)
+ else:
+ if p == pid:
+ killPid(p)
+
+
+
+def killPid(p):
+ """kill a process"""
+
+ s = SIGNALS.index(SIG) + 1
+ try:
+ os.kill(p,s)
+ except Exception, (errno, strerror):
+ sys.stderr.write('%s: unable to kill PID %d: %s\n' % \
+ (sys.argv[0], p, strerror))
+
+###
+### Main
+###
+
+if __name__ == "__main__":
+ parseOpts()
+ killAllStalePids()
Please sign in to comment.
Something went wrong with that request. Please try again.