NAME

pfor - A server-friendly (parallel) blast program

DESCRIPTION

pfor executes a series of commands, usually on remote sites, usually in parallel. Pfor should run in a way that shouldn't trash the server running pfor. There is some facility for verifying hostnames and specifying hostnames with patterns. The expansion is done with the help of an external program. But since pfor is general, the preceding description may be inadequate.

USAGE

pfor [simple switches...] [execution switches...] [hostnames or hostname patterns...]

See "Simple Switches" and "Execution-mode Switches" and "Hostname Patterns".

Simple and Execution-mode switches may be interspersed, but the hostnames and hostname patterns must come at the end of the argument list.

Simple Switches

Simple Switches include:

-a

Arguments. Show how pfor interpreted the arguments.

-h

Help. Give a help message and exit. If pfor is run with what it thinks are invalid arguments, it gives an abbreviated version of this help.

-n

No run. Don't really run anything, just show what pfor would do. I recommend using this switch to test any complex pfor invocation first.

-t

Terse. Don't show host separator bar or give a message when a command completes with an error return.

-p

Parallel. Execute in parallel. However note that the default -z implies -p. If the -z switch is not used, the parent of forked processes becomes init (process id 1). This is done so that if the pfor program is killed processes are not zombies.

-v

Verbose. Show the ssh and scp commands.

-z

(Obsolete) Zombies. Execute in parallel but this program stays the parent and terminates only after all of its kids. Use this switch if you want to make sure all output is done by the time this program completes, or if you need to know when all commands have been executed. This may occur if the program is executed from another.

Since this is the default, it needn't be specified anymore.

-sequential

Sequential. Don't fork processes but run them one at a time.

-nohostcheck

Don't verify that trailing arguments are valid hostnames.

-nopattern

Don't do pattern expansion on the hostnames. Without this switch /usr/local/dns/dns_extract.pl is called to do the actual expansion. See "Hostname Patterns" below.

-show_hostnames

list hostnames

-abort_on_hostname_error

Abort processing if any hostname error detected. If this switch is not supplied, invalid hostnames will be removed but the commands will be run against the remaining (valid) hosts.

In cases where you want to defer processing unless all hosts are valid, use this switch. Note however just because all hostnames are valid does not mean that the host is reachable. See also the -errhostf switch.

-logf logfile

Log the results of execution to logfile. Without this switch the output of the commands are sent to standard output.

-hostf hostfile

Read the list of hosts to run commands from file hostfile. The file should contain one hostname per line. Blank lines are ignored and comments can be entered by inserting # in column 1 (or after a newline, for you Unix bigots). See the section on Example 1: introductory use below for an example of the -hostf switch and for the contents of a simple hostfile.

-errhostf logfile

When a nonzero return code is returned from the execution of a command (from pfor's standpoint), the name of the host is logged to logfile. The output is added as a comment before the host name in the file. This file is in a suitable format to be used on a subsequent pfor run with this logfile as the argument of the -hostf switch.

-abort_on_hostname_error

Execution-mode Switches

The section on "Simple Switches" above describes switches that modify the behavior of pfor slightly. The switches described here are the real meat of pfor: what gets done.

The heart of pfor contains an "engine" for running a series of commands and managing the execution. The mechanism is pretty simple and general: an array of list of commands to run and an array of CPUs to run them on. However in order to make it easier to do common tasks, some canned command sequences have been built in. In particular:

• Run an executable stored locally at a remote site, removing the file after successful execution. See the -lcmdf switch below.

• Copy a file stored locally to a remote site, like scp. See the -scp switches below.

• Copy a rpm and and install that. See the -rpm switch below

• Untar a file across the network. See the -tarf switch below.

• Untar a file across the network and then run an installation program contained in the file. See the -tar_install switch below

• Finally, there is a way just to specify how to put commands into the command array. To be useful though one needs to get access to the thing that's being looped over, usually a hostname. See the -exec2 switch below.

It usually doesn't make sense to copy a file to a remote site and untar it across the network; or execute a tar file remotely and then remove it (that's usually done with an executable file). So, for the most part, the switches described here are mutually exclusive. However one of these switches need to be specified.

Now to the specific list of switches...

-ssh commands

Execute commands on the specified hosts. An implied ssh $host is prepended to the beginning of the command.

In order to keep pfor's grubby while loops from interpreting the blank-delimited arguments intended for the command, you should surround the command and its arguments in some sort of quotes (or more cumbersome: escape the blanks). If the command also contains shell expansion characters (like *), probably the best way to go with most shells is with single quotes.

See "Example 1: introductory use" and "Example 2: running several commands remotely using -ssh" for some simple invocations of this format.

-scp local-file -rcmdf remote-path-or-file

Copy local-file to remote-path-or-file. If remote-path-or-file is omitted, the file will be copied to the the same place as local-file as scp would do.

If remote-cmdfile ends in a /, then the name is interpreted as a directory and the copy will have the same basename as cmdfile. Again the same as scp would do.

See "Example 6: running scp in parallel using -scp" for an example of this kind of invocation.

-rpm rpm-file

Copy rpm-file and then run the rpm command to install that.

-lcmdf cmdfile [-rcmdf remote-cmdfile] [-args arguments] [-nrsh]

Copy cmdfile to remote-cmdfile supplying arguments; then run cmdfile and finally remove remote-cmdfile. If remote-cmdfile is omitted, the file will be copied to the /tmp directory.

If remote-cmdfile ends in a slash (/), then the name is interpreted as a directory and the copy will have the same basename as cmdfile.

See "Example 4: Copying and running a program remotely using the -lcmdf switch" for some examples of this kind of invocation.

The -nrsh switch is probably historical laziness. Probably you can do what you want with -scp with less hassle.

If the -nrsh switch is used, the effect is like scp, except that the default destination is in the /tmp directory.

-tarf tar-file

Untar tar-file across the network to the hosts specified. If the <tar-file> extension ends in .gz or .Z the program is uncompressed courtesy of GNU tar.

In particular, run:

cat tar-file ⎪ "rsh $host 'cd / && tar -xvpf -'"

or

cat tar-file ⎪ "rsh $host 'cd / && /usr/local/bin/tar -xzvpf -'"

-tar_install tar-file

Untar a tar-file across the network to the hosts specified. Then run an installation program. The installation program is assumed be in the tar-file in the /tmp directory and having the same name as the tar file (minus the .tar extension).

For example if tar-file is /usr/sa/rockyshits/myfavorites.tar there would be an executable called tmp/myfavorites in the tar file. Presumably this code would do installation on the host after the tar file has be extracted across the network.

But some code is worth a thousand words. Here's what pfor really does:

cat path/pkg.tar ⎪ "rsh $host 'cd / && tar -xvpf -'" && /tmp/pkg && rm /tmp/pkg

where pkg is the name of the package. If the tar file ends in .gz or .Z the -z switch is added and /usr/local/bin/tar (GNU tar) is used.

See "Example 3: blasting out a tar file using the -tarf switch" for an example of this format.

-exec2 commands

This is the most general form for running commands. The only thing that gets run over each argument (usually hostname) in pfor's loop is what you specify. To get access to the loop variable use $host. See "Example 5: running locally using -exec2" below.

Hostname Patterns

pfor owes its existence to its ability to run commands looping over host names. Thus the final arguments to pfor may specify hostnames. To make it easy to blast, the arguments can either be hostnames or patterns which will be expanded to hostnames.

Unless the -nohostcheck switch is specified, a gethostbyname() library call is issued to determine whether an argument is a host name or pattern. If this command returns without error, the argument is a hostname; otherwise it is treated as a pattern.

A hostname pattern expands to a list of hostnames. When available, the program /usr/local/dns/dns_extract.pl is called to do the actual expansion. So you might want to consult that for a description of patterns. Patterns beginning with ! may have to be escaped so the shell doesn't interpret them. See the section on "Example 5: running locally using -exec2" below.

Non-comment entries contained in a hostfile specified by -hostf are treated exactly as a hostname or hostname pattern would be if specified on the command line.

EXAMPLES

Example 1: introductory use

I've sometimes wondered what's the workstation that so-and-so uses. Since who shows who is logged onto the workstation, the user that appears most often is probably the person sitting at the console. So to glean the non-macho users sitting at a given workstation try:

pfor -p -ssh who drmcl002 drmcl021 drmcl004 drmcl005 canton

This is like the POSIX shell code:

for host in drmcl002 drmcl0021 drmcl004 drmcl005 canton ; do
   ssh $host who &
done

But there is are some subtle differences between this and the above shell code.

First, pfor is more caring about the server that the command is run on. If the load average is too high, pfor will delay in forking more processes. Also pfor will not forge ahead if the AIX/SunOS's network card's memory buffers (mbufs) are overloaded.

Second, the above shell program may intersperse output from various workstations making it hard to tell who's logged on where. pfor will give all output from a site together with a separator line to indicate where the output came from.

If you actually try it (I just did) you may find that drmcl021 doesn't want to respond to a who command. So you may find that the output for drmcl021 comes after the output for canton.

Instead of typing out the names of the hosts, it might be more convenient to put them all into a file and point pfor at that.

That's what the -hostf switch is all about. So that above example could be done as:

pfor -z -hostf /usr/local/etc/banner.hostf -ssh who

where /urs/local/etc/banner.hostf is a file containing:

banner1

# Blank lines and lines beginning with # are comments.

banner2
%include banner-vms.hostf

Now let's change that so that we just get the users and remove all of that other crap. Instead of who we really want to run

export PATH=/usr
who ⎪ awk '{ print $1 }' ⎪ sort ⎪ uniq

One way to do this is to put this in a command file with #!/bin/ksh and setting a PATH so we know what programs we're getting. That is:

#!/bin/ksh
export PATH=/usr/bin
who ⎪ awk '{ print $1 }' ⎪ sort ⎪ uniq

Let's say the above was put in whouniq.ksh. Then we could run:

pfor -z -hostf /usr/local/etc/banner.hostf -lcmdf whouniq.ksh

Example 2: running several commands remotely using -ssh

To get some statistics about servers sa, bonzo, and filesvr, try this:

pfor -z -ssh 'uptime;pstat -s;vmstat;iostat' sa bonzo filesvr

In contrast to the examples above, the invocations is done in parallel since the -z switch is not used: first sa is tried; after that completes bonzo, and after that completes filesvr.

Note that the argument to -ssh is enclosed in quotes to keep the shell and pfor from munging it. Similarly, if you wanted to list all files in /usr/local on the sites listed in file /usr/local/etc/cpu10.hosts you might try:

pfor -ssh 'ls -l /usr/local/*' -hostf /usr/local/etc/cpu10.hosts

Again the single quotes keep the shell from expanding * on the server issuing pfor before pfor even gets to see the command.

Example 3: blasting out a tar file using the -tarf switch

To blast out home-opcuser.tar to the sites listed in file cpu27.hosts and log the results:

pfor -logf /tmp/opcuser.log -tarf 'home-opcuser.tar' -hostf cpu27.hosts

Note that if this is called from another program, the log file will be complete before pfor returns.

But I'd recommend that before you really do something like this to lots of sites, you consider using the -n switch to see what damage you may be about to do.

Example 4: Copying and running a program remotely using the -lcmdf switch

To execute script /rockyshits/fix_io.ksh on all branch servers:

pfor -lcmdf /rockyshits/fix_io.ksh -hostf /usr/local/dns/branch_servers

Here is an equivalent, but longer, way to do this:

pfor -lcmdf /rockyshits/fix_io.sh -rcmdf /tmp/ -hostf /usr/local/dns/branch_servers

Since the argument of -rcmdf ends in a slash (/tmp/) it is taken to be directory name. Therefore the file /rockyshits/fix_io.sh is copied to $host:/tmp/fix_io.ksh.

Example 5: running locally using -exec2

Suppose we want to get the IP addresses for all of the cpu27's except the wire code for Boston, jj. The /usr/bin/host command run locally will probably do the trick. So try:

pfor -exec2 '/usr/bin/host $host' cpu27 \!jj

Note that here we don't have to run any command remotely. So -exec2 is used. The variable $host expands to each host that matches the pattern cpu27 !jj. As of Oct '95 this pattern expands to:

cpu14.cg, cpu14.cp, cpu14.fn, cpu14.jg, cpu14.jx, cpu14.pw,
cpu14.ts, cpu14.we

In order for pattern expansion to work, the program /usr/local/dns/dns_extract.pl needs to be around. Right now, it's only available on sa.

Example 6: running scp in parallel using -scp

To copy root's /.rhost to cpu10's:

pfor -scp /.rhost cpu10

This also can be accomplished by:

pfor -scp /.rhost -rcmdf / cpu10

or even more verbosely:

pfor -scp /.rhost -rcmdf /.rhosts cpu10

Local Server Friendliness

Watch this space for a description netstat -m, load average checking, and handling of zombies.

BUGS

The program is necessarily complex. However the user interface could be better. Blame it on Jim; he's not around anyway.

The program is now a bit over bloated. It's a little bit AIX/SunOS oriented (netstat -m doesn't cut it on Solaris).

More sophisticated command sequences as are found in -tarf and -tar_install will probably be added.

For example, checking checksums on untarring across the network should be done. A mechanism for running a program locally after a tar install might be needed. A mechanism for catching error codes remotely is needed: ssh doesn't cut it; something more like /home/opcuser/execute might be used. And so on.

All this means more complexity; but perhaps it can be added in a clean way.

I've offloaded the DNS pattern part to another program dns_extract.pl. That program should be using nslookup or dig to get DNS info if local maps are not available.

History

Rocky Bernstein wrote the first cuts of the program in desperation. He then convinced Jim Anderson that it'd be a fun thing to work on. Jim handled the program during his brief tenure here. Rocky also wrote the first cut of the manual after Sean Coates suggested that it was used enough to warrant one. (Any volunteers for program/manual maintenance?)

Since it's come up a lot, here's how the name pfor was "chosen." There was a program of the same name written by Andy Lowry (at Morgan- Stanley, but I knew him from IBM Watson). Andy stole the name from a construct in the IBM "Parallel Fortran" compiler. So pfor stands for "Parallel For." Maybe it'd be simpler if we renamed it to "blast."

The program has gotten better (or worse), but definitely more complex from suggestions by numerous PaineWebber folk. The people I can think of offhand are: Joel Weisberg and Spencer Siu.

POD ERRORS

Hey! The above document had some coding errors, which are explained below:

Around line 244:

Non-ASCII character seen before =encoding in '⎪'. Assuming UTF-8