install.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\label{sec:install}Installation, Start Up, Shut Down, and Reconfiguration}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This section contains the instructions for installing HTCondor.
The installation will have a default configuration that can
be customized.
Sections of the manual below explain customization.

Please read this \emph{entire} section before starting installation.

Please read the copyright and disclaimer information in
section~\ref{sec:license}.
Installation and
use of HTCondor is acknowledgment that you have read and agree to the
terms.

Before installing HTCondor, please consider joining the htcondor-world mailing
list.
Traffic on this list is kept to an absolute minimum;
it is only used to announce new releases of HTCondor.
To subscribe, go to
\URL{https://lists.cs.wisc.edu/mailman/listinfo/htcondor-world},
and fill out the online form.

You might also want to consider joining the htcondor-users mailing list.
This list is meant to be a forum for HTCondor users to learn from each
other and discuss using HTCondor. It is an excellent place to ask
the HTCondor community about using and configuring HTCondor.
To subscribe, go to
\URL{https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users},
and fill out the online form.

\Bold{Note that forward and reverse DNS lookup must be enabled
for HTCondor to work properly.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:pre-install-procedure}
Obtaining the HTCondor Software}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\index{installation!download}
\index{Unix installation!download}
\index{download}
The first step to installing HTCondor is to download it from the HTCondor
web site, \URL{http://htcondor.org/}.
The downloads are available from the downloads page,
at \URL{http://htcondor.org/downloads/}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Unix-Install}
Installation on Unix}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The HTCondor binary distribution is packaged in the following files
and directories:

\begin{description}
\item[\File{LICENSE-2.0.txt}] the licensing agreement.
                  By installing HTCondor, you agree to the contents of
		  this file
\item[\File{README}] general information
\item[\File{bin}] directory which contains the distribution HTCondor
		  user programs.
\item[\File{bosco\_install}] the Perl script used to install Bosco.
\item[\File{condor\_configure}] the Perl script used to install and
                  configure HTCondor.
\item[\File{condor\_install}] the Perl script used to install HTCondor.
\item[\File{etc}] directory which contains the distribution HTCondor
		  configuration data.
\item[\File{examples}] directory containing C, Fortran and C++ example
		  programs to run with HTCondor.
\item[\File{include}] directory containing HTCondor header files.
\item[\File{lib}] directory which contains the distribution HTCondor
		  libraries.
\item[\File{libexec}] directory which contains the distribution HTCondor
		  auxiliary programs for use internally by HTCondor.
\item[\File{man}] directory which contains the distribution HTCondor
		  manual pages.
\item[\File{sbin}] directory containing HTCondor daemon binaries and admin
		  tools.
\item[\File{src}] directory containing source for some interfaces.
\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:Preparing-to-Install}Preparation} 

Before installation, you need to make a few important
decisions about the basic layout of your pool.
These decisions answer the following questions:

\begin{enumerate}
\item What machine will be the central manager?
\item What machines should be allowed to submit jobs?
\item Will HTCondor run as root or not?
\item Who will be administering HTCondor on the machines in your pool?
\item Will you have a Unix user named condor and will its home directory be
   shared? 
\item Where should the machine-specific directories for HTCondor go?
\item Where should the parts of the HTCondor system be installed? 
	\begin{itemize}
	\item Configuration files
	\item Release directory
		\begin{itemize}
		\item user binaries
		\item system binaries 
		\item \File{lib} directory
	  	\item \File{etc} directory
		\end{itemize}
	\item Documentation
	\end{itemize}
\item Am I using AFS?
\item Do I have enough disk space for HTCondor?
\end{enumerate}

% Note: I guess this is a description rather than an enumerate because
% enumerate isn't bold.  Need to make sure to keep numbers correct!
% wenger 2017-01-12
\begin{description}

\item[1. What machine will be the central manager?]

One machine in your pool must be the central manager.
\index{central manager!installation issues}
Install HTCondor on this machine first.
This is the centralized information repository for the HTCondor pool,
and it is also the
machine that does match-making between available machines and
submitted jobs.
If the central manager machine crashes, any currently active
matches in the system will keep running, but no new matches will be
made.  Moreover, most HTCondor tools will stop working.  Because of the
importance of this machine for the proper functioning of HTCondor,
install the central manager on a machine that is likely to stay up all the
time, or on one that will be rebooted quickly if it does crash.

Also consider
network traffic and your network layout when choosing your central
manager.
All the daemons send updates (by default, every 5 minutes) to this machine.
Memory requirements for the central manager differ by the number of machines
in the pool:
a pool with up to about 100 machines will require approximately
25 Mbytes of memory for the central manager's tasks,
and a pool with about 1000 machines will require approximately
100 Mbytes of memory for the central manager's tasks.

A faster CPU will speed up matchmaking. 

Generally jobs should not be either submitted or run on the central
manager machine.

\item[2. Which machines should be allowed to submit jobs?]

HTCondor can restrict the machines allowed to submit jobs.  Alternatively, 
it can allow any machine the network allows to connect to a submit machine
to submit jobs.  If the HTCondor pool is behind a firewall, and all machines
inside the firewall are trusted, the \Macro{ALLOW\_WRITE} configuration
entry can be set to */*.  Otherwise, it should be set to reflect
the set of machines permitted to submit jobs to this pool.
HTCondor tries to be secure by default: it is shipped with an invalid value
that allows no machine to connect and submit jobs.

\item[3. Will HTCondor run as root or not?]

\index{installation!running as root}
We strongly recommend that the HTCondor daemons be installed and run
as the Unix user root.
Without this,
HTCondor can do very little to enforce security and policy
decisions.
You can install HTCondor as any user;
however there are serious security and performance consequences
do doing a non-root installation.
Please see section~\ref{sec:Non-Root}
in the manual for the details and ramifications of
installing and running HTCondor as a Unix user other than root.

\item[4. Who will administer HTCondor?]

\index{HTCondor!Unix administrator}
\index{Unix administrator}
\index{Unix user!root}

% administrator is a person, not an account
% responsibilities of the administrator needed here
% 1. to install
% 2. to customize policies
% 3. to receive e-mail

%TEMP -- I'd like to re-word this...
Either root will be administering HTCondor directly, or someone else
will be acting as the HTCondor administrator.  If root has delegated
the responsibility to another person, keep in mind that as long as
HTCondor is started up as root, it should be clearly understood that
whoever has the ability to edit the condor configuration files can
effectively run arbitrary programs as root.

The HTCondor administrator will be regularly updating HTCondor by following
these instructions or by using the system-specific installation methods below.
The administrator
will also customize policies of the HTCondor submit and execute nodes.
This person will also receive information from HTCondor if 
something goes wrong with the pool, 
as described in the documentation of the \Macro{CONDOR\_ADMIN}
configuration variable.

\item[5. Will you have a Unix user named condor, and will its home
directory be shared?]

\index{Unix user!condor}
To simplify installation of HTCondor, you should
create a Unix user named condor on all machines in the pool.
The HTCondor daemons will create files
(such as the log files) owned by this user,
and the home directory can be used to specify the location of files
and directories needed by HTCondor.  The home directory of this user can
either be shared among all machines in your pool, or could be a
separate home directory on the local partition of each machine.  Both
approaches have advantages and disadvantages.  Having the directories
centralized can make administration easier, but also concentrates the
resource usage such that you potentially need a lot of space for a
single shared home directory.  See the section below on
machine-specific directories for more details.

Note that the user condor must not be an account into which a person
can log in.
If a person can log in as user condor, 
it permits a major security breach,
in that the user condor could submit jobs that run as any other user,
providing complete access to the user's data by the jobs. 
A standard way of not allowing log in to an account on Unix platforms
is to enter an invalid shell in the password file.

If you choose not to create a user named condor,
then you must specify either via the
\index{CONDOR\_IDS environment variable}
\index{environment variables!CONDOR\_IDS}
\Env{CONDOR\_IDS} environment variable or the \Macro{CONDOR\_IDS}
config file setting which uid.gid pair should be used for
the ownership of various HTCondor files.  
See section~\ref{sec:uids} on UIDs in HTCondor
in the Administrator's Manual for details.

\item[6. Where should the machine-specific directories for
HTCondor go?]

HTCondor needs a few directories that are unique on every machine in
your pool.  These are 
\File{execute},
\File{spool}, 
\File{log},
(and possibly \File{lock}). 
Generally, all
of them are subdirectories of a single machine specific directory called
the local directory (specified by the \Macro{LOCAL\_DIR} macro
in the configuration file).
\index{owner!of directories}
Each should be owned by the user that HTCondor is to be run as.
Do not stage other files in any of these directories;
any files not created by HTCondor in these directories are subject to removal.

If you have a Unix user named \Login{condor} with a local home directory on each
machine, the \MacroNI{LOCAL\_DIR} could just be user \Login{condor}'s home
directory (\MacroNI{LOCAL\_DIR} = \MacroUNI{TILDE} in the 
configuration file).
If this user's home directory is shared among all machines in your
pool, you would want to create a directory for each host (named by
host name) for the local directory (for example, \MacroNI{LOCAL\_DIR} =
\MacroUNI{TILDE}/hosts/\MacroUNI{HOSTNAME}).  If you do not
have a \Login{condor} account on your machines, you can put these directories
wherever you'd like.
However, where to place the directories will require some
thought, as each one has its own resource needs:

\begin{description}
\index{Unix directory!\File{execute}}
\index{disk space requirement!\File{execute} directory}
\item[\File{execute}] This is the directory that acts as the current working
directory for any HTCondor jobs that run on a given execute machine.
The binary for the remote job is copied into this directory, so
there
must be enough space for it.  (HTCondor will not send a job to a
machine that does not have enough disk space to hold the initial
binary..)  In addition, if the remote job dumps core for some reason,
it is first dumped to the execute directory before it is sent back to
the submit machine.  So, put the execute directory on
a partition with enough space to hold a possible core file from the
jobs submitted to your pool.

\index{Unix directory!\File{spool}}
\index{disk space requirement!\File{spool} directory}
\item[\File{spool}] The \File{spool} directory holds the job queue
and history files,
and the checkpoint files for all jobs submitted from a given machine.
As a result, disk space requirements for the \File{spool} directory
can be quite large,
particularly if users are submitting jobs with very large
executables or image sizes.
By using a checkpoint server
(see section~\ref{sec:Ckpt-Server} on Installing a Checkpoint Server on
for details),
you can ease the disk
space requirements, since all checkpoint files are stored on the
server instead of the spool directories for each machine.  However,
the initial checkpoint files (the executables for all the clusters you
submit) are still stored in the spool directory, so you will need
%
% how much?!?
%
some space, even with a checkpoint server. The amount of space will 
depend on how many executables, and what size they are, that need to be stored
in the spool directory.

\index{Unix directory!\File{log}}
\index{disk space requirement!\File{log} directory}
\item[\File{log}] Each HTCondor daemon writes its own log file,
and each log file is placed
in the \File{log} directory.  You can specify what size you want these files
to grow to before they are rotated,
%
% rotated?  Maybe this is talking about wrapping around to
% overwrite the oldest entries first
%
so the disk space requirements of
the directory are configurable.
The larger the log files, the more
historical information they will hold if there is a problem, but the
more disk space they use up.  If you have a network file system
installed at your pool, you might want to place the log directories in
a shared location (such as \File{/usr/local/condor/logs/\$(HOSTNAME)}),
so that you can view the log files from all your machines in a single
location.  However, if you take this approach, you will have to
specify a local partition for the \File{lock} directory (see below).

\index{Unix directory!\File{lock}}
\item[\File{lock}] HTCondor uses a small number of lock files to synchronize
access to certain files that are shared between multiple daemons.
Because of problems encountered with file locking and network
file systems (particularly NFS), these lock files should be placed on a
local partition on each machine.  By default, they are placed in
the \File{log} directory.  If you place your \File{log}
directory on a network file system partition,
specify a local partition for the
lock files with the \Macro{LOCK} parameter in the configuration file (such as
\File{/var/lock/condor}).

\end{description}

\index{disk space requirement!HTCondor files}
Generally speaking, it is recommended that you do not put these directories
(except \File{lock}) on the same partition as \File{/var},
since if the partition
fills up, you will fill up \File{/var} as well. 
This will cause lots of
problems for your machines.  Ideally, you will have a separate partition
for the HTCondor directories. Then, the only consequence of filling up
the directories
will be HTCondor's malfunction, not your whole machine.

\item[7. Where should the parts of the HTCondor system be installed?]

	\begin{itemize}
	\item Configuration Files
	\item Release directory
		\begin{itemize}
		\item User Binaries
		\item System Binaries 
		\item \File{lib} Directory
	  	\item \File{etc} Directory
		\end{itemize}
	\item Documentation
	\end{itemize}

\label{sec:Config-File-Locations}
\begin{description}
\item[Configuration Files] There can be more than one configuration file.
They allow
different levels of control over how HTCondor is configured on each
machine in the pool.  
The global configuration file is shared by all machines in the pool.
For ease of administration, this file should be located on a shared
file system, if possible.
Local configuration files override settings in the
global file permitting different daemons to run,
different policies for when to start and stop HTCondor jobs, and so on.
There may be configuration files specific to each platform in the pool.
See section~\ref{sec:Multiple-Platforms} on about Configuring HTCondor for
Multiple Platforms for details.

\index{configuration files!location}
The location of configuration files is described in
section~\ref{sec:Ordering-Config-File}.

\item[Release Directory]

Every binary distribution contains a contains
five subdirectories: \File{bin}, \File{etc}, \File{lib}, \File{sbin},
and \File{libexec}. Wherever you
choose to install these five directories we call the release directory
(specified by the \Macro{RELEASE\_DIR} macro in the configuration file).
Each
release directory contains platform-dependent binaries and libraries,
so you will need to install a separate one for each kind of machine in
your pool.  For ease of administration, these directories should be
located on a shared file system, if possible.

\begin{itemize}
     \item User Binaries:

     All of the files in the \File{bin} directory are programs that
     HTCondor users should expect to have in their path.  You could
     either put them in a well known location (such as
     \File{/usr/local/condor/bin}) which you have HTCondor users add to
     their \Env{PATH} environment variable, or copy those files
     directly into a well known place already in the user's PATHs (such as
     \File{/usr/local/bin}).  With the above examples, you could also
     leave the binaries in \File{/usr/local/condor/bin} and put in
     soft links from \File{/usr/local/bin} to point to each program.

     \item System Binaries:

     All of the files in the \File{sbin} directory are HTCondor daemons and
     agents, or programs that only the HTCondor administrator would need
     to run.  Therefore, add these programs only
     to the \Env{PATH} of the HTCondor administrator.

     \item Private HTCondor Binaries:

     All of the files in the \File{libexec} directory are HTCondor
     programs that should never be run by hand, but are only used
     internally by HTCondor. 

     \item \File{lib} Directory:

     The files in the \File{lib} directory are the HTCondor libraries that
     must be linked in with user jobs for all of HTCondor's
     checkpointing and migration features to be used.  \File{lib} also
     contains scripts used by the \Condor{compile} program to help
     re-link jobs with the HTCondor libraries.  These files should be
     placed in a location that is world-readable, but they do not need
     to be placed in anyone's \Env{PATH}.  The \Condor{compile} script checks
     the configuration file for the location of the \File{lib} directory.

     \item \File{etc} Directory:

     \File{etc} contains an \File{examples} subdirectory which holds various
     example configuration files and other files used for installing HTCondor.
     \File{etc} is the recommended location to keep the master copy of your
     configuration files.  You can put in soft links from one of the places
     mentioned above that HTCondor checks automatically to find its
     global configuration file. 
\end{itemize}

\item[Documentation]

The documentation provided with HTCondor is currently available in
     HTML, Postscript and PDF (Adobe Acrobat).  It can be locally installed
     wherever is customary at your site.  You can also find the HTCondor
     documentation on the web at:
     \URL{http://htcondor.org/manual}.

\end{description}

\item[8. Am I using AFS?]

If you are using AFS at your site, be sure to read the
section~\ref{sec:HTCondor-AFS} in the
manual.
HTCondor does not currently have a way to authenticate itself to AFS.
A solution is not ready for
\VersionNotice.
This implies that you are probably not going to want
to have the \Macro{LOCAL\_DIR} for HTCondor on AFS.
However, you can
(and probably should) have the HTCondor \MacroNI{RELEASE\_DIR} on AFS, so
that you can share one copy of those files and upgrade them in a
centralized location.  You will also have to do something special if
you submit jobs to HTCondor from a directory on AFS.  Again, read manual
section~\ref{sec:HTCondor-AFS} for all the details.

\item[9. Do I have enough disk space for HTCondor?]

\index{disk space requirement!all versions}
The compressed downloads of HTCondor currently range from a low of about 13
Mbytes for 64-bit Ubuntu 12/Linux to about 115 Mbytes for Windows.  The
compressed source code takes approximately 17 Mbytes.

In addition, you will need a lot of disk space in the local directory
of any machines that are submitting jobs to HTCondor.  See question 6
above for details on this.

\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:install-rpms}
Unix Installation from an RPM}
\index{installation!using Red Hat RPMs}
\index{RPM installation on Red Hat}

RPMs are available for HTCondor \VersionNotice.
We provide a Yum repository, as well as 
installation and configuration in one easy step.
This RPM installation is currently available for Red Hat-compatible
systems only.
As of HTCondor version 7.5.1, 
the HTCondor RPM installs into File Hierachy Standard locations.

Yum repositories and instructions are at
 \URL{http://htcondor.org/yum/} .
The repositories are named to distinguish stable releases from
development releases and by Red Hat version number. 
The 4 repositories are:
\begin{itemize}
  \item \File{htcondor-stable-rhel6.repo}
  \item \File{htcondor-stable-rhel7.repo}
  \item \File{htcondor-development-rhel6.repo}
  \item \File{htcondor-development-rhel7.repo}
\end{itemize}

Here is an ordered set of steps that get HTCondor running using the RPM.
\begin{enumerate}
\item The HTCondor package will automatically add a \Login{condor} user/group,
if it does not exist already.
Sites wishing to control the attributes of this user/group 
should add the \Login{condor} user/group manually before installation.

\item Download and install the meta-data that describes 
the appropriate YUM repository. 
This example is for the stable series, on RHEL 7.
\footnotesize
\begin{verbatim}
  cd /etc/yum.repos.d
  wget http://htcondor.org/yum/repo.d/htcondor-stable-rhel7.repo
\end{verbatim}
\normalsize
Note that this step need be done only once;
do not get the same repository more than once.

\item Import signing key
The RPMs are signed in the Redhat 6 and RedHat 7 repositories. 
\begin{verbatim}
  wget http://htcondor.org/yum/RPM-GPG-KEY-HTCondor
  rpm --import RPM-GPG-KEY-HTCondor
\end{verbatim}

\item Install HTCondor.
\begin{verbatim}
  yum install condor-all
\end{verbatim}

\item As needed, edit the HTCondor configuration files to customize.
The configuration files are in the directory \File{/etc/condor/} .
Do \emph{not} use \Condor{configure} or \Condor{install} for configuration.
The installation will be able to find configuration files without
additional administrative intervention,
as the configuration files are placed in \File{/etc},
and HTCondor searches this directory.

\item Start HTCondor daemons:
\begin{verbatim}
  /sbin/service condor start
\end{verbatim}

\end{enumerate}

% Alain thinks that upgrades DO work.
%RPM upgrade (\Opt{-u} option) does not currently
%work for HTCondor \VersionNotice.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:install-debs}
Unix Installation from a Debian Package}
\index{installation!using Debian packages}
\index{Debian installation with Debian packages}

Debian packages are available in HTCondor \VersionNotice.
We provide an APT repository, as well as 
installation and configuration in one easy step.
These Debian packages of HTCondor are currently available for 
Debian 7 (wheezy) and Debian 8 (jessie).
As of HTCondor version 7.5.1, 
the HTCondor Debian package installs into File Hierachy Standard locations.

The HTCondor APT repositories are specified at
 \URL{http://htcondor.org/debian/} .
See this web page for repository information.

Here is an ordered set of steps that get HTCondor running.
\begin{enumerate}
\item The HTCondor package will automatically add a \Login{condor} user/group,
if it does not exist already.
Sites wishing to control the attributes of this user/group 
should add the \Login{condor} user/group manually before installation.

\item If not already present,
set up access to the appropriate APT repository;
they are distinguished as stable or development release,
and by operating system. 
Ensure that the correct one of the following release and 
operating system-specific lines is in 
the file \File{/etc/apt/sources.list} .
\footnotesize
\begin{verbatim}
deb http://htcondor.org/debian/stable/ wheezy contrib
deb http://htcondor.org/debian/development/ wheezy contrib
deb http://htcondor.org/debian/stable/ jessie contrib
deb http://htcondor.org/debian/development/ jessie contrib
\end{verbatim}
\normalsize
Note that this step need be done only once;
do not add the same repository more than once.

\item Install and start HTCondor services:
\begin{verbatim}
  apt-get update
  apt-get install condor
\end{verbatim}

\item As needed, edit the HTCondor configuration files to customize.
The configuration files are in the directory \File{/etc/condor/} .
Do \emph{not} use \Condor{configure} or \Condor{install} for configuration.
The installation will be able to find configuration files without
additional administrative intervention,
as the configuration files are placed in \File{/etc},
and HTCondor searches this directory.

Then, if any configuration changes are made, restart HTCondor with
\begin{verbatim}
  /etc/init.d/condor restart
\end{verbatim}

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{sec:unix-install-from-tarball}
Unix Installation from a Tarball}

\Bold{Note that installation from a tarball is no longer the preferred
method for installing HTCondor on Unix systems.  Installation
via RPM or Debian package is recommended if available for your
Unix version.}

An overview of the tarball-based installation process is as follows:
\begin{enumerate}
\item Untar the HTCondor software.
\item Run \Condor{install} or \Condor{configure} to install the software.
\end{enumerate}
Details are given below.

\medskip

After download, all the files are in a compressed, tar format.
They need to be untarred, as
\begin{verbatim}
  tar xzf <completename>.tar.gz
\end{verbatim}
After untarring, the directory will have the Perl scripts
\Condor{configure} and \Condor{install} (and \Bosco{install}), as well as 
\File{bin}, \File{etc}, \File{examples}, \File{include},
\File{lib}, \File{libexec}, \File{man}, \File{sbin},
\File{sql} and \File{src} subdirectories.

\index{installation!with \Condor{configure}}
\index{condor\_configure command}
The Perl script \Condor{configure} installs HTCondor.
Command-line arguments specify all needed information to this
script.  The script can be executed multiple times, to modify or further
set the configuration.  \Condor{configure} has been tested using Perl 5.003.
Use this or a more recent version of Perl.

\Condor{configure} and \Condor{install} are the same program, but have
different default behaviors.  \Condor{install} is identical to
running 
\begin{verbatim}
  condor_configure --install=.
\end{verbatim}
\Condor{configure} and \Condor{install} work on the named directories.
As the names imply, \Condor{install} is used to
install HTCondor, whereas \Condor{configure} is used to modify the
configuration of an existing HTCondor install.

\Condor{configure} and \Condor{install} are completely command-line
driven and are not interactive.  Several command-line arguments are
always needed with \Condor{configure} and \Condor{install}.
The argument
\begin{verbatim}
  --install=/path/to/release
\end{verbatim}
specifies the path to the HTCondor release directories.
The default command-line argument for \Condor{install} is 
\begin{verbatim}
  --install=.
\end{verbatim}
The argument
\begin{verbatim}
  --install-dir=<directory>
\end{verbatim}
or
\begin{verbatim}
  --prefix=<directory>
\end{verbatim}
specifies the path to the install directory.

The argument
\begin{verbatim}
  --local-dir=<directory>
\end{verbatim}
specifies the path to the local directory.

The \verb@--@\Opt{type} option to \Condor{configure}
specifies one or more of the roles that a machine can take on
within the HTCondor pool: central manager, submit or execute.
These options are given in a comma separated list.
So, if a machine is both a submit and execute
machine, 
the proper command-line option is
\begin{verbatim}
  --type=submit,execute
\end{verbatim}

Install HTCondor on the central manager machine first.  If HTCondor
will run as root in this pool (Item 3 above), run \Condor{install} 
as root, and it will install and set the file permissions correctly.  
On the central manager machine, run \Condor{install} as follows.
\begin{verbatim}
% condor_install --prefix=~condor \
	--local-dir=/scratch/condor --type=manager
\end{verbatim}

To update the above HTCondor installation, for example, to also be
submit machine:
\begin{verbatim}
% condor_configure --prefix=~condor \
	--local-dir=/scratch/condor --type=manager,submit
\end{verbatim}

As in the above example, the central manager can also be a submit
point or an execute machine, but this is only recommended for very
small pools.  If this is the case, the 
\verb@--@\Opt{type}
option changes to
\Expr{manager,execute} or \Expr{manager,submit}  or 
\Expr{manager,submit,execute}.

After the central manager is installed, the execute and submit machines
should then be configured.  Decisions about whether to run HTCondor as root
should be consistent throughout the pool. For each machine in the pool,
run

\begin{verbatim}
% condor_install --prefix=~condor \
	--local-dir=/scratch/condor --type=execute,submit
\end{verbatim}

See the \Condor{configure} manual 
page~\pageref{man-condor-configure}
for details.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\label{installed-now-what}
Starting HTCondor Under Unix After Installation}

\index{starting HTCondor!Unix platforms}

Now that HTCondor has been installed on the machine(s), there are a few
things to check before starting up HTCondor.

\begin{enumerate}
\item Read through the \Release{etc/condor\_config} file.  There are a
    lot of possible settings and you should at least take a look at
    the first two main sections to make sure everything looks okay.
    In particular, you might want to set up security for
    HTCondor.  See the section~\ref{sec:Security-Model}
    to learn how to do this.

\item For Linux platforms, run the \Condor{kbdd} to monitor keyboard
    and mouse activity on all machines within the pool that will
    run a \Condor{startd}; these are machines that execute jobs.
    To do this, the subsystem \Expr{KBDD} will need to be added to
    the \MacroNI{DAEMON\_LIST} configuration variable definition.

    For Unix platforms other than Linux,
    HTCondor can monitor the activity of your mouse and keyboard,
    provided that you tell it where to look.  You do this with the
    \Macro{CONSOLE\_DEVICES} entry in the \condor{startd} section of
    the configuration file.  On most platforms, reasonable
    defaults are provided.
    For example, the default device for the mouse
    is 'mouse', since most installations have a soft link from
    \File{/dev/mouse} that points to the right device (such as
    \File{tty00} if you have a serial mouse, \File{psaux} if you have
    a PS/2 bus mouse, etc).  If you do not have a \File{/dev/mouse}
    link, you should either create one (you will be glad you did), or
    change the \MacroNI{CONSOLE\_DEVICES} entry in HTCondor's
    configuration file.
    This entry is a comma separated list, so you can have any
    devices in \File{/dev} count as 'console devices' and activity
    will be reported in the \condor{startd}'s ClassAd as
    \AdAttr{ConsoleIdleTime}.

\item  (Linux only) HTCondor needs to be able to find the \File{utmp} file.
    According to the Linux File System Standard, this file should be
    \File{/var/run/utmp}.  If HTCondor cannot find it there, it looks in
    \File{/var/adm/utmp}.  If it still cannot find it, it gives up.  So, if
    your Linux distribution places this file somewhere else, be sure to
    put a soft link from \File{/var/run/utmp} to point to the real location.

\end{enumerate}

To start up the HTCondor daemons, execute the command
\Release{sbin/condor\_master}.  This is the HTCondor master, whose
only job in life is to make sure the other HTCondor daemons are running.
The master keeps track of the daemons, restarts them if they crash,
and periodically checks to see if you have installed new binaries (and,
if so, restarts the affected daemons).

If you are setting up your own pool, you should start HTCondor on your
central manager machine first.  If you have done a submit-only
installation and are adding machines to an existing pool,
the start order does not
matter.

To ensure that HTCondor is running, you can run either:
\begin{verbatim}
        ps -ef | egrep condor_
\end{verbatim}
or
\begin{verbatim}
        ps -aux | egrep condor_
\end{verbatim}
depending on your flavor of Unix.  
On a central manager machine that can submit jobs as well
as execute them, there will be processes for:
\begin{itemize}
	\item \condor{master}
	\item \condor{collector}
	\item \condor{negotiator}
	\item \condor{startd}
	\item \condor{schedd}
\end{itemize}
On a central manager machine that does not submit jobs nor
execute them, there will be processes for:
\begin{itemize}
	\item \condor{master}
	\item \condor{collector}
	\item \condor{negotiator}
\end{itemize}
For a machine that only submits jobs, there will be processes for:
\begin{itemize}
	\item \condor{master}
	\item \condor{schedd}
\end{itemize}
For a machine that only executes jobs, there will be processes for:
\begin{itemize}
	\item \condor{master}
	\item \condor{startd}
\end{itemize}

Once you are sure the HTCondor daemons are running, check to make sure
that they are communicating with each other.  You can run
\Condor{status} to get a one line summary of the status of each
machine in your pool.

Once you are sure HTCondor is working properly, you should add
\Condor{master} into your startup/bootup scripts (i.e. \File{/etc/rc} ) so
that your machine runs \Condor{master} upon bootup.  \Condor{master}
will then fire up the necessary HTCondor daemons whenever your machine
is rebooted.  

If your system uses System-V style init scripts, you can look in
\Release{etc/examples/condor.boot} for a script that can be used
to start and stop HTCondor automatically by init.  Normally, you would
install this script as \File{/etc/init.d/condor} and put in soft link from
various directories (for example, \File{/etc/rc2.d}) that point back to
\File{/etc/init.d/condor}.  The exact location of these scripts and links
will vary on different platforms.

If your system uses BSD style boot scripts, you probably have an
\File{/etc/rc.local} file.  Add a line to start up
\Release{sbin/condor\_master}.


Now that the HTCondor daemons are running, there are a few things you
can and should do:

\begin{enumerate}
\item (Optional) Do a full install for the \Condor{compile} script.
    \condor{compile} assists in linking jobs with the HTCondor libraries
    to take advantage of all of HTCondor's features.  As it is currently
    installed, it will work by placing it in front of any of the
    following commands that you would normally use to link your code:
    gcc, g++, g77, cc, acc, c89, CC, f77, fort77 and ld.  If you
    complete the full install, you will be able to use
    \condor{compile} with any command whatsoever, in particular, make.
    See section~\ref{sec:full-condor-compile} in the manual for directions.

\item Try building and submitting some test jobs.  See
    \File{examples/README} for details.

\item If your site uses the AFS network file system, see
section~\ref{sec:HTCondor-AFS} in the
manual.

\item We strongly recommend that you start up HTCondor (run the
\Condor{master} daemon) as user root.  If you must start HTCondor as
some user other than root, see section~\ref{sec:Non-Root}.

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{admin-man/install-windows.tex}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Pool-Upgrade}
Upgrading -- Installing a New Version on an Existing Pool}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\index{pool management!installing a new version on an existing pool}
\index{installation!installing a new version on an existing pool}

An upgrade changes the running version of HTCondor
from the current installation to a newer version.
The safe method
to install and start running a newer version of HTCondor
in essence is:
shut down the current installation of HTCondor,
install the newer version,
and then restart HTCondor using the newer version.
To allow for falling back to the current version,
place the new version in a separate directory.
Copy the existing configuration files,
and modify the copy to point to and use the new version,
as well as incorporate any configuration variables that are new or changed
in the new version.
Set the \Env{CONDOR\_CONFIG} environment variable
to point to the new copy of the configuration,
so the new version of HTCondor will use the new configuration when restarted.

As of HTCondor version 8.2.0, 
the default configuration file has been substantially reduced
in size by defining compile-time default values for most configuration
variables.
Therefore,
when upgrading from a version of HTCondor earlier than 8.2.0 
to a more recent version,
the option of reducing the size of the configuration file is an option.
The goal is to identify and use only
the configuration variable values that differ from 
the compile-time default values.  
This is facilitated by using \Condor{config\_val} with 
the \OptArg{-writeconfig:upgrade} argument,
to create a file that behaves the same as the current configuration, 
but is much smaller,
because values matching the default values (as well as some obsolete variables)
have been removed.
Items in the file created by running
\Condor{config\_val} with the \OptArg{-writeconfig:upgrade} argument
will be in the order that they were read from the original configuration files.
This file is a convenient guide to stripping the cruft from 
old configuration files.

When upgrading from a version of HTCondor earlier than 6.8 to more recent version,
note that the configuration settings must be modified for security reasons.
Specifically, the \Macro{HOSTALLOW\_WRITE} configuration variable
must be explicitly changed,
or no jobs can be submitted,
and error messages will be issued by HTCondor tools.

Another way to upgrade leaves HTCondor running. 
HTCondor will automatically restart itself if the \Condor{master} binary
is updated,
and this method takes advantage of this. 
Download the newer version, placing it such that it does not 
overwrite the currently running version.
With the download will be a new set of configuration files;
update this new set with any specializations implemented in the currently
running version of HTCondor.
Then, modify the currently running installation by changing its
configuration such that the path to binaries points instead
to the new binaries.
One way to do that (under Unix) is to use a symbolic link that points 
to the current HTCondor installation directory (for example, \File{/opt/condor}).
Change the symbolic link to point to the new directory. 
If HTCondor is configured to locate its binaries via the symbolic link, 
then after the symbolic link changes, 
the \Condor{master} daemon notices the new binaries and restarts itself. 
How frequently it checks is controlled by the configuration variable 
\Macro{MASTER\_CHECK\_NEW\_EXEC\_INTERVAL}, which defaults 5 minutes.

When the \Condor{master} notices new binaries, 
it begins a graceful restart. 
On an execute machine, 
a graceful restart means that running jobs are preempted. 
Standard universe jobs will attempt to take a checkpoint. 
This could be a bottleneck if all machines in a large pool 
attempt to do this at the same time. 
If they do not complete within the cutoff time specified by the \MacroNI{KILL} 
policy expression (defaults to 10 minutes), 
then the jobs are killed without producing a checkpoint. 
It may be appropriate to increase this cutoff time, 
and a better approach may be to upgrade the pool in stages 
rather than all at once. 

For universes other than the standard universe, jobs are preempted. 
If jobs have been guaranteed a certain amount of uninterrupted run time 
with \MacroNI{MaxJobRetirementTime}, 
then the job is not killed until the specified amount of 
retirement time has been exceeded (which is 0 by default). 
The first step of killing the job is a soft kill signal, 
which can be intercepted by the job so that it can exit gracefully, 
perhaps saving its state. 
If the job has not gone away once the \MacroNI{KILL} expression fires 
(10 minutes by default), 
then the job is forcibly hard-killed. 
Since the graceful shutdown of jobs may rely on shared resources such as disks 
where state is saved, 
the same reasoning applies as for the standard universe: 
it may be appropriate to increase the cutoff time 
for large pools, 
and a better approach may be to upgrade the pool in stages 
to avoid jobs running out of time. 

Another time limit to be aware of is the configuration variable 
\MacroNI{SHUTDOWN\_GRACEFUL\_TIMEOUT}. 
This defaults to 30 minutes. 
If the graceful restart is not completed within this time, 
a fast restart ensues. 
This causes jobs to be hard-killed. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Pool-Shutdown-and-Restart}
Shutting Down and Restarting an HTCondor Pool}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\index{pool management!shutting down HTCondor}
\index{pool management!restarting HTCondor}

All of the commands described in this section are subject to the
security policy chosen for the HTCondor pool.
As such, the commands must be either run from a
machine that has the proper authorization, 
or run by a user that is authorized to issue the commands.
Section~\ref{sec:Security} details the implementation of 
security in HTCondor.

\begin{description}
\item[Shutting Down HTCondor]
There are a variety of ways to shut down all or parts of an HTCondor pool.
All utilize the \Condor{off} tool.

To stop a single execute machine from running jobs,
the \Condor{off} command specifies the machine by host name.
\begin{verbatim}
  condor_off -startd <hostname>
\end{verbatim}
A running \SubmitCmdNI{standard} universe job will be allowed to 
take a checkpoint before the job is killed.
A running job under another universe will be killed.
If it is instead desired that the machine stops running jobs
only after the currently executing job completes, the command is
\begin{verbatim}
  condor_off -startd -peaceful <hostname>
\end{verbatim}
Note that this waits indefinitely for the running job to finish,
before the \Condor{startd} daemon exits.

Th shut down all execution machines within the pool,
\begin{verbatim}
  condor_off -all -startd
\end{verbatim}

To wait indefinitely for each machine in the pool to finish its current
HTCondor job,
shutting down all of the execute machines as they no longer
have a running job,
\begin{verbatim}
  condor_off -all -startd -peaceful
\end{verbatim}

To shut down HTCondor on a machine from which jobs are submitted,
\begin{verbatim}
  condor_off -schedd <hostname>
\end{verbatim}

If it is instead desired that the submit machine shuts down
only after all jobs that are currently in the queue are finished,
first disable new submissions to the queue 
by setting the configuration variable
\begin{verbatim}
  MAX_JOBS_SUBMITTED = 0
\end{verbatim}
See instructions below in section~\ref{sec:Reconfigure-Pool} for how
to reconfigure a pool.
After the reconfiguration, the command to wait for all jobs to complete
and shut down the submission of jobs is
\begin{verbatim}
  condor_off -schedd -peaceful <hostname>
\end{verbatim}

Substitute the option \Opt{-all} for the host name,
if all submit machines in the pool are to be shut down.

\item[Restarting HTCondor, If HTCondor Daemons Are Not Running]
If HTCondor is not running,
perhaps because one of the \Condor{off} commands was used,
then starting HTCondor daemons back up depends on which part of
HTCondor is currently not running.

If no HTCondor daemons are running, then starting HTCondor is a matter
of executing the \Condor{master} daemon.
The \Condor{master} daemon will then invoke all other specified daemons
on that machine.
The \Condor{master} daemon executes on every machine that is to run HTCondor.

If a specific daemon needs to be started up, and the \Condor{master} daemon
is already running, then issue the command on the specific machine with
\begin{verbatim}
  condor_on -subsystem <subsystemname>
\end{verbatim}
where \verb@<subsystemname>@ is replaced by the daemon's subsystem
name.
Or, this command might be issued from another machine in the pool
(which has administrative authority) with
\begin{verbatim}
  condor_on <hostname> -subsystem <subsystemname>
\end{verbatim}
where \verb@<subsystemname>@ is replaced by the daemon's subsystem
name, and \verb@<hostname>@ is replaced by the host name of the
machine where this \Condor{on} command is to be directed.

\item[Restarting HTCondor, If HTCondor Daemons Are Running]
If HTCondor daemons are currently running, but need to be killed and
newly invoked,
the \Condor{restart} tool does this.
This would be the case for a new value of a configuration variable for
which using \Condor{reconfig} is inadequate.

To restart all daemons on all machines in the pool,
\begin{verbatim}
  condor_restart -all
\end{verbatim}

To restart all daemons on a single machine in the pool,
\begin{verbatim}
  condor_restart <hostname>
\end{verbatim}
where \verb@<hostname>@ is replaced by the host name of the
machine to be restarted.

\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\label{sec:Reconfigure-Pool}Reconfiguring an HTCondor Pool}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\index{pool management!reconfiguration}

To change a global configuration variable and have all the
machines start to use the new setting, change the value within the file,
and send a \Condor{reconfig} command to each host.
Do this with a \emph{single} command,
\begin{verbatim}
  condor_reconfig -all
\end{verbatim}

If the global configuration file is not shared among all the machines,
as it will be if using a shared file system,
the change must be made to each copy of the global configuration file
before issuing the \Condor{reconfig} command.

Issuing a \Condor{reconfig} command is inadequate for some
configuration variables.
For those, a restart of HTCondor is required.
Those configuration variables that require a restart are listed in
section~\ref{sec:Macros-Requiring-Restart}.
The manual page for \Condor{restart} is at
~\pageref{man-condor-restart}.