cfg-buildsteps.texinfo

@code{BuildStep}s are usually specified in the buildmaster's
configuration file, in a list that goes into the @code{BuildFactory}.
The @code{BuildStep} instances in this list are used as templates to
construct new independent copies for each build (so that state can be
kept on the @code{BuildStep} in one build without affecting a later
build). Each @code{BuildFactory} can be created with a list of steps,
or the factory can be created empty and then steps added to it using
the @code{addStep} method:

@example
from buildbot.steps import source, shell
from buildbot.process import factory

f = factory.BuildFactory()
f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/"))
f.addStep(shell.ShellCommand(command=["make", "all"]))
f.addStep(shell.ShellCommand(command=["make", "test"]))
@end example

The rest of this section lists all the standard BuildStep objects
available for use in a Build, and the parameters which can be used to
control each.

@menu
* Common Parameters::
* Using Build Properties::
* Source Checkout::
* ShellCommand::
* Slave Filesystem Steps::
* Python BuildSteps::
* Transferring Files::
* Transferring Strings::
* Running Commands on the Master::
* Setting Properties::
* Triggering Schedulers::
* Miscellaneous BuildSteps::
* Writing New BuildSteps::
* Build Step Index::
@end menu

@node Common Parameters
@subsection Common Parameters

All BuildSteps accept some common parameters. Some of these control
how their individual status affects the overall build. Others are used
to specify which @code{Locks} (see @pxref{Interlocks}) should be
acquired before allowing the step to run.

Arguments common to all @code{BuildStep} subclasses:

@table @code
@item name
the name used to describe the step on the status display. It is also
used to give a name to any LogFiles created by this step.

@item haltOnFailure
if True, a FAILURE of this build step will cause the build to halt
immediately. Steps with @code{alwaysRun=True} are still run. Generally
speaking, haltOnFailure implies flunkOnFailure (the default for most
BuildSteps). In some cases, particularly series of tests, it makes sense
to haltOnFailure if something fails early on but not flunkOnFailure.
This can be achieved with haltOnFailure=True, flunkOnFailure=False.

@item flunkOnWarnings
when True, a WARNINGS or FAILURE of this build step will mark the
overall build as FAILURE. The remaining steps will still be executed.

@item flunkOnFailure
when True, a FAILURE of this build step will mark the overall build as
a FAILURE. The remaining steps will still be executed.

@item warnOnWarnings
when True, a WARNINGS of this build step will mark the
overall build as having WARNINGS. The remaining steps will still be
executed.

@item warnOnFailure
when True, a FAILURE of this build step will mark the overall build as
having WARNINGS. The remaining steps will still be executed.

@item alwaysRun
if True, this build step will always be run, even if a previous buildstep
with @code{haltOnFailure=True} has failed.

@item doStepIf
A step can be configured to only run under certain conditions.  To do this, set
the step's @code{doStepIf} to a boolean value, or to a function that returns a
boolean value or Deferred.  If the value or function result is false, then the
step will return SKIPPED without doing anything.  Oherwise, the step will be
executed normally.  If you set @code{doStepIf} to a function, that function
should accept one parameter, which will be the @code{Step} object itself.

@item locks
a list of Locks (instances of @code{buildbot.locks.SlaveLock} or
@code{buildbot.locks.MasterLock}) that should be acquired before starting this
Step. The Locks will be released when the step is complete. Note that this is a
list of actual Lock instances, not names. Also note that all Locks must have
unique names.  See @ref{Interlocks}.

@end table

@node Using Build Properties
@subsection Using Build Properties
@cindex Properties

Build properties are a generalized way to provide configuration
information to build steps; see @ref{Build Properties}.

Some build properties are inherited from external sources -- global
properties, schedulers, or buildslaves.  Some build properties are
set when the build starts, such as the SourceStamp information. Other
properties can be set by BuildSteps as they run, for example the
various Source steps will set the @code{got_revision} property to the
source revision that was actually checked out (which can be useful
when the SourceStamp in use merely requested the ``latest revision'':
@code{got_revision} will tell you what was actually built).

In custom BuildSteps, you can get and set the build properties with
the @code{getProperty}/@code{setProperty} methods. Each takes a string
for the name of the property, and returns or accepts an
arbitrary@footnote{Build properties are serialized along with the
build results, so they must be serializable. For this reason, the
value of any build property should be simple inert data: strings,
numbers, lists, tuples, and dictionaries. They should not contain
class instances.} object. For example:

@example
class MakeTarball(ShellCommand):
    def start(self):
        if self.getProperty("os") == "win":
            self.setCommand([ ... ]) # windows-only command
        else:
            self.setCommand([ ... ]) # equivalent for other systems
        ShellCommand.start(self)
@end example

@menu
* Property::
* WithProperties::
* Common Build Properties::
@end menu

@node Property
@subsubsection Property
@cindex Property

You can use build properties in most step paramaters.  Please file bugs for any
parameters which do not accept properties.  The simplest form is to wrap the
property name with @code{Property}, passing an optional default argument.

@example
from buildbot.steps.trigger import Trigger
form buildbot.process.properties import Property

f.addStep(Trigger(waitForFinish=False, schedulerNames=['build-dependents'], alwaysUseLatest=True,
			set_properties=@{'coq_revision': Property("got_revision")@}))
@end example

You can specify a default value by passing a @code{default} argument to
@code{Property}. This is normally used when the property doesn't exist,
or when the value is something Python regards as "false". The @code{defaultWhenFalse}
argument can be used to force buildbot to use the default argument only
if the parameter is not set.

@node WithProperties
@subsubsection WithProperties
@cindex WithProperties

You can use build properties in ShellCommands by using the
@code{WithProperties} wrapper when setting the arguments of
the ShellCommand. This interpolates the named build properties
into the generated shell command.  Most step parameters accept
@code{WithProperties}.

You can use python dictionary-style string interpolation by using
the @code{%(propname)s} syntax. In this form, the property name goes
in the parentheses:

@example
f.addStep(ShellCommand(
          command=["tar", "czf",
                   WithProperties("build-%(revision)s.tar.gz"),
                   "source"]))
@end example

If this BuildStep were used in a tree obtained from Subversion, it
would create a tarball with a name like @file{build-1234.tar.gz}.

Don't forget the extra ``s'' after the closing parenthesis! This is
the cause of many confusing errors.

The dictionary-style interpolation supports a number of more advanced
syntaxes, too.

@table @code

@item propname:-replacement
If @code{propname} exists, substitute its value; otherwise,
substitute @code{replacement}.  @code{replacement} may be empty
(@code{%(propname:-)s})

@item propname:~replacement
Like @code{propname:-replacement}, but only substitutes the value
of property @code{propname} if it is something Python regards as "true".
Python considers @code{None}, 0, empty lists, and the empty string to be 
false, so such values will be replaced by @code{replacement}.

@item propname:+replacement
If @code{propname} exists, substitute @code{replacement}; otherwise,
substitute an empty string.

@end table

Although these are similar to shell substitutions, no other
substitutions are currently supported, and @code{replacement} in the
above cannot contain more substitutions.

If you need to do more complex substitution, you can pass keyword
arguments to @code{WithProperties}. The value of each keyword argument
should be a function that takes one argument (the existing properties)
and returns a string value that will be used to replace that key:

@example
WithProperties('%(now)s', now=lambda _: time.clock())
@end example

@example
def determine_foo(props):
    if props.has_key('bar'):
        return props['bar']
    elif props.has_key('baz'):
        return props['baz']
    return 'qux'

WithProperties('%(foo)s', foo=determine_foo)
@end example

The @code{WithProperties} function also does @code{printf}-style string
interpolation, using strings obtained by calling
@code{build.getProperty(propname)}. Note that for every @code{%s} (or
@code{%d}, etc), you must have exactly one additional argument to
indicate which build property you want to insert.

@example
from buildbot.steps.shell import ShellCommand
from buildbot.process.properties import WithProperties

f.addStep(ShellCommand(
          command=["tar", "czf",
                   WithProperties("build-%s.tar.gz", "revision"),
                   "source"]))
@end example

Note: like python, you can either do positional-argument interpolation
@emph{or} keyword-argument interpolation, not both. Thus you cannot use
a string like @code{WithProperties("foo-%(revision)s-%s", "branch")}.

@node Common Build Properties
@subsubsection Common Build Properties

The following build properties are set when the build is started, and
are available to all steps.

@table @code
@item branch

This comes from the build's SourceStamp, and describes which branch is
being checked out. This will be @code{None} (which interpolates into
@code{WithProperties} as an empty string) if the build is on the
default branch, which is generally the trunk. Otherwise it will be a
string like ``branches/beta1.4''. The exact syntax depends upon the VC
system being used.

@item revision

This also comes from the SourceStamp, and is the revision of the source code
tree that was requested from the VC system. When a build is requested of a
specific revision (as is generally the case when the build is triggered by
Changes), this will contain the revision specification. This is always a
string, although the syntax depends upon the VC system in use: for SVN it is an
integer, for Mercurial it is a short string, for Darcs it is a rather large
string, etc.

If the ``force build'' button was pressed, the revision will be @code{None},
which means to use the most recent revision available.  This is a ``trunk
build''. This will be interpolated as an empty string.

@item got_revision

This is set when a Source step checks out the source tree, and
provides the revision that was actually obtained from the VC system.
In general this should be the same as @code{revision}, except for
trunk builds, where @code{got_revision} indicates what revision was
current when the checkout was performed. This can be used to rebuild
the same source code later.

Note that for some VC systems (Darcs in particular), the revision is a
large string containing newlines, and is not suitable for interpolation
into a filename.

@item buildername

This is a string that indicates which Builder the build was a part of.
The combination of buildername and buildnumber uniquely identify a
build.

@item buildnumber

Each build gets a number, scoped to the Builder (so the first build
performed on any given Builder will have a build number of 0). This
integer property contains the build's number.

@item slavename

This is a string which identifies which buildslave the build is
running on.

@item scheduler

If the build was started from a scheduler, then this property will
contain the name of that scheduler.

@item repository

The repository of the sourcestamp for this build

@item project

The project of the sourcestamp for this build

@item workdir

The absolute path of the base working directory on the slave, of the current
builder.

@end table

@node Source Checkout
@subsection Source Checkout

The first step of any build is typically to acquire the source code
from which the build will be performed. There are several classes to
handle this, one for each of the different source control system that
Buildbot knows about. For a description of how Buildbot treats source
control in general, see @ref{Version Control Systems}.

All source checkout steps accept some common parameters to control how
they get the sources and where they should be placed. The remaining
per-VC-system parameters are mostly to specify where exactly the
sources are coming from.

@table @code
@item mode

a string describing the kind of VC operation that is desired. Defaults
to @code{update}.

@table @code
@item update
specifies that the CVS checkout/update should be performed directly
into the workdir. Each build is performed in the same directory,
allowing for incremental builds. This minimizes disk space, bandwidth,
and CPU time. However, it may encounter problems if the build process
does not handle dependencies properly (sometimes you must do a ``clean
build'' to make sure everything gets compiled), or if source files are
deleted but generated files can influence test behavior (e.g. python's
.pyc files), or when source directories are deleted but generated
files prevent CVS from removing them. Builds ought to be correct
regardless of whether they are done ``from scratch'' or incrementally,
but it is useful to test both kinds: this mode exercises the
incremental-build style.

@item copy
specifies that the CVS workspace should be maintained in a separate
directory (called the 'copydir'), using checkout or update as
necessary. For each build, a new workdir is created with a copy of the
source tree (rm -rf workdir; cp -r copydir workdir). This doubles the
disk space required, but keeps the bandwidth low (update instead of a
full checkout). A full 'clean' build is performed each time. This
avoids any generated-file build problems, but is still occasionally
vulnerable to CVS problems such as a repository being manually
rearranged, causing CVS errors on update which are not an issue with a
full checkout.

@c TODO: something is screwy about this, revisit. Is it the source
@c directory or the working directory that is deleted each time?

@item clobber
specifies that the working directory should be deleted each time,
necessitating a full checkout for each build. This insures a clean
build off a complete checkout, avoiding any of the problems described
above. This mode exercises the ``from-scratch'' build style.

@item export
this is like @code{clobber}, except that the 'cvs export' command is
used to create the working directory. This command removes all CVS
metadata files (the CVS/ directories) from the tree, which is
sometimes useful for creating source tarballs (to avoid including the
metadata in the tar file).
@end table

@item workdir
like all Steps, this indicates the directory where the build will take
place. Source Steps are special in that they perform some operations
outside of the workdir (like creating the workdir itself).

@item alwaysUseLatest
if True, bypass the usual ``update to the last Change'' behavior, and
always update to the latest changes instead.

@item retry
If set, this specifies a tuple of @code{(delay, repeats)} which means
that when a full VC checkout fails, it should be retried up to
@var{repeats} times, waiting @var{delay} seconds between attempts. If
you don't provide this, it defaults to @code{None}, which means VC
operations should not be retried. This is provided to make life easier
for buildslaves which are stuck behind poor network connections.

@item repository
The name of this parameter might vary depending on the Source step you
are running. The concept explained here is common to all steps and
applies to @code{repourl} as well as for @code{baseURL} (when
applicable). Buildbot, now being aware of the repository name via the
ChangeSource step might in some cases not need the repository URL. There
are multiple way to pass it through to this step, corresponding to
the type of the parameter given to this step:

@table @code
@item None
In the case where no parameter is specified, the repository URL will
be taken directly from the Change property. This value should be used
if your ChangeSource step has all the information about how to reach
the Change.

@item string
The parameter might be a string. In this case, this string will be
used as the full repository URL. The value coming from the
ChangeSource step will be ignored.

@item format string
If the parameter is a string containing @code{%s}, then the repository
property from the Change will be substituted in place of the
@code{%s}. This is usefull when the ChangeSource step knows where the
repository resides locally, but doesn't know the scheme used to access
it. For instance, @code{ssh://server/%s} makes sense if the repository
property is the local path of the repository.

@item dict
In this case, the repository URL will be the value indexed by the
repository property in the dict given as parameter.

@item callable
The callable given as parameter will take the repository property from
the Change and its return value will be used as repository URL.

@end table

@item timeout
Specifies the timeout for slave-side operations, in seconds.  If your
repositories are particularly large, then you may need to increase this 
value from its default of 1200 (20 minutes).

@end table

Use of WithProperties with string, dict and callable is supported.

Note that this is quite similar to the mechanism used by the WebStatus
for the @code{changecommentlink}, @code{projects} or @code{repositories} parameter.

My habit as a developer is to do a @code{cvs update} and @code{make} each
morning. Problems can occur, either because of bad code being checked in, or
by incomplete dependencies causing a partial rebuild to fail where a
complete from-scratch build might succeed. A quick Builder which emulates
this incremental-build behavior would use the @code{mode='update'}
setting.

On the other hand, other kinds of dependency problems can cause a clean
build to fail where a partial build might succeed. This frequently results
from a link step that depends upon an object file that was removed from a
later version of the tree: in the partial tree, the object file is still
around (even though the Makefiles no longer know how to create it).

``official'' builds (traceable builds performed from a known set of
source revisions) are always done as clean builds, to make sure it is
not influenced by any uncontrolled factors (like leftover files from a
previous build). A ``full'' Builder which behaves this way would want
to use the @code{mode='clobber'} setting.

Each VC system has a corresponding source checkout class: their
arguments are described on the following pages.


@menu
* CVS::
* SVN::
* Darcs::
* Mercurial::
* Bzr::
* P4::
* Git::
* BitKeeper::
* Repo::
* Monotone::
@end menu

@node CVS
@subsubsection CVS
@cindex CVS Checkout
@bsindex buildbot.steps.source.CVS


The @code{CVS} build step performs a @uref{http://www.nongnu.org/cvs/,
CVS} checkout or update. It takes the following arguments:

@table @code
@item cvsroot
(required): specify the CVSROOT value, which points to a CVS
repository, probably on a remote machine. For example, the cvsroot
value you would use to get a copy of the Buildbot source code is
@code{:pserver:anonymous@@cvs.sourceforge.net:/cvsroot/buildbot}

@item cvsmodule
(required): specify the cvs @code{module}, which is generally a
subdirectory of the CVSROOT. The cvsmodule for the Buildbot source
code is @code{buildbot}.

@item branch
a string which will be used in a @code{-r} argument. This is most
useful for specifying a branch to work on. Defaults to @code{HEAD}.

@item global_options
a list of flags to be put before the verb in the CVS command.

@item checkout_options
@item export_options
@item extra_options
a list of flags to be put after the verb in the CVS command.
@code{checkout_options} is only used for checkout operations,
@code{export_options} is only used for export operations, and
@code{extra_options} is used for both.

@item checkoutDelay
if set, the number of seconds to put between the timestamp of the last
known Change and the value used for the @code{-D} option. Defaults to
half of the parent Build's treeStableTimer.

@end table


@node SVN
@subsubsection SVN

@cindex SVN Checkout
@bsindex buildbot.steps.source.SVN


The @code{SVN} build step performs a
@uref{http://subversion.tigris.org, Subversion} checkout or update.
There are two basic ways of setting up the checkout step, depending
upon whether you are using multiple branches or not.

The most versatile way to create the @code{SVN} step is with the
@code{svnurl} argument:

@table @code
@item svnurl
(required): this specifies the @code{URL} argument that will be given
to the @code{svn checkout} command. It dictates both where the
repository is located and which sub-tree should be extracted. In this
respect, it is like a combination of the CVS @code{cvsroot} and
@code{cvsmodule} arguments. For example, if you are using a remote
Subversion repository which is accessible through HTTP at a URL of
@code{http://svn.example.com/repos}, and you wanted to check out the
@code{trunk/calc} sub-tree, you would use
@code{svnurl="http://svn.example.com/repos/trunk/calc"} as an argument
to your @code{SVN} step.
@end table

The @code{svnurl} argument can be considered as a universal means to
create the @code{SVN} step as it ignores the branch information in the
SourceStamp.

Alternatively, if you are building from multiple branches, then you
should preferentially create the @code{SVN} step with the
@code{baseURL} and @code{defaultBranch} arguments instead:

@table @code
@item baseURL
(required): this specifies the base repository URL, to which a branch name will
be appended. Alternatively, @code{baseURL} can contain a @code{%%BRANCH%%}
placeholder, which will be replaced with the branch name. @code{baseURL} should
probably end in a slash.

@item defaultBranch
(optional): this specifies the name of the branch to use when a Build does not
provide one of its own. This is a string that will be appended to
@code{baseURL} to create the URL that will be passed to the @code{svn checkout}
command. If you use @code{baseURL} without specifying @code{defaultBranch}
every @code{ChangeStamp} must come with a valid (not None) @code{branch}.

It is possible to mix to have a mix of @code{SVN} steps that use
either the @code{svnurl} or @code{baseURL} arguments but not both at
the same time.

@item username
(optional): if specified, this will be passed to the @code{svn}
binary with a @code{--username} option.

@item password
(optional): if specified, this will be passed to the @code{svn}
binary with a @code{--password} option.  The password itself will be
suitably obfuscated in the logs.

@item extra_args
(optional): if specified, an array of strings that will be passed as
extra arguments to the @code{svn} binary.

@item keep_on_purge
(optional): specific files or directories to keep between purges,
like some build outputs that can be reused between builds.

@item ignore_ignores
(optional): when purging changes, don't use rules defined in
svn:ignore properties and global-ignores in subversion/config.

@item always_purge
(optional): if set to True, always purge local changes before updating. This deletes unversioned files and reverts everything that would appear in a @code{svn status}.

@item depth
(optional): Specify depth argument to achieve sparse checkout.  Only available if slave has Subversion 1.5 or higher.

If set to "empty" updates will not pull in any files or subdirectories not already present. If set to "files", updates will pull in any files not already present, but not directories.  If set to "immediates", updates willl pull in any files or subdirectories not already present, the new subdirectories will have depth: empty.  If set to "infinity", updates will pull in any files or subdirectories not already present; the new subdirectories will have depth-infinity. Infinity is equivalent to SVN default update behavior, without specifying any depth argument.

@end table

If you are using branches, you must also make sure your
@code{ChangeSource} will report the correct branch names.

@heading branch example

Let's suppose that the ``MyProject'' repository uses branches for the
trunk, for various users' individual development efforts, and for
several new features that will require some amount of work (involving
multiple developers) before they are ready to merge onto the trunk.
Such a repository might be organized as follows:

@example
svn://svn.example.org/MyProject/trunk
svn://svn.example.org/MyProject/branches/User1/foo
svn://svn.example.org/MyProject/branches/User1/bar
svn://svn.example.org/MyProject/branches/User2/baz
svn://svn.example.org/MyProject/features/newthing
svn://svn.example.org/MyProject/features/otherthing
@end example

Further assume that we want the Buildbot to run tests against the
trunk and against all the feature branches (i.e., do a
checkout/compile/build of branch X when a file has been changed on
branch X, when X is in the set [trunk, features/newthing,
features/otherthing]). We do not want the Buildbot to automatically
build any of the user branches, but it should be willing to build a
user branch when explicitly requested (most likely by the user who
owns that branch).

There are three things that need to be set up to accommodate this
system. The first is a ChangeSource that is capable of identifying the
branch which owns any given file. This depends upon a user-supplied
function, in an external program that runs in the SVN commit hook and
connects to the buildmaster's @code{PBChangeSource} over a TCP
connection. (you can use the ``@code{buildbot sendchange}'' utility
for this purpose, but you will still need an external program to
decide what value should be passed to the @code{--branch=} argument).
For example, a change to a file with the SVN URL of
``svn://svn.example.org/MyProject/features/newthing/src/foo.c'' should
be broken down into a Change instance with
@code{branch='features/newthing'} and @code{file='src/foo.c'}.

The second piece is an @code{AnyBranchScheduler} which will pay
attention to the desired branches. It will not pay attention to the
user branches, so it will not automatically start builds in response
to changes there. The AnyBranchScheduler class requires you to
explicitly list all the branches you want it to use, but it would not
be difficult to write a subclass which used
@code{branch.startswith('features/'} to remove the need for this
explicit list. Or, if you want to build user branches too, you can use
AnyBranchScheduler with @code{branches=None} to indicate that you want
it to pay attention to all branches.

The third piece is an @code{SVN} checkout step that is configured to
handle the branches correctly, with a @code{baseURL} value that
matches the way the ChangeSource splits each file's URL into base,
branch, and file.

@example
from buildbot.changes.pb import PBChangeSource
from buildbot.scheduler import AnyBranchScheduler
from buildbot.process import source, factory
from buildbot.steps import source, shell

c['change_source'] = PBChangeSource()
s1 = AnyBranchScheduler('main',
                        ['trunk', 'features/newthing', 'features/otherthing'],
                        10*60, ['test-i386', 'test-ppc'])
c['schedulers'] = [s1]

f = factory.BuildFactory()
f.addStep(source.SVN(mode='update',
                     baseURL='svn://svn.example.org/MyProject/',
                     defaultBranch='trunk'))
f.addStep(shell.Compile(command="make all"))
f.addStep(shell.Test(command="make test"))

c['builders'] = [
  @{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
                       'factory':f @},
  @{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
                      'factory':f @},
 ]
@end example

In this example, when a change arrives with a @code{branch} attribute of
``trunk'', the resulting build will have a SVN step that concatenates
``svn://svn.example.org/MyProject/'' (the baseURL) with ``trunk'' (the branch
name) to get the correct svn command. If the ``newthing'' branch has a change
to ``src/foo.c'', then the SVN step will concatenate
``svn://svn.example.org/MyProject/'' with ``features/newthing'' to get the
svnurl for checkout.

For added flexibility, @code{baseURL} may contain a @code{%%BRANCH%%}
placeholder, which will be replaced either by the branch in the SourceStamp or
the default specified in @code{defaultBranch}.

@example
source.SVN( mode='update',
            baseURL='svn://svn.example.org/svn/%%BRANCH%%/myproject',
            defaultBranch='trunk' )
@end example

@node Darcs
@subsubsection Darcs

@cindex Darcs Checkout
@bsindex buildbot.steps.source.Darcs


The @code{Darcs} build step performs a
@uref{http://darcs.net/, Darcs} checkout or update.

Like @xref{SVN}, this step can either be configured to always check
out a specific tree, or set up to pull from a particular branch that
gets specified separately for each build. Also like SVN, the
repository URL given to Darcs is created by concatenating a
@code{baseURL} with the branch name, and if no particular branch is
requested, it uses a @code{defaultBranch}. The only difference in
usage is that each potential Darcs repository URL must point to a
fully-fledged repository, whereas SVN URLs usually point to sub-trees
of the main Subversion repository. In other words, doing an SVN
checkout of @code{baseURL} is legal, but silly, since you'd probably
wind up with a copy of every single branch in the whole repository.
Doing a Darcs checkout of @code{baseURL} is just plain wrong, since
the parent directory of a collection of Darcs repositories is not
itself a valid repository.

The Darcs step takes the following arguments:

@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Darcs source repository is available.

@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{darcs get} command.
@end table

@node Mercurial
@subsubsection Mercurial

@cindex Mercurial Checkout
@bsindex buildbot.steps.source.Mercurial


The @code{Mercurial} build step performs a
@uref{http://selenic.com/mercurial, Mercurial} (aka ``hg'') checkout
or update.

Branches are available in two modes: ''dirname'' like @xref{Darcs}, or
''inrepo'', which uses the repository internal branches. Make sure this
setting matches your changehook, if you have that installed.

The Mercurial step takes the following arguments:

@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Mercurial source repository is available.

@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{hg clone} command.

@item branchType
either 'dirname' (default) or 'inrepo' depending on whether
the branch name should be appended to the @code{baseURL}
or the branch is a mercurial named branch and can be
found within the @code{repourl}.

@item clobberOnBranchChange
boolean, defaults to True. If set and
using inrepos branches, clobber the tree
at each branch change. Otherwise, just
update to the branch.

@end table


@node Bzr
@subsubsection Bzr

@cindex Bzr Checkout
@bsindex buildbot.steps.source.Bzr

@code{bzr} is a descendant of Arch/Baz, and is frequently referred to
as simply ``Bazaar''. The repository-vs-workspace model is similar to
Darcs, but it uses a strictly linear sequence of revisions (one
history per branch) like Arch. Branches are put in subdirectories.
This makes it look very much like Mercurial. It takes the following
arguments:

@table @code

@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Bzr source repository is available.

@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{bzr checkout} command.

@item forceSharedRepo
(boolean, optional, defaults to False): If set to True, the working directory
will be made into a bzr shared repository if it is not already. Shared
repository greatly reduces the amount of history data that needs to be
downloaded if not using update/copy mode, or if using update/copy mode with
multiple branches.
@end table



@node P4
@subsubsection P4

@cindex Perforce Update
@bsindex buildbot.steps.source.P4
@c TODO @bsindex buildbot.steps.source.P4Sync


The @code{P4} build step creates a @uref{http://www.perforce.com/,
Perforce} client specification and performs an update.

@table @code
@item p4base
A view into the Perforce depot without branch name or trailing "...".
Typically "//depot/proj/".
@item defaultBranch
A branch name to append on build requests if none is specified.
Typically "trunk".
@item p4port
(optional): the host:port string describing how to get to the P4 Depot
(repository), used as the -p argument for all p4 commands.
@item p4user
(optional): the Perforce user, used as the -u argument to all p4
commands.
@item p4passwd
(optional): the Perforce password, used as the -p argument to all p4
commands.
@item p4extra_views
(optional): a list of (depotpath, clientpath) tuples containing extra
views to be mapped into the client specification. Both will have
"/..." appended automatically. The client name and source directory
will be prepended to the client path.
@item p4client
(optional): The name of the client to use. In mode='copy' and
mode='update', it's particularly important that a unique name is used
for each checkout directory to avoid incorrect synchronization. For
this reason, Python percent substitution will be performed on this value
to replace %(slave)s with the slave name and %(builder)s with the
builder name. The default is "buildbot_%(slave)s_%(build)s".
@item p4line_end
(optional): The type of line ending handling P4 should use.  This is
added directly to the cliet spec's LineEnd property.  The default is
"local".
@end table


@node Git
@subsubsection Git

@cindex Git Checkout
@bsindex buildbot.steps.source.Git

The @code{Git} build step clones or updates a @uref{http://git.or.cz/,
Git} repository and checks out the specified branch or revision. Note
that the buildbot supports Git version 1.2.0 and later: earlier
versions (such as the one shipped in Ubuntu 'Dapper') do not support
the @command{git init} command that the buildbot uses.

The Git step takes the following arguments:

@table @code
@item repourl
(required): the URL of the upstream Git repository.

@item branch
(optional): this specifies the name of the branch to use when a Build
does not provide one of its own. If this this parameter is not
specified, and the Build does not provide a branch, the ``master''
branch will be used.

@item ignore_ignores
(optional): when purging changes, don't use .gitignore and
.git/info/exclude.

@item submodules
(optional): when initializing/updating a Git repository, this decides whether
or not buildbot should consider git submodules.  Default: False.

@item reference
(optional): use the specified string as a path to a reference repository on the
local machine. Git will try to grab objects from this path first instead of the
main repository, if they exist.

@item shallow
(optional): instructs git to attempt shallow clones (@code{--depth 1}).  If the
user/scheduler asks for a specific revision, this parameter is ignored.

@item progress
(optional): passes the (@code{--progress}) flag to (@code{git fetch}). This
solves issues of long fetches being killed due to lack of output, but requires
Git 1.7.2 or later.

@end table

This Source step integrates with @ref{GerritChangeSource}, and will automatically use
Gerrit's "virtual branch" (@code{refs/changes/*}) to download the additionnal changes
introduced by a pending changeset.

Gerrit integration can be also triggered using forced build with @code{gerrit_change}
property with value in format: @code{change_number/patchset_number}.


@node BitKeeper
@subsubsection BitKeeper

@cindex BitKeeper Checkout
@bsindex buildbot.steps.source.BK

The @code{BK} build step performs a @uref{http://www.bitkeeper.com/, BitKeeper}
checkout or update.

The BitKeeper step takes the following arguments:

@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
BitKeeper source repository is available.

@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

@end table

@node Repo
@subsubsection Repo
@cindex Repo init sync
@bsindex buildbot.steps.source.Repo

The @code{Repo} build step performs a @uref{http://lwn.net/Articles/304488/, Repo}
init and sync.

The Repo step takes the following arguments:

@table @code
@item manifest_url
(required): the URL at which the Repo's manifests source repository is available.

@item manifest_branch
(optional, defaults to @code{master}): the manifest repository branch on which
repo will take its manifest. Corresponds to the @code{-b} argument to the @code{repo init} command.
@item manifest_file
(optional, defaults to @code{default.xml}): the manifest filename.
Corresponds to the @code{-m} argument to the @code{repo init} command.
@item tarball
(optional, defaults to @code{None}): the repo tarball used for fast bootstrap.
If not present the tarball will be created automatically after first sync.
It is a copy of the @code{.repo} directory which contains all the git objects.
This feature helps to minimize network usage on very big projects.
@end table

This Source step integrates with @ref{GerritChangeSource}, and will automatically use the
@code{repo download} command of repo to download the additionnal changes introduced by a
pending changeset.

Gerrit integration can be also triggered using forced build with following properties:
@code{repo_d}, @code{repo_d[0-9]}, @code{repo_download}, @code{repo_download[0-9]}
with values in format: @code{project/change_number/patchset_number}.
All of these properties will be translated into a @code{repo download}.
This feature allows integrators to build with several pending interdependent changes,
which at the moment cannot be described properly in Gerrit, and can only be described
by humans.


@node Monotone
@subsubsection Monotone

@cindex Monotone Checkout
@bsindex buildbot.steps.source.Monotone


The @code{Monotone} build step performs a
@uref{http://www.monotone.ca, Monotone} (aka ``mtn'') checkout
or update.

The Monotone step takes the following arguments:

@table @code
@item repourl
the URL at which the Monotone source repository is available.

@item branch
this specifies the name of the branch to use when a Build does not
provide one of its own.

@item progress
this is a boolean that has a pull from the repository use
``--ticker=dot'' instead of the default ``--ticker=none''.

@end table


@node ShellCommand
@subsection ShellCommand

Most interesting steps involve executing a process of some sort on the
buildslave.  The @code{ShellCommand} class handles this activity.

Several subclasses of ShellCommand are provided as starting points for
common build steps.

@menu
* Using ShellCommands::
* Configure::
* Compile::
* Visual C++::
* Test::
* TreeSize::
* PerlModuleTest::
* Testing with mysql-test-run::
* SetProperty::
* SubunitShellCommand::
@end menu

@node Using ShellCommands
@subsubsection Using ShellCommands

@bsindex buildbot.steps.shell.ShellCommand

This is a useful base class for just about everything you might want
to do during a build (except for the initial source checkout). It runs
a single command in a child shell on the buildslave. All stdout/stderr
is recorded into a LogFile. The step finishes with a status of FAILURE
if the command's exit code is non-zero, otherwise it has a status of
SUCCESS.

The preferred way to specify the command is with a list of argv strings,
since this allows for spaces in filenames and avoids doing any fragile
shell-escaping. You can also specify the command with a single string, in
which case the string is given to '/bin/sh -c COMMAND' for parsing.

On Windows, commands are run via @code{cmd.exe /c} which works well. However,
if you're running a batch file, the error level does not get propagated
correctly unless you add 'call' before your batch file's name:
@code{cmd=['call', 'myfile.bat', ...]}.

@code{ShellCommand} arguments:

@table @code
@item command
a list of strings (preferred) or single string (discouraged) which
specifies the command to be run. A list of strings is preferred
because it can be used directly as an argv array. Using a single
string (with embedded spaces) requires the buildslave to pass the
string to /bin/sh for interpretation, which raises all sorts of
difficult questions about how to escape or interpret shell
metacharacters.

@item workdir
All ShellCommands are run by default in the ``workdir'', which
defaults to the ``@file{build}'' subdirectory of the slave builder's
base directory. The absolute path of the workdir will thus be the
slave's basedir (set as an option to @code{buildbot create-slave},
@pxref{Creating a buildslave}) plus the builder's basedir (set in the
builder's @code{c['builddir']} key in master.cfg) plus the workdir
itself (a class-level attribute of the BuildFactory, defaults to
``@file{build}'').

For example:

@example
f.addStep(ShellCommand(command=["make", "test"],
                       workdir="build/tests"))
@end example

@item env
a dictionary of environment strings which will be added to the child
command's environment. For example, to run tests with a different i18n
language setting, you might use

@example
f.addStep(ShellCommand(command=["make", "test"],
                       env=@{'LANG': 'fr_FR'@}))
@end example

These variable settings will override any existing ones in the
buildslave's environment or the environment specified in the
Builder. The exception is PYTHONPATH, which is merged
with (actually prepended to) any existing $PYTHONPATH setting. The
value is treated as a list of directories to prepend, and a single
string is treated like a one-item list. For example, to prepend both
@file{/usr/local/lib/python2.3} and @file{/home/buildbot/lib/python}
to any existing $PYTHONPATH setting, you would do something like the
following:

@example
f.addStep(ShellCommand(
              command=["make", "test"],
              env=@{'PYTHONPATH': ["/usr/local/lib/python2.3",
                                   "/home/buildbot/lib/python"] @}))
@end example

Those variables support expansion so that if you just want to prepend
@file{/home/buildbot/bin} to the PATH environment variable, you can do
it by putting the value @code{$@{PATH@}} at the end of the string like
in the example below. Variables that doesn't exists on the slave will be
replaced by @code{""}.


@example
f.addStep(ShellCommand(
              command=["make", "test"],
              env=@{'PATH': "/home/buildbot/bin:$@{PATH@}"@}))
@end example

@item want_stdout
if False, stdout from the child process is discarded rather than being
sent to the buildmaster for inclusion in the step's LogFile.

@item want_stderr
like @code{want_stdout} but for stderr. Note that commands run through
a PTY do not have separate stdout/stderr streams: both are merged into
stdout.

@item usePTY
Should this command be run in a pseudo-terminal?  The default is to observe the
configuration of the client (@pxref{Buildslave Options}), but specifying
@code{True} or @code{False} here will override the default.  This option is not
available on Windows.

In general, you do not want to use a pseudo-terminal.  This is @i{only} useful
for running commands that require a terminal - for example, testing a
command-line application that will only accept passwords read from a terminal.
Using a pseudo-terminal brings lots of compatibility problems, and prevents
Buildbot from distinguishing the standard error (red) and standard output
(black) streams.

In previous versions, the advantage of using a pseudo-terminal was that
``grandchild'' processes were more likely to be cleaned up if the build was
interrupted or times out.  This occurred because using a pseudo-terminal
incidentally puts the command into its own process group. 

As of Buildbot-0.8.4, all commands are placed in process groups, and thus
grandchild processes will be cleaned up properly.

@item logfiles
Sometimes commands will log interesting data to a local file, rather
than emitting everything to stdout or stderr. For example, Twisted's
``trial'' command (which runs unit tests) only presents summary
information to stdout, and puts the rest into a file named
@file{_trial_temp/test.log}. It is often useful to watch these files
as the command runs, rather than using @command{/bin/cat} to dump
their contents afterwards.

The @code{logfiles=} argument allows you to collect data from these
secondary logfiles in near-real-time, as the step is running. It
accepts a dictionary which maps from a local log name (which is how
the log data is presented in the build results) to either a remote filename
(interpreted relative to the build's working directory), or a dictionary
of options. Each named file will be polled on a regular basis (every couple
of seconds) as the build runs, and any new text will be sent over to the
buildmaster.

If you provide a dictionary of options instead of a string, you must specify
the @code{filename} key. You can optionally provide a @code{follow} key which
is a boolean controlling whether a logfile is followed or concatenated in its
entirety.  Following is appropriate for logfiles to which the build step will
append, where the pre-existing contents are not interesting.  The default value
for @code{follow} is @code{False}, which gives the same behavior as just
providing a string filename.

@example
f.addStep(ShellCommand(
              command=["make", "test"],
              logfiles=@{"triallog": "_trial_temp/test.log"@}))
@end example

The above example will add a log named 'triallog' on the master, based on
@code{_trial_temp/test.log} on the slave.

@example
f.addStep(ShellCommand(
              command=["make", "test"],
              logfiles=@{"triallog": @{"filename": "_trial_temp/test.log",
			       "follow": True,@}@}))
@end example


@item lazylogfiles
If set to @code{True}, logfiles will be tracked lazily, meaning that they will
only be added when and if something is written to them. This can be used to
suppress the display of empty or missing log files. The default is @code{False}.


@item timeout
if the command fails to produce any output for this many seconds, it
is assumed to be locked up and will be killed. This defaults to 1200 seconds.
Pass @code{None} to disable.

@item maxTime
if the command takes longer than this many seconds, it will be killed. This
is disabled by default.

@item description
This will be used to describe the command (on the Waterfall display)
while the command is still running. It should be a single
imperfect-tense verb, like ``compiling'' or ``testing''. The preferred
form is a list of short strings, which allows the HTML 
displays to create narrower columns by emitting a <br> tag between each
word. You may also provide a single string.

@item descriptionDone
This will be used to describe the command once it has finished. A
simple noun like ``compile'' or ``tests'' should be used. Like
@code{description}, this may either be a list of short strings or a
single string.

If neither @code{description} nor @code{descriptionDone} are set, the
actual command arguments will be used to construct the description.
This may be a bit too wide to fit comfortably on the Waterfall
display.

@example
f.addStep(ShellCommand(command=["make", "test"],
                       description=["testing"],
                       descriptionDone=["tests"]))
@end example

@item logEnviron
If this option is true (the default), then the step's logfile will describe the
environment variables on the slave.  In situations where the environment is not
relevant and is long, it may be easier to set @code{logEnviron=False}.

@end table

@node Configure
@subsubsection Configure

@bsindex buildbot.steps.shell.Configure

This is intended to handle the @code{./configure} step from
autoconf-style projects, or the @code{perl Makefile.PL} step from perl
MakeMaker.pm-style modules. The default command is @code{./configure}
but you can change this by providing a @code{command=} parameter.

@node Compile
@subsubsection Compile

@bsindex buildbot.steps.shell.Compile

This is meant to handle compiling or building a project written in C.
The default command is @code{make all}. When the compile is finished,
the log file is scanned for GCC warning messages, a summary log is
created with any problems that were seen, and the step is marked as
WARNINGS if any were discovered. Through the @code{WarningCountingShellCommand}
superclass, the number of warnings is stored in a Build Property named
``warnings-count'', which is accumulated over all Compile steps (so if two
warnings are found in one step, and three are found in another step, the
overall build will have a ``warnings-count'' property of 5).

The default regular expression used to detect a warning is
@code{'.*warning[: ].*'} , which is fairly liberal and may cause
false-positives. To use a different regexp, provide a
@code{warningPattern=} argument, or use a subclass which sets the
@code{warningPattern} attribute:

@example
f.addStep(Compile(command=["make", "test"],
                  warningPattern="^Warning: "))
@end example

The @code{warningPattern=} can also be a pre-compiled python regexp
object: this makes it possible to add flags like @code{re.I} (to use
case-insensitive matching).

Note that the compiled @code{warningPattern} will have its @code{match} method
called, which is subtly different from a @code{search}. Your regular
expression must match the from the beginning of the line. This means that to
look for the word "warning" in the middle of a line, you will need to
prepend @code{'.*'} to your regular expression.

The @code{suppressionFile=} argument can be specified as the (relative) path
of a file inside the workdir defining warnings to be suppressed from the
warning counting and log file. The file will be uploaded to the master from
the slave before compiling, and any warning matched by a line in the
suppression file will be ignored. This is useful to accept certain warnings
(eg. in some special module of the source tree or in cases where the compiler
is being particularly stupid), yet still be able to easily detect and fix the
introduction of new warnings.

The file must contain one line per pattern of warnings to ignore. Empty lines
and lines beginning with @code{#} are ignored. Other lines must consist of a
regexp matching the file name, followed by a colon (@code{:}), followed by a
regexp matching the text of the warning. Optionally this may be followed by
another colon and a line number range. For example:

@example
# Sample warning suppression file

mi_packrec.c : .*result of 32-bit shift implicitly converted to 64 bits.* : 560-600
DictTabInfo.cpp : .*invalid access to non-static.*
kernel_types.h : .*only defines private constructors and has no friends.* : 51
@end example

If no line number range is specified, the pattern matches the whole file; if
only one number is given it matches only on that line.

The default warningPattern regexp only matches the warning text, so line
numbers and file names are ignored. To enable line number and file name
matching, privide a different regexp and provide a function (callable) as the
argument of @code{warningExtractor=}. The function is called with three
arguments: the BuildStep object, the line in the log file with the warning,
and the @code{SRE_Match} object of the regexp search for @code{warningPattern}. It
should return a tuple @code{(filename, linenumber, warning_test)}. For
example:

@example
f.addStep(Compile(command=["make"],
                  warningPattern="^(.*?):([0-9]+): [Ww]arning: (.*)$",
                  warningExtractor=Compile.warnExtractFromRegexpGroups,
                  suppressionFile="support-files/compiler_warnings.supp"))
@end example

(@code{Compile.warnExtractFromRegexpGroups} is a pre-defined function that
returns the filename, linenumber, and text from groups (1,2,3) of the regexp
match).

In projects with source files in multiple directories, it is possible to get
full path names for file names matched in the suppression file, as long as the
build command outputs the names of directories as they are entered into and
left again. For this, specify regexps for the arguments
@code{directoryEnterPattern=} and @code{directoryLeavePattern=}. The
@code{directoryEnterPattern=} regexp should return the name of the directory
entered into in the first matched group. The defaults, which are suitable for
GNU Make, are these:

@example
directoryEnterPattern = "make.*: Entering directory [\"`'](.*)['`\"]"
directoryLeavePattern = "make.*: Leaving directory"
@end example

(TODO: this step needs to be extended to look for GCC error messages
as well, and collect them into a separate logfile, along with the
source code filenames involved).

@node Visual C++
@subsubsection Visual C++

@bsindex buildbot.steps.vstudio.VC6
@bsindex buildbot.steps.vstudio.VC7
@bsindex buildbot.steps.vstudio.VC8
@bsindex buildbot.steps.vstudio.VS2003
@bsindex buildbot.steps.vstudio.VS2005
@bsindex buildbot.steps.vstudio.VS2008
@bsindex buildbot.steps.vstudio.VCExpress9

This step is meant to handle compilation using Microsoft compilers. 
VC++ 6-9, VS2003, VS2005, VS2008, and VCExpress9 are supported. This step will take care
of setting up a clean compilation environment, parse the generated
output in real time and deliver as detailed as possible information
about the compilation executed.

All of the classes are in @code{buildbot.steps.vstudio}.  The available classes are:

@table @code

@item VC6

@item VC7

@item VC8

@item VC9

@item VS2003

@item VC2005

@item VC2008

@item VCExpress9

@end table

The available constructor arguments are

@table @code
@item mode
The mode default to @code{"rebuild"}, which means that first all the
remaining object files will be cleaned by the compiler. The alternate
value is @code{"build"}, where only the updated files will be recompiled.

@item projectfile
This is a mandatory argument which specifies the project file to be used
during the compilation.

@item config
This argument defaults to @code{"release"} an gives to the compiler the
configuration to use.

@item installdir
This is the place where the compiler is installed. The default value is
compiler specific and is the default place where the compiler is installed.

@item useenv
This boolean parameter, defaulting to @code{False} instruct the compiler
to use its own settings or the one defined through the environment
variables @code{%PATH%}, @code{%INCLUDE%}, and @code{%LIB%}. If any of
the @code{INCLUDE} or @code{LIB} parameter is defined, this parameter
automatically switches to @code{True}.

@item PATH
This is a list of path to be added to the PATH environment
variable. The default value is the one defined in the compiler options.

@item INCLUDE
This is a list of path where the compiler will first look for include
files. Then comes the default paths defined in the compiler options.

@item LIB
This is a list of path where the compiler will first look for
libraries. Then comes the default path defined in the compiler options.

@item arch
That one is only available with the class VS2005 (VC8). It gives the
target architecture of the built artifact. It defaults to @code{"x86''}.

@item project
This gives the specific project to build
from within a workspace. It defaults to building all projects. This is useful
for building cmake generate projects.
@end table

Here is an example on how to use this step:

@example
from buildbot.steps.VisualStudio import VS2005

f.addStep(VS2005(
        projectfile="project.sln", config="release",
        arch="x64", mode="build",
        INCLUDE=[r'D:\WINDDK\Include\wnet'],
        LIB=[r'D:\WINDDK\lib\wnet\amd64']))
@end example

@node Test
@subsubsection Test

@bsindex buildbot.steps.shell.Test

This is meant to handle unit tests. The default command is @code{make
test}, and the @code{warnOnFailure} flag is set.

@node TreeSize
@subsubsection TreeSize

@bsindex buildbot.steps.shell.TreeSize

This is a simple command that uses the 'du' tool to measure the size
of the code tree. It puts the size (as a count of 1024-byte blocks,
aka 'KiB' or 'kibibytes') on the step's status text, and sets a build
property named 'tree-size-KiB' with the same value.

@node PerlModuleTest
@subsubsection PerlModuleTest

@bsindex buildbot.steps.shell.PerlModuleTest

This is a simple command that knows how to run tests of perl modules.
It parses the output to determine the number of tests passed and
failed and total number executed, saving the results for later query.

@node Testing with mysql-test-run
@subsubsection Testing with mysql-test-run

The @code{process.mtrlogobserver.MTR} class is a subclass of @code{Test}
(@ref{Test}). It is used to run test suites using the mysql-test-run program,
as used in MySQL, Drizzle, MariaDB, and MySQL storage engine plugins.

The shell command to run the test suite is specified in the same way as for
the Test class. The MTR class will parse the output of running the test suite,
and use the count of tests executed so far to provide more accurate completion
time estimates. Any test failures that occur during the test are summarized on
the Waterfall Display.

Server error logs are added as additional log files, useful to debug test
failures.

Optionally, data about the test run and any test failures can be inserted into
a database for further analysis and report generation. To use this facility,
create an instance of @code{twisted.enterprise.adbapi.ConnectionPool} with
connections to the database. The necessary tables can be created automatically
by setting @code{autoCreateTables} to @code{True}, or manually using the SQL
found in the @file{mtrlogobserver.py} source file.

One problem with specifying a database is that each reload of the
configuration will get a new instance of @code{ConnectionPool} (even if the
connection parameters are the same). To avoid that Buildbot thinks the builder
configuration has changed because of this, use the
@code{process.mtrlogobserver.EqConnectionPool} subclass of
@code{ConnectionPool}, which implements an equiality operation that avoids
this problem.

Example use:

@example
from buildbot.process.mtrlogobserver import MTR, EqConnectionPool
myPool = EqConnectionPool("MySQLdb", "host", "buildbot", "password", "db")
myFactory.addStep(MTR(workdir="mysql-test", dbpool=myPool,
                      command=["perl", "mysql-test-run.pl", "--force"]))
@end example

@code{MTR} arguments:

@table @code

@item textLimit
Maximum number of test failures to show on the waterfall page (to not flood
the page in case of a large number of test failures). Defaults to 5.

@item testNameLimit
Maximum length of test names to show unabbreviated in the waterfall page, to
avoid excessive column width. Defaults to 16.

@item parallel
Value of @code{--parallel} option used for mysql-test-run.pl (number of processes
used to run the test suite in parallel). Defaults to 4. This is used to
determine the number of server error log files to download from the
slave. Specifying a too high value does not hurt (as nonexisting error logs
will be ignored), however if using @code{--parallel} value greater than the default
it needs to be specified, or some server error logs will be missing.

@item dbpool
An instance of twisted.enterprise.adbapi.ConnectionPool, or None.  Defaults to
None. If specified, results are inserted into the database using the
ConnectionPool.

@item autoCreateTables
Boolean, defaults to False. If True (and @code{dbpool} is specified), the
necessary database tables will be created automatically if they do not exist
already. Alternatively, the tables can be created manually from the SQL
statements found in the mtrlogobserver.py source file.

@item test_type
Short string that will be inserted into the database in the row for the test
run. Defaults to the empty string, but can be specified to identify different
types of test runs.

@item test_info
Descriptive string that will be inserted into the database in the row for the test
run. Defaults to the empty string, but can be specified as a user-readable
description of this particular test run.

@item mtr_subdir
The subdirectory in which to look for server error log files. Defaults to
``mysql-test'', which is usually correct. WithProperties is supported.

@end table


@node SetProperty
@subsubsection SetProperty

@bsindex buildbot.steps.shell.SetProperty

This buildstep is similar to ShellCommand, except that it captures the
output of the command into a property.  It is usually used like this:

@example
from buildbot.steps import shell
f.addStep(shell.SetProperty(command="uname -a", property="uname"))
@end example

This runs @code{uname -a} and captures its output (including stderr), stripped
of leading and trailing whitespace, in the property "uname".  To avoid
stripping, add @code{strip=False}. 

The @code{property} argument can be specified as a @code{WithProperties}
object, allowing the property name to be built from other property values.

The more advanced usage allows you to specify a function to extract properties
from the command output.  Here you can use regular expressions, string
interpolation, or whatever you would like.  In this form, @code{extract_fn}
should be passed, and not @code{property}.  The @code{extract_fn} function is
called with three arguments: the exit status of the command, its standard
output as a string, and its standard error as a string.  It should return a
dictionary containing all new properties.

@example
def glob2list(rc, stdout, stderr):
    jpgs = [ l.strip() for l in stdout.split('\n') ]
    return @{ 'jpgs' : jpgs @}
f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list))
@end example

Note that any ordering relationship of the contents of stdout and
stderr is lost.  For example, given

@example
f.addStep(SetProperty(
    command="echo output1; echo error >&2; echo output2",
    extract_fn=my_extract))
@end example

Then @code{my_extract} will see @code{stdout="output1\noutput2\n"}
and @code{stderr="error\n"}.

See also @ref{Setting Properties}.

@node SubunitShellCommand
@subsubsection SubunitShellCommand

@bsindex buildbot.steps.subunit.SubunitShellCommand

This buildstep is similar to ShellCommand, except that it runs the log content
through a subunit filter to extract test and failure counts.

@example
from buildbot.steps.subunit import SubunitShellCommand
f.addStep(SubunitShellCommand(command="make test", failureOnNoTests=False))
@end example

This runs @code{make test} and filters it through subunit. The 'tests' and
'test failed' progress metrics will now accumulate test data from the test run.

If @code{failureOnNoTests} is True, this step will fail if no test is run. By
default @code{failureOnNoTests} is False.

@node Slave Filesystem Steps
@subsection Slave Filesystem Steps

Here are some buildsteps for manipulating the slaves filesystem.

@menu
* RemoveDirectory::
@end menu

@node RemoveDirectory
@subsubsection RemoveDirectory

This command recursively deletes a directory on the slave.

@example
from buildbot.steps.slave import RemoveDirectory
f.addStep(RemoveDirectory(dir="build/build"))
@end example

@node Python BuildSteps
@subsection Python BuildSteps

Here are some BuildSteps that are specifcally useful for projects
implemented in Python.

@menu
* BuildEPYDoc::
* PyFlakes::
* PyLint::
* Trial::
* RemovePYCs::
@end menu

@node BuildEPYDoc
@subsubsection BuildEPYDoc

@bsindex buildbot.steps.python.BuildEPYDoc

@url{http://epydoc.sourceforge.net/, epydoc} is a tool for generating
API documentation for Python modules from their docstrings. It reads
all the .py files from your source tree, processes the docstrings
therein, and creates a large tree of .html files (or a single .pdf
file).

The @code{buildbot.steps.python.BuildEPYDoc} step will run
@command{epydoc} to produce this API documentation, and will count the
errors and warnings from its output.

You must supply the command line to be used. The default is
@command{make epydocs}, which assumes that your project has a Makefile
with an ``epydocs'' target. You might wish to use something like
@command{epydoc -o apiref source/PKGNAME} instead. You might also want
to add @command{--pdf} to generate a PDF file instead of a large tree
of HTML files.

The API docs are generated in-place in the build tree (under the
workdir, in the subdirectory controlled by the ``-o'' argument). To
make them useful, you will probably have to copy them to somewhere
they can be read. A command like @command{rsync -ad apiref/
dev.example.com:~public_html/current-apiref/} might be useful. You
might instead want to bundle them into a tarball and publish it in the
same place where the generated install tarball is placed.

@example
from buildbot.steps.python import BuildEPYDoc

...
f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"]))
@end example

@node PyFlakes
@subsubsection PyFlakes

@bsindex buildbot.steps.python.PyFlakes

@url{http://divmod.org/trac/wiki/DivmodPyflakes, PyFlakes} is a tool
to perform basic static analysis of Python code to look for simple
errors, like missing imports and references of undefined names. It is
like a fast and simple form of the C ``lint'' program. Other tools
(like pychecker) provide more detailed results but take longer to run.

The @code{buildbot.steps.python.PyFlakes} step will run pyflakes and
count the various kinds of errors and warnings it detects.

You must supply the command line to be used. The default is
@command{make pyflakes}, which assumes you have a top-level Makefile
with a ``pyflakes'' target. You might want to use something like
@command{pyflakes .} or @command{pyflakes src}.

@example
from buildbot.steps.python import PyFlakes

...
f.addStep(PyFlakes(command=["pyflakes", "src"]))
@end example

@node PyLint
@subsubsection PyLint

@bsindex buildbot.steps.python.PyLint

Similarly, the @code{buildbot.steps.python.PyLint} step will run pylint and
analyze the results.

You must supply the command line to be used. There is no default.

@example
from buildbot.steps.python import PyLint

...
f.addStep(PyLint(command=["pylint", "src"]))
@end example

@node Trial
@subsubsection Trial

@bsindex buildbot.steps.python_twisted.Trial

This step runs a unit test suite using @code{trial}, a unittest-like testing
framework that is a component of Twisted Python. Trial is used to implement
Twisted's own unit tests, and is the unittest-framework of choice for many
projects that use Twisted internally.

Projects that use trial typically have all their test cases in a 'test'
subdirectory of their top-level library directory. For example, for a package
@code{petmail}, the tests might be in @code{petmail/test/test_*.py}. More
complicated packages (like Twisted itself) may have multiple test directories,
like @code{twisted/test/test_*.py} for the core functionality and
@code{twisted/mail/test/test_*.py} for the email-specific tests.

To run trial tests manually, you run the @code{trial} executable and tell it
where the test cases are located. The most common way of doing this is with a
module name. For petmail, this might look like @code{trial petmail.test}, which
would locate all the @code{test_*.py} files under @code{petmail/test/}, running
every test case it could find in them.  Unlike the @code{unittest.py} that
comes with Python, it is not necessary to run the @code{test_foo.py} as a
script; you always let trial do the importing and running. The step's
@code{tests} parameter controls which tests trial will run: it can be a string
or a list of strings.

To find the test cases, the Python search path must allow something like
@code{import petmail.test} to work. For packages that don't use a separate
top-level @code{lib} directory, @code{PYTHONPATH=.} will work, and will use the
test cases (and the code they are testing) in-place.
@code{PYTHONPATH=build/lib} or @code{PYTHONPATH=build/lib.somearch} are also
useful when you do a @code{python setup.py build} step first. The
@code{testpath} attribute of this class controls what @code{PYTHONPATH} is set
to before running @code{trial}.

Trial has the ability, through the @code{--testmodule} flag, to run only the
set of test cases named by special @code{test-case-name} tags in source files.
We can get the list of changed source files from our parent Build and provide
them to trial, thus running the minimal set of test cases needed to cover the
Changes.  This is useful for quick builds, especially in trees with a lot of
test cases.  The @code{testChanges} parameter controls this feature: if set, it
will override @code{tests}.

The trial executable itself is typically just @code{trial}, and is typically
found in the shell search path.  It can be overridden with the @code{trial}
parameter.  This is useful for Twisted's own unittests, which want to use the
copy of bin/trial that comes with the sources.

To influence the version of python being used for the tests, or to add flags to
the command, set the @code{python} parameter. This can be a string (like
@code{python2.2}) or a list (like @code{['python2.3', '-Wall']}).

Trial creates and switches into a directory named @code{_trial_temp/} before
running the tests, and sends the twisted log (which includes all exceptions) to
a file named @code{test.log}. This file will be pulled up to the master where
it can be seen as part of the status output.

@example
from buildbot.steps.python_twisted import Trial
f.addStep(Trial(tests='petmail.test'))
@end example

@node RemovePYCs
@subsubsection RemovePYCs

@bsindex buildbot.steps.python_twisted.RemovePYCs

This is a simple built-in step that will remove @code{.pyc} files from the
workdir.  This is useful in builds that update their source (and thus do not
automatically delete @code{.pyc} files) but where some part of the build
process is dynamically searching for Python modules.  Notably, trial has a bad
habit of finding old test modules.

@example
from buildbot.steps.python_twisted import RemovePYCs
f.addStep(RemovePYCs())
@end example

@node Transferring Files
@subsection Transferring Files

@cindex File Transfer
@bsindex buildbot.steps.transfer.FileUpload
@bsindex buildbot.steps.transfer.FileDownload
@bsindex buildbot.steps.transfer.DirectoryUpload

Most of the work involved in a build will take place on the
buildslave. But occasionally it is useful to do some work on the
buildmaster side. The most basic way to involve the buildmaster is
simply to move a file from the slave to the master, or vice versa.
There are a pair of BuildSteps named @code{FileUpload} and
@code{FileDownload} to provide this functionality. @code{FileUpload}
moves a file @emph{up to} the master, while @code{FileDownload} moves
a file @emph{down from} the master.

As an example, let's assume that there is a step which produces an
HTML file within the source tree that contains some sort of generated
project documentation. We want to move this file to the buildmaster,
into a @file{~/public_html} directory, so it can be visible to
developers. This file will wind up in the slave-side working directory
under the name @file{docs/reference.html}. We want to put it into the
master-side @file{~/public_html/ref.html}.

@example
from buildbot.steps.shell import ShellCommand
from buildbot.steps.transfer import FileUpload

f.addStep(ShellCommand(command=["make", "docs"]))
f.addStep(FileUpload(slavesrc="docs/reference.html",
                     masterdest="~/public_html/ref.html"))
@end example

The @code{masterdest=} argument will be passed to os.path.expanduser,
so things like ``~'' will be expanded properly. Non-absolute paths
will be interpreted relative to the buildmaster's base directory.
Likewise, the @code{slavesrc=} argument will be expanded and
interpreted relative to the builder's working directory.


To move a file from the master to the slave, use the
@code{FileDownload} command. For example, let's assume that some step
requires a configuration file that, for whatever reason, could not be
recorded in the source code repository or generated on the buildslave
side:

@example
from buildbot.steps.shell import ShellCommand
from buildbot.steps.transfer import FileDownload

f.addStep(FileDownload(mastersrc="~/todays_build_config.txt",
                       slavedest="build_config.txt"))
f.addStep(ShellCommand(command=["make", "config"]))
@end example

Like @code{FileUpload}, the @code{mastersrc=} argument is interpreted
relative to the buildmaster's base directory, and the
@code{slavedest=} argument is relative to the builder's working
directory. If the buildslave is running in @file{~buildslave}, and the
builder's ``builddir'' is something like @file{tests-i386}, then the
workdir is going to be @file{~buildslave/tests-i386/build}, and a
@code{slavedest=} of @file{foo/bar.html} will get put in
@file{~buildslave/tests-i386/build/foo/bar.html}. Both of these commands
will create any missing intervening directories.

@subheading Other Parameters

The @code{maxsize=} argument lets you set a maximum size for the file
to be transferred. This may help to avoid surprises: transferring a
100MB coredump when you were expecting to move a 10kB status file
might take an awfully long time. The @code{blocksize=} argument
controls how the file is sent over the network: larger blocksizes are
slightly more efficient but also consume more memory on each end, and
there is a hard-coded limit of about 640kB.

The @code{mode=} argument allows you to control the access permissions
of the target file, traditionally expressed as an octal integer. The
most common value is probably 0755, which sets the ``x'' executable
bit on the file (useful for shell scripts and the like). The default
value for @code{mode=} is None, which means the permission bits will
default to whatever the umask of the writing process is. The default
umask tends to be fairly restrictive, but at least on the buildslave
you can make it less restrictive with a --umask command-line option at
creation time (@pxref{Buildslave Options}).

@subheading Transfering Directories

To transfer complete directories from the buildslave to the master, there
is a BuildStep named @code{DirectoryUpload}. It works like @code{FileUpload},
just for directories. As an example, let's assume a step has
generated project documentation, which consists of many files (like the output
of doxygen or epydoc). We want to move the entire documentation to the
buildmaster, into a @code{~/public_html/docs} directory. On the slave-side
the directory can be found under @code{docs}:

@example
from buildbot.steps.shell import ShellCommand
from buildbot.steps.transfer import DirectoryUpload

f.addStep(ShellCommand(command=["make", "docs"]))
f.addStep(DirectoryUpload(slavesrc="docs",
				masterdest="~/public_html/docs"))
@end example

The DirectoryUpload step will create all necessary directories and
transfers empty directories, too.

The @code{maxsize} and @code{blocksize} parameters are the same as for
@code{FileUpload}, although note that the size of the transferred data is
implementation-dependent, and probably much larger than you expect due to the
encoding used (currently tar).

The optional @code{compress} argument can be given as @code{'gz'} or
@code{'bz2'} to compress the datastream.

@node Transferring Strings
@subsection Transferring Strings

@cindex String Transfer
@bsindex buildbot.steps.transfer.StringDownload
@bsindex buildbot.steps.transfer.JSONStringDownload
@bsindex buildbot.steps.transfer.JSONPropertiesDownload

Sometimes it is useful to transfer a calculated value from the master
to the slave.  Instead of having to create a temporary file and then
use @code{FileDownload}, you can use one of the string download steps.

@code{StringDownload} works just like @code{FileDownload} except it takes a
single argument, @code{s}, representing the string to download instead of a
@code{mastersrc} argument.  @code{WithProperties} will be properly rendered, if
used in this argument.

@code{JSONStringDownload} is similar, except it takes an @code{o}
argument, which must be json serializable, and transfers that as a
json-encoded string to the slave.

@code{JSONPropertiesDownload} transfers a json-encoded string that
represents a dictionary where @code{properties} maps to a dictionary
of build property name to property value; and @code{sourcestamp}
represents the build's sourcestamp.

@node Running Commands on the Master
@subsection Running Commands on the Master

@bsindex buildbot.steps.master.MasterShellCommand

Occasionally, it is useful to execute some task on the master, for example to
create a directory, deploy a build result, or trigger some other centralized
processing.  This is possible, in a limited fashion, with the
@code{MasterShellCommand} step.

This step operates similarly to a regular @code{ShellCommand}, but executes on
the master, instead of the slave.  To be clear, the enclosing @code{Build}
object must still have a slave object, just as for any other step -- only, in
this step, the slave does not do anything.

In this example, the step renames a tarball based on the day of the week.

@example
from buildbot.steps.transfer import FileUpload
from buildbot.steps.master import MasterShellCommand

f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz",
                     masterdest="/var/buildoutputs/widgetsoft-new.tar.gz"))
f.addStep(MasterShellCommand(command="""
    cd /var/buildoutputs;
    mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz"""))
@end example

Note that, by default, this step passes a copy of the buildmaster's environment
variables to the subprocess.  To pass an explicit environment instead, add an
@code{env=@{..@}} argument.

@node Setting Properties
@subsection Setting Properties

@bsindex buildbot.steps.slave.SetPropertiesFromEnv

In Buildbot-0.8.3 and higher, slaves provide their environment variables to the
master on connect.  These can be copied into Buildbot properties with the
@code{SetPropertiesFromEnv} step.  Pass a variable or list of variables in the
@code{variables} parameter, then simply use the values as properties in a later
step.

Note that on Windows, environment variables are case-insensitive, but Buildbot
property names are case sensitive.  The property will have exactly the variable
name you specify, even if the underlying environment variable is capitalized
differently.  If, for example, you use @code{variables=['Tmp']}, the result
will be a property named @code{Tmp}, even though the environment variable is
displayed as @code{TMP} in the Windows GUI.

@example
from buildbot.steps.slave import SetPropertiesFromEnv
from buildbot.steps.shell import Compile

f.addStep(SetPropertiesFromEnv(variables=["SOME_JAVA_LIB_HOME", "JAVAC"]))
f.addStep(Compile(commands=[WithProperties("%s","JAVAC"), "-cp", WithProperties("%s", "SOME_JAVA_LIB_HOME")))
@end example

Note that this step requires that the Buildslave be at least version 0.8.3.
For previous versions, no environment variables are available (the slave
environment will appear to be empty).

See also @ref{SetProperty}, which runs a command on the slave and sets a
property based on the result.

@node Triggering Schedulers
@subsection Triggering Schedulers
@bsindex buildbot.steps.trigger.Trigger

The counterpart to the Triggerable described in section
@pxref{Triggerable Scheduler} is the Trigger BuildStep.

@example
from buildbot.steps.trigger import Trigger
f.addStep(Trigger(schedulerNames=['build-prep'],
                  waitForFinish=True,
                  updateSourceStamp=True,
                  set_properties=@{ 'quick' : False @},
                  copy_properties=[ 'release_code_name' ]))
@end example

The @code{schedulerNames=} argument lists the Triggerables
that should be triggered when this step is executed.  Note that
it is possible, but not advisable, to create a cycle where a build
continually triggers itself, because the schedulers are specified
by name.

If @code{waitForFinish} is True, then the step will not finish until
all of the builds from the triggered schedulers have finished. If this
argument is False (the default) or not given, then the buildstep
succeeds immediately after triggering the schedulers.

The SourceStamp to use for the triggered build is controlled by the arguments
@code{updateSourceStamp}, @code{alwaysUseLatest}, and @code{sourceStamp}.
If @code{updateSourceStamp} is True (the default), then step updates
the SourceStamp given to the Triggerables to include
@code{got_revision} (the revision actually used in this build) as
@code{revision} (the revision to use in the triggered builds). This is
useful to ensure that all of the builds use exactly the same
SourceStamp, even if other Changes have occurred while the build was
running. If @code{updateSourceStamp} is False (and neither of the other
arguments are specified), then the exact same SourceStamp is used. If
@code{alwaysUseLatest} is True, then no SourceStamp is given, corresponding to
using the latest revision of the repository specified in the Source step. This
is useful if the triggered builds use to a different source repository.
@code{sourceStamp} accepts a dictionary containing the keys @code{branch},
@code{revision}, @code{branch}, @code{repository}, @code{project}, and
optionally @code{patch_level}, @code{patch_level} and @code{patch_subdir} and
creates the corresponding SourceStamp.

Two parameters allow control of the properties that are passed to the triggered
scheduler.  To simply copy properties verbatim, list them in the
@code{copy_properties} parameter.  To set properties explicitly, use the more
sophisticated @code{set_properties}, which takes a dictionary mapping property
names to values.  You may use @code{WithProperties} here to dynamically
construct new property values.

@node Miscellaneous BuildSteps
@subsection Miscellaneous BuildSteps

A number of steps do not fall into any particular category.

@menu
* HLint::
@end menu

@node HLint
@subsubsection HLint

@bsindex buildbot.steps.python_twisted.HLint

The HLint step runs Twisted Lore, a lint-like checker over a set of
@code{.xhtml} files.  Any deviations from recommended style is flagged and put
in the output log.  

The step looks at the list of changes in the build to determine which files to
check - it does not check all files.  It specifically excludes any @code{.xhtml}
files in the top-level @code{sandbox/} directory.

The step takes a single, optional, parameter: @code{python}.  This specifies the
Python executable to use to run Lore.

@example
from buildbot.steps.python_twisted import HLint
f.addStep(HLint())
@end example

@node Writing New BuildSteps
@subsection Writing New BuildSteps

While it is a good idea to keep your build process self-contained in
the source code tree, sometimes it is convenient to put more
intelligence into your Buildbot configuration. One way to do this is
to write a custom BuildStep. Once written, this Step can be used in
the @file{master.cfg} file.

The best reason for writing a custom BuildStep is to better parse the
results of the command being run. For example, a BuildStep that knows
about JUnit could look at the logfiles to determine which tests had
been run, how many passed and how many failed, and then report more
detailed information than a simple @code{rc==0} -based ``good/bad''
decision.

@menu
* Writing BuildStep Constructors::
* BuildStep LogFiles::
* Reading Logfiles::
* Adding LogObservers::
* BuildStep URLs::
@end menu

@node Writing BuildStep Constructors
@subsubsection Writing BuildStep Constructors

BuildStep classes have some extra equipment, because they are their own
factories.  Consider the use of a BuildStep in @file{master.cfg}:

@example
f.addStep(MyStep(someopt="stuff", anotheropt=1))
@end example

This creates a single instance of class @code{MyStep}.  However, Buildbot needs
a new object each time the step is executed.  this is accomplished by storing
the information required to instantiate a new object in the @code{factory}
attribute.  When the time comes to construct a new Build, BuildFactory consults
this attribute (via @code{getStepFactory}) and instantiates a new step object.

When writing a new step class, then, keep in mind are that you cannot do
anything "interesting" in the constructor -- limit yourself to checking and
storing arguments.  To ensure that these arguments are provided to any new
objects, call @code{self.addFactoryArguments} with any keyword arguments your
constructor needs.

Keep a @code{**kwargs} argument on the end of your options, and pass that up to
the parent class's constructor.

The whole thing looks like this:

@example
class Frobinfy(LoggingBuildStep):
    def __init__(self,
            frob_what="frobee",
            frob_how_many=None,
            frob_how=None,
            **kwargs):

        # check
        if frob_how_many is None:
            raise TypeError("Frobinfy argument how_many is required")

        # call parent
        LoggingBuildStep.__init__(self, **kwargs)

        # set Frobnify attributes
        self.frob_what = frob_what
        self.frob_how_many = how_many
        self.frob_how = frob_how

        # and record arguments for later
        self.addFactoryArguments(
            frob_what=frob_what,
            frob_how_many=frob_how_many,
            frob_how=frob_how)

class FastFrobnify(Frobnify):
    def __init__(self,
            speed=5,
            **kwargs)
        Frobnify.__init__(self, **kwargs)
        self.speed = speed
        self.addFactoryArguments(
            speed=speed)
@end example

@node BuildStep LogFiles
@subsubsection BuildStep LogFiles

Each BuildStep has a collection of ``logfiles''. Each one has a short
name, like ``stdio'' or ``warnings''. Each LogFile contains an
arbitrary amount of text, usually the contents of some output file
generated during a build or test step, or a record of everything that
was printed to stdout/stderr during the execution of some command.

These LogFiles are stored to disk, so they can be retrieved later.

Each can contain multiple ``channels'', generally limited to three
basic ones: stdout, stderr, and ``headers''. For example, when a
ShellCommand runs, it writes a few lines to the ``headers'' channel to
indicate the exact argv strings being run, which directory the command
is being executed in, and the contents of the current environment
variables. Then, as the command runs, it adds a lot of ``stdout'' and
``stderr'' messages. When the command finishes, a final ``header''
line is added with the exit code of the process.

Status display plugins can format these different channels in
different ways. For example, the web page shows LogFiles as text/html,
with header lines in blue text, stdout in black, and stderr in red. A
different URL is available which provides a text/plain format, in
which stdout and stderr are collapsed together, and header lines are
stripped completely. This latter option makes it easy to save the
results to a file and run @command{grep} or whatever against the
output.

Each BuildStep contains a mapping (implemented in a python dictionary)
from LogFile name to the actual LogFile objects. Status plugins can
get a list of LogFiles to display, for example, a list of HREF links
that, when clicked, provide the full contents of the LogFile.

@heading Using LogFiles in custom BuildSteps

The most common way for a custom BuildStep to use a LogFile is to
summarize the results of a ShellCommand (after the command has
finished running). For example, a compile step with thousands of lines
of output might want to create a summary of just the warning messages.
If you were doing this from a shell, you would use something like:

@example
grep "warning:" output.log >warnings.log
@end example

In a custom BuildStep, you could instead create a ``warnings'' LogFile
that contained the same text. To do this, you would add code to your
@code{createSummary} method that pulls lines from the main output log
and creates a new LogFile with the results:

@example
    import StringIO
    def createSummary(self, log):
        warnings = []
        sio = StringIO.StringIO(log.getText())
        for line in sio.readlines():
            if "warning:" in line:
                warnings.append()
        self.addCompleteLog('warnings', "".join(warnings))
@end example

This example uses the @code{addCompleteLog} method, which creates a
new LogFile, puts some text in it, and then ``closes'' it, meaning
that no further contents will be added. This LogFile will appear in
the HTML display under an HREF with the name ``warnings'', since that
is the name of the LogFile.

You can also use @code{addHTMLLog} to create a complete (closed)
LogFile that contains HTML instead of plain text. The normal LogFile
will be HTML-escaped if presented through a web page, but the HTML
LogFile will not. At the moment this is only used to present a pretty
HTML representation of an otherwise ugly exception traceback when
something goes badly wrong during the BuildStep.

In contrast, you might want to create a new LogFile at the beginning
of the step, and add text to it as the command runs. You can create
the LogFile and attach it to the build by calling @code{addLog}, which
returns the LogFile object. You then add text to this LogFile by
calling methods like @code{addStdout} and @code{addHeader}. When you
are done, you must call the @code{finish} method so the LogFile can be
closed. It may be useful to create and populate a LogFile like this
from a LogObserver method @xref{Adding LogObservers}.

The @code{logfiles=} argument to @code{ShellCommand} (see
@pxref{ShellCommand}) creates new LogFiles and fills them in realtime
by asking the buildslave to watch a actual file on disk. The
buildslave will look for additions in the target file and report them
back to the BuildStep. These additions will be added to the LogFile by
calling @code{addStdout}. These secondary LogFiles can be used as the
source of a LogObserver just like the normal ``stdio'' LogFile.

@node Reading Logfiles
@subsubsection Reading Logfiles

Once a LogFile has been added to a BuildStep with @code{addLog()},
@code{addCompleteLog()}, @code{addHTMLLog()}, or @code{logfiles=},
your BuildStep can retrieve it by using @code{getLog()}:

@example
class MyBuildStep(ShellCommand):
    logfiles = @{ "nodelog": "_test/node.log" @}

    def evaluateCommand(self, cmd):
        nodelog = self.getLog("nodelog")
        if "STARTED" in nodelog.getText():
            return SUCCESS
        else:
            return FAILURE
@end example

For a complete list of the methods you can call on a LogFile, please
see the docstrings on the @code{IStatusLog} class in
@file{buildbot/interfaces.py}.


@node Adding LogObservers
@subsubsection Adding LogObservers

@cindex LogObserver
@cindex LogLineObserver

Most shell commands emit messages to stdout or stderr as they operate,
especially if you ask them nicely with a @code{--verbose} flag of some
sort. They may also write text to a log file while they run. Your
BuildStep can watch this output as it arrives, to keep track of how
much progress the command has made. You can get a better measure of
progress by counting the number of source files compiled or test cases
run than by merely tracking the number of bytes that have been written
to stdout. This improves the accuracy and the smoothness of the ETA
display.

To accomplish this, you will need to attach a @code{LogObserver} to
one of the log channels, most commonly to the ``stdio'' channel but
perhaps to another one which tracks a log file. This observer is given
all text as it is emitted from the command, and has the opportunity to
parse that output incrementally. Once the observer has decided that
some event has occurred (like a source file being compiled), it can
use the @code{setProgress} method to tell the BuildStep about the
progress that this event represents.

There are a number of pre-built @code{LogObserver} classes that you
can choose from (defined in @code{buildbot.process.buildstep}, and of
course you can subclass them to add further customization. The
@code{LogLineObserver} class handles the grunt work of buffering and
scanning for end-of-line delimiters, allowing your parser to operate
on complete stdout/stderr lines. (Lines longer than a set maximum
length are dropped; the maximum defaults to 16384 bytes, but you can
change it by calling @code{setMaxLineLength()} on your
@code{LogLineObserver} instance.  Use @code{sys.maxint} for effective
infinity.)

For example, let's take a look at the @code{TrialTestCaseCounter}, which is
used by the Trial step (@pxref{Trial}) to count test cases as they are
run.  As Trial executes, it emits lines like the following:

@example
buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK]
buildbot.test.test_config.ConfigTest.testEmpty ... [OK]
buildbot.test.test_config.ConfigTest.testIRC ... [FAIL]
buildbot.test.test_config.ConfigTest.testLocks ... [OK]
@end example

When the tests are finished, trial emits a long line of ``======'' and
then some lines which summarize the tests that failed. We want to
avoid parsing these trailing lines, because their format is less
well-defined than the ``[OK]'' lines.

The parser class looks like this:

@example
from buildbot.process.buildstep import LogLineObserver

class TrialTestCaseCounter(LogLineObserver):
    _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$')
    numTests = 0
    finished = False

    def outLineReceived(self, line):
        if self.finished:
            return
        if line.startswith("=" * 40):
            self.finished = True
            return

        m = self._line_re.search(line.strip())
        if m:
            testname, result = m.groups()
            self.numTests += 1
            self.step.setProgress('tests', self.numTests)
@end example

This parser only pays attention to stdout, since that's where trial
writes the progress lines. It has a mode flag named @code{finished} to
ignore everything after the ``===='' marker, and a scary-looking
regular expression to match each line while hopefully ignoring other
messages that might get displayed as the test runs.

Each time it identifies a test has been completed, it increments its
counter and delivers the new progress value to the step with
@code{self.step.setProgress}. This class is specifically measuring
progress along the ``tests'' metric, in units of test cases (as
opposed to other kinds of progress like the ``output'' metric, which
measures in units of bytes). The Progress-tracking code uses each
progress metric separately to come up with an overall completion
percentage and an ETA value.

To connect this parser into the @code{Trial} BuildStep,
@code{Trial.__init__} ends with the following clause:

@example
        # this counter will feed Progress along the 'test cases' metric
        counter = TrialTestCaseCounter()
        self.addLogObserver('stdio', counter)
        self.progressMetrics += ('tests',)
@end example

This creates a TrialTestCaseCounter and tells the step that the
counter wants to watch the ``stdio'' log. The observer is
automatically given a reference to the step in its @code{.step}
attribute.

@subheading A Somewhat Whimsical Example

Let's say that we've got some snazzy new unit-test framework called
Framboozle. It's the hottest thing since sliced bread. It slices, it
dices, it runs unit tests like there's no tomorrow. Plus if your unit
tests fail, you can use its name for a Web 2.1 startup company, make
millions of dollars, and hire engineers to fix the bugs for you, while
you spend your afternoons lazily hang-gliding along a scenic pacific
beach, blissfully unconcerned about the state of your
tests.@footnote{framboozle.com is still available. Remember, I get 10%
:).}

To run a Framboozle-enabled test suite, you just run the 'framboozler'
command from the top of your source code tree. The 'framboozler'
command emits a bunch of stuff to stdout, but the most interesting bit
is that it emits the line "FNURRRGH!" every time it finishes running a
test case@footnote{Framboozle gets very excited about running unit
tests.}. You'd like to have a test-case counting LogObserver that
watches for these lines and counts them, because counting them will
help the buildbot more accurately calculate how long the build will
take, and this will let you know exactly how long you can sneak out of
the office for your hang-gliding lessons without anyone noticing that
you're gone.

This will involve writing a new BuildStep (probably named
"Framboozle") which inherits from ShellCommand. The BuildStep class
definition itself will look something like this:

@example
# START
from buildbot.steps.shell import ShellCommand
from buildbot.process.buildstep import LogLineObserver

class FNURRRGHCounter(LogLineObserver):
    numTests = 0
    def outLineReceived(self, line):
        if "FNURRRGH!" in line:
            self.numTests += 1
            self.step.setProgress('tests', self.numTests)

class Framboozle(ShellCommand):
    command = ["framboozler"]

    def __init__(self, **kwargs):
        ShellCommand.__init__(self, **kwargs)   # always upcall!
        counter = FNURRRGHCounter())
        self.addLogObserver('stdio', counter)
        self.progressMetrics += ('tests',)
# FINISH
@end example

So that's the code that we want to wind up using. How do we actually
deploy it?

You have a couple of different options.

Option 1: The simplest technique is to simply put this text
(everything from START to FINISH) in your master.cfg file, somewhere
before the BuildFactory definition where you actually use it in a
clause like:

@example
f = BuildFactory()
f.addStep(SVN(svnurl="stuff"))
f.addStep(Framboozle())
@end example

Remember that master.cfg is secretly just a python program with one
job: populating the BuildmasterConfig dictionary. And python programs
are allowed to define as many classes as they like. So you can define
classes and use them in the same file, just as long as the class is
defined before some other code tries to use it.

This is easy, and it keeps the point of definition very close to the
point of use, and whoever replaces you after that unfortunate
hang-gliding accident will appreciate being able to easily figure out
what the heck this stupid "Framboozle" step is doing anyways. The
downside is that every time you reload the config file, the Framboozle
class will get redefined, which means that the buildmaster will think
that you've reconfigured all the Builders that use it, even though
nothing changed. Bleh.

Option 2: Instead, we can put this code in a separate file, and import
it into the master.cfg file just like we would the normal buildsteps
like ShellCommand and SVN.

Create a directory named ~/lib/python, put everything from START to
FINISH in ~/lib/python/framboozle.py, and run your buildmaster using:

@example
 PYTHONPATH=~/lib/python buildbot start MASTERDIR
@end example

or use the @file{Makefile.buildbot} to control the way
@command{buildbot start} works. Or add something like this to
something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc:

@example
 export PYTHONPATH=~/lib/python
@end example

Once we've done this, our master.cfg can look like:

@example
from framboozle import Framboozle
f = BuildFactory()
f.addStep(SVN(svnurl="stuff"))
f.addStep(Framboozle())
@end example

or:

@example
import framboozle
f = BuildFactory()
f.addStep(SVN(svnurl="stuff"))
f.addStep(framboozle.Framboozle())
@end example

(check out the python docs for details about how "import" and "from A
import B" work).

What we've done here is to tell python that every time it handles an
"import" statement for some named module, it should look in our
~/lib/python/ for that module before it looks anywhere else. After our
directories, it will try in a bunch of standard directories too
(including the one where buildbot is installed). By setting the
PYTHONPATH environment variable, you can add directories to the front
of this search list.

Python knows that once it "import"s a file, it doesn't need to
re-import it again. This means that reconfiguring the buildmaster
(with "buildbot reconfig", for example) won't make it think the
Framboozle class has changed every time, so the Builders that use it
will not be spuriously restarted. On the other hand, you either have
to start your buildmaster in a slightly weird way, or you have to
modify your environment to set the PYTHONPATH variable.


Option 3: Install this code into a standard python library directory

Find out what your python's standard include path is by asking it:

@example
80:warner@@luther% python
Python 2.4.4c0 (#2, Oct  2 2006, 00:57:46)
[GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/usr/lib/python24.zip',
 '/usr/lib/python2.4',
 '/usr/lib/python2.4/plat-linux2',
 '/usr/lib/python2.4/lib-tk',
 '/usr/lib/python2.4/lib-dynload',
 '/usr/local/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages/Numeric',
 '/var/lib/python-support/python2.4',
 '/usr/lib/site-python']
@end example

In this case, putting the code into
/usr/local/lib/python2.4/site-packages/framboozle.py would work just
fine. We can use the same master.cfg "import framboozle" statement as
in Option 2. By putting it in a standard include directory (instead of
the decidedly non-standard ~/lib/python), we don't even have to set
PYTHONPATH to anything special. The downside is that you probably have
to be root to write to one of those standard include directories.


Option 4: Submit the code for inclusion in the Buildbot distribution

Make a fork of buildbot on http://github.com/djmitche/buildbot or post a patch
in a bug at http://buildbot.net.  In either case, post a note about your patch
to the mailing list, so others can provide feedback and, eventually, commit it.

@example
from buildbot.steps import framboozle
f = BuildFactory()
f.addStep(SVN(svnurl="stuff"))
f.addStep(framboozle.Framboozle())
@end example

And then you don't even have to install framboozle.py anywhere on your
system, since it will ship with Buildbot. You don't have to be root,
you don't have to set PYTHONPATH. But you do have to make a good case
for Framboozle being worth going into the main distribution, you'll
probably have to provide docs and some unit test cases, you'll need to
figure out what kind of beer the author likes, and then you'll have to
wait until the next release. But in some environments, all this is
easier than getting root on your buildmaster box, so the tradeoffs may
actually be worth it.



Putting the code in master.cfg (1) makes it available to that
buildmaster instance. Putting it in a file in a personal library
directory (2) makes it available for any buildmasters you might be
running. Putting it in a file in a system-wide shared library
directory (3) makes it available for any buildmasters that anyone on
that system might be running. Getting it into the buildbot's upstream
repository (4) makes it available for any buildmasters that anyone in
the world might be running. It's all a matter of how widely you want
to deploy that new class.



@node BuildStep URLs
@subsubsection BuildStep URLs

@cindex links
@cindex BuildStep URLs
@cindex addURL

Each BuildStep has a collection of ``links''. Like its collection of
LogFiles, each link has a name and a target URL. The web status page
creates HREFs for each link in the same box as it does for LogFiles,
except that the target of the link is the external URL instead of an
internal link to a page that shows the contents of the LogFile.

These external links can be used to point at build information hosted
on other servers. For example, the test process might produce an
intricate description of which tests passed and failed, or some sort
of code coverage data in HTML form, or a PNG or GIF image with a graph
of memory usage over time. The external link can provide an easy way
for users to navigate from the buildbot's status page to these
external web sites or file servers. Note that the step itself is
responsible for insuring that there will be a document available at
the given URL (perhaps by using @command{scp} to copy the HTML output
to a @file{~/public_html/} directory on a remote web server). Calling
@code{addURL} does not magically populate a web server.

To set one of these links, the BuildStep should call the @code{addURL}
method with the name of the link and the target URL. Multiple URLs can
be set.

In this example, we assume that the @command{make test} command causes
a collection of HTML files to be created and put somewhere on the
coverage.example.org web server, in a filename that incorporates the
build number.

@example
class TestWithCodeCoverage(BuildStep):
    command = ["make", "test",
               WithProperties("buildnum=%s", "buildnumber")]

    def createSummary(self, log):
        buildnumber = self.getProperty("buildnumber")
        url = "http://coverage.example.org/builds/%s.html" % buildnumber
        self.addURL("coverage", url)
@end example

You might also want to extract the URL from some special message
output by the build process itself:

@example
class TestWithCodeCoverage(BuildStep):
    command = ["make", "test",
               WithProperties("buildnum=%s", "buildnumber")]

    def createSummary(self, log):
        output = StringIO(log.getText())
        for line in output.readlines():
            if line.startswith("coverage-url:"):
                url = line[len("coverage-url:"):].strip()
                self.addURL("coverage", url)
                return
@end example

Note that a build process which emits both stdout and stderr might
cause this line to be split or interleaved between other lines. It
might be necessary to restrict the getText() call to only stdout with
something like this:

@example
        output = StringIO("".join([c[1]
                                   for c in log.getChunks()
                                   if c[0] == LOG_CHANNEL_STDOUT]))
@end example

Of course if the build is run under a PTY, then stdout and stderr will
be merged before the buildbot ever sees them, so such interleaving
will be unavoidable.


@node Build Step Index
@subsection Build Step Index
@printindex bs