Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

1130 lines (907 sloc) 44.787 kb
A Version Control System mantains a source tree, and tells the
buildmaster when it changes. The first step of each Build is typically
to acquire a copy of some version of this tree.
This chapter describes how the Buildbot learns about what Changes have
occurred. For more information on VC systems and Changes, see
@ref{Version Control Systems}.
Changes can be provided by a variety of ChangeSource types, although any given
project will typically have only a single ChangeSource active. This section
provides a description of all available ChangeSource types and explains how to
set up each of them.
In general, each Buildmaster watches a single source tree. It is possible to
work around this, but true support for multi-tree builds remains elusive.
* Choosing a Change Source::
* Configuring Change Sources::
* CVSToys - PBService::
* Mail-parsing ChangeSources::
* PBChangeSource::
* P4Source::
* BonsaiPoller::
* SVNPoller::
* Bzr Poller::
* Change Source Index::
@end menu
@node Choosing a Change Source
@subsection Choosing a Change Source
There are a variety of ChangeSources available, some of which are
meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.
As a quick guide, here is a list of VC systems and the ChangeSources
that might be useful with them. All of these ChangeSources are in the
@code{buildbot.changes} module.
@table @code
@item CVS
@itemize @bullet
@item freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon)
@item mail.FCMaildirSource (watching for email sent by a freshcvs daemon)
@item mail.BonsaiMaildirSource (watching for email sent by Bonsai)
@item mail.SyncmailMaildirSource (watching for email sent by syncmail)
@item pb.PBChangeSource (listening for connections from @code{buildbot
sendchange} run in a loginfo script)
@item pb.PBChangeSource (listening for connections from a long-running
@code{contrib/} polling process which examines the ViewCVS
database directly
@end itemize
@item SVN
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} run in a postcommit script)
@item pb.PBChangeSource (listening for connections from a long-running
@code{contrib/} or @code{contrib/} polling
@item mail.SVNCommitEmailMaildirSource (watching for email sent by
@item svnpoller.SVNPoller (polling the SVN repository)
@end itemize
@item Darcs
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} in a commit script
@end itemize
@item Mercurial
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} run in an 'incoming' hook)
@item pb.PBChangeSource (listening for connections from
@code{buildbot/changes/} run as an in-process 'changegroup'
@end itemize
@item Arch/Bazaar
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} run in a commit hook)
@end itemize
@item Bzr (the newer Bazaar)
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} run in a post-change-branch-tip or commit hook)
@item @code{contrib/}'s BzrPoller (polling the Bzr repository)
@end itemize
@item Git
@itemize @bullet
@item pb.PBChangeSource (listening for connections from
@code{contrib/} run in the post-receive hook)
@end itemize
@end table
All VC systems can be driven by a PBChangeSource and the
@code{buildbot sendchange} tool run from some form of commit script.
If you write an email parsing function, they can also all be driven by
a suitable @code{MaildirSource}.
@node Configuring Change Sources
@subsection Configuring Change Sources
@bcindex c['change_source']
The @code{master.cfg} configuration file has a dictionary key named
@code{BuildmasterConfig['change_source']}, which holds the active
@code{IChangeSource} object. The config file will typically create an
object from one of the classes described below and stuff it into this
Each buildmaster typically has just a single ChangeSource, since it is
only watching a single source tree. But if, for some reason, you need
multiple sources, just set @code{c['change_source']} to a list of
s = FreshCVSSourceNewcred(host="host", port=4519,
user="alice", passwd="secret",
c['change_source'] = [s]
@end example
Each source tree has a nominal @code{top}. Each Change has a list of
filenames, which are all relative to this top location. The
ChangeSource is responsible for doing whatever is necessary to
accomplish this. Most sources have a @code{prefix} argument: a partial
pathname which is stripped from the front of all filenames provided to
that @code{ChangeSource}. Files which are outside this sub-tree are
ignored by the changesource: it does not generate Changes for those
@heading Repository and Project
ChangeSources will, in general, automatically provide the proper 'repository'
attribute for any changes they produce. For systems which operate on URL-like
specifiers, this is a repository URL. Other ChangeSources adapt the concept as
Many ChangeSources allow you to specify a project, as well. This attribute is
useful when building from several distinct codebases in the same buildmaster:
the project string can serve to differentiate the different codebases.
Schedulers can filter on project, so you can configure different builders to
run for each project.
@node CVSToys - PBService
@subsection CVSToys - PBService
@csindex buildbot.changes.freshcvs.FreshCVSSource
@csindex buildbot.changes.freshcvs.FreshCVSSourceOldcred
The @uref{, CVSToys} package provides a
server which runs on the machine that hosts the CVS repository it
watches. It has a variety of ways to distribute commit notifications,
and offers a flexible regexp-based way to filter out uninteresting
changes. One of the notification options is named @code{PBService} and
works by listening on a TCP port for clients. These clients subscribe
to hear about commit notifications.
The buildmaster has a CVSToys-compatible @code{PBService} client built in.
There are two versions of it, one for old versions of CVSToys (1.0.9 and
earlier, @code{FreshCVSSourceOldcred}) which used the @code{oldcred}
authentication framework, and one for newer versions (1.0.10 and later) which
use @code{newcred}. Both are classes in the @code{buildbot.changes.freshcvs}
@code{FreshCVSSource} objects are created with the following
@table @samp
@item @code{host} and @code{port}
these specify where the CVSToys server can be reached
@item @code{user} and @code{passwd}
these specify the login information for the CVSToys server
(@code{freshcvs}). These must match the server's values, which are
defined in the @code{freshCfg} configuration file (which lives in the
CVSROOT directory of the repository).
@item @code{prefix}
this is the prefix to be found and stripped from filenames delivered
by the CVSToys server. Most projects live in sub-directories of the
main repository, as siblings of the CVSROOT sub-directory, so
typically this prefix is set to that top sub-directory name.
@end table
@heading Example
To set up the freshCVS server, add a statement like the following to
your @file{freshCfg} file:
pb = ConfigurationSet([
(None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
@end example
This will announce all changes to a client which connects to port 4519
using a username of 'foo' and a password of 'bar'.
Then add a clause like this to your buildmaster's @file{master.cfg}:
c['change_source'] = FreshCVSSource("", 4519,
"foo", "bar",
@end example
where "" is the host that is running the FreshCVS daemon, and
"glib" is the top-level directory (relative to the repository's root) where
all your source code lives. Most projects keep one or more projects in the
same repository (along with CVSROOT/ to hold admin files like loginfo and
freshCfg); the prefix= argument tells the buildmaster to ignore everything
outside that directory, and to strip that common prefix from all pathnames
it handles.
@node Mail-parsing ChangeSources
@subsection Mail-parsing ChangeSources
Many projects publish information about changes to their source tree
by sending an email message out to a mailing list, frequently named
PROJECT-commits or PROJECT-changes. Each message usually contains a
description of the change (who made the change, which files were
affected) and sometimes a copy of the diff. Humans can subscribe to
this list to stay informed about what's happening to the source tree.
The Buildbot can also be subscribed to a -commits mailing list, and
can trigger builds in response to Changes that it hears about. The
buildmaster admin needs to arrange for these email messages to arrive
in a place where the buildmaster can find them, and configure the
buildmaster to parse the messages correctly. Once that is in place,
the email parser will create Change objects and deliver them to the
Schedulers (see @pxref{Schedulers}) just like any other ChangeSource.
There are two components to setting up an email-based ChangeSource.
The first is to route the email messages to the buildmaster, which is
done by dropping them into a ``maildir''. The second is to actually
parse the messages, which is highly dependent upon the tool that was
used to create them. Each VC system has a collection of favorite
change-emailing tools, and each has a slightly different format, so
each has a different parsing function. There is a separate
ChangeSource variant for each parsing function.
Once you've chosen a maildir location and a parsing function, create
the change source and put it in @code{c['change_source']}:
from buildbot.changes.mail import SyncmailMaildirSource
c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot",
@end example
* Subscribing the Buildmaster::
* Using Maildirs::
* Parsing Email Change Messages::
* FCMaildirSource::
* SyncmailMaildirSource::
* BonsaiMaildirSource::
* SVNCommitEmailMaildirSource::
* BzrLaunchpadEmailMaildirSource::
@end menu
@node Subscribing the Buildmaster
@subsubsection Subscribing the Buildmaster
The recommended way to install the buildbot is to create a dedicated
account for the buildmaster. If you do this, the account will probably
have a distinct email address (perhaps
@email{}). Then just arrange for this
account's email to be delivered to a suitable maildir (described in
the next section).
If the buildbot does not have its own account, ``extension addresses''
can be used to distinguish between email intended for the buildmaster
and email intended for the rest of the account. In most modern MTAs,
the e.g. @code{} account has control over every email
address at which begins with "foo", such that email
addressed to @email{} can be delivered to a
different destination than @email{}. qmail
does this by using separate .qmail files for the two destinations
(@file{.qmail-foo} and @file{.qmail-bar}, with @file{.qmail}
controlling the base address and @file{.qmail-default} controlling all
other extensions). Other MTAs have similar mechanisms.
Thus you can assign an extension address like
@email{} to the buildmaster, and retain
@email{} for your own use.
@node Using Maildirs
@subsubsection Using Maildirs
A ``maildir'' is a simple directory structure originally developed for
qmail that allows safe atomic update without locking. Create a base
directory with three subdirectories: ``new'', ``tmp'', and ``cur''.
When messages arrive, they are put into a uniquely-named file (using
pids, timestamps, and random numbers) in ``tmp''. When the file is
complete, it is atomically renamed into ``new''. Eventually the
buildmaster notices the file in ``new'', reads and parses the
contents, then moves it into ``cur''. A cronjob can be used to delete
files in ``cur'' at leisure.
Maildirs are frequently created with the @command{maildirmake} tool,
but a simple @command{mkdir -p ~/MAILDIR/@{cur,new,tmp@}} is pretty much
Many modern MTAs can deliver directly to maildirs. The usual .forward
or .procmailrc syntax is to name the base directory with a trailing
slash, so something like @code{~/MAILDIR/} . qmail and postfix are
maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail
Delivery Agent).
For MTAs which cannot put files into maildirs directly, the
``safecat'' tool can be executed from a .forward file to accomplish
the same thing.
The Buildmaster uses the linux DNotify facility to receive immediate
notification when the maildir's ``new'' directory has changed. When
this facility is not available, it polls the directory for new
messages, every 10 seconds by default.
@node Parsing Email Change Messages
@subsubsection Parsing Email Change Messages
The second component to setting up an email-based ChangeSource is to
parse the actual notices. This is highly dependent upon the VC system
and commit script in use.
A couple of common tools used to create these change emails are:
@table @b
@item CVS
@table @samp
@item CVSToys MailNotifier
@item Bonsai notification
@item syncmail
@end table
@item SVN
@table @samp
@item svnmailer
@end table
@item Bzr
@table @samp
@item Launchpad
@end table
@item Mercurial
@table @samp
@item NotifyExtension
@end table
@item Git
@table @samp
@item post-receive-email;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD
@end table
@end table
The following sections describe the parsers available for each of
these tools.
Most of these parsers accept a @code{prefix=} argument, which is used
to limit the set of files that the buildmaster pays attention to. This
is most useful for systems like CVS and SVN which put multiple
projects in a single repository (or use repository names to indicate
branches). Each filename that appears in the email is tested against
the prefix: if the filename does not start with the prefix, the file
is ignored. If the filename @emph{does} start with the prefix, that
prefix is stripped from the filename before any further processing is
done. Thus the prefix usually ends with a slash.
@node FCMaildirSource
@subsubsection FCMaildirSource
@csindex buildbot.changes.mail.FCMaildirSource
This parser works with the CVSToys @code{MailNotification} action,
which will send email to a list of recipients for each commit. This
tends to work better than using @code{/bin/mail} from within the
CVSROOT/loginfo file directly, as CVSToys will batch together all
files changed during the same CVS invocation, and can provide more
information (like creating a ViewCVS URL for each file changed).
The Buildbot's @code{FCMaildirSource} knows for to parse these CVSToys
messages and turn them into Change objects. It can be given two
parameters: the directory name of the maildir root, and the prefix to
from buildbot.changes.mail import FCMaildirSource
c['change_source'] = FCMaildirSource("~/maildir-buildbot")
@end example
@node SyncmailMaildirSource
@subsubsection SyncmailMaildirSource
@csindex buildbot.changes.mail.SyncmailMaildirSource
@code{SyncmailMaildirSource} knows how to parse the message format used by
the CVS ``syncmail'' script.
from buildbot.changes.mail import SyncmailMaildirSource
c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot")
@end example
@node BonsaiMaildirSource
@subsubsection BonsaiMaildirSource
@csindex buildbot.changes.mail.BonsaiMaildirSource
@code{BonsaiMaildirSource} parses messages sent out by Bonsai, the CVS
tree-management system built by Mozilla.
from buildbot.changes.mail import BonsaiMaildirSource
c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot")
@end example
@node SVNCommitEmailMaildirSource
@subsubsection SVNCommitEmailMaildirSource
@csindex buildbot.changes.mail.SVNCommitEmailMaildirSource
@code{SVNCommitEmailMaildirSource} parses message sent out by the
@code{} script, which is included in the Subversion
It does not currently handle branches: all of the Change objects that
it creates will be associated with the default (i.e. trunk) branch.
from buildbot.changes.mail import SVNCommitEmailMaildirSource
c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot")
@end example
@node BzrLaunchpadEmailMaildirSource
@subsubsection BzrLaunchpadEmailMaildirSource
@csindex buildbot.changes.mail.BzrLaunchpadEmailMaildirSource
@code{BzrLaunchpadEmailMaildirSource} parses the mails that are sent to
addresses that subscribe to branch revision notifications for a bzr branch
hosted on Launchpad.
The branch name defaults to @code{lp:<Launchpad path>}. For example
If only a single branch is used, the default branch name can be changed by
setting @code{defaultBranch}.
For multiple branches, pass a dictionary as the value of the @code{branchMap}
option to map specific repository paths to specific branch names (see example
below). The leading @code{lp:} prefix of the path is optional.
The @code{prefix} option is not supported (it is silently ignored). Use the
@code{branchMap} and @code{defaultBranch} instead to assign changes to
branches (and just do not subscribe the buildbot to branches that are not of
The revision number is obtained from the email text. The bzr revision id is
not available in the mails sent by Launchpad. However, it is possible to set
the bzr @code{append_revisions_only} option for public shared repositories to
avoid new pushes of merges changing the meaning of old revision numbers.
from buildbot.changes.mail import BzrLaunchpadEmailMaildirSource
bm = { 'lp:~maria-captains/maria/5.1' : '5.1', 'lp:~maria-captains/maria/6.0' : '6.0' }
c['change_source'] = BzrLaunchpadEmailMaildirSource("~/maildir-buildbot", branchMap = bm)
@end verbatim
@node PBChangeSource
@subsection PBChangeSource
@csindex buildbot.changes.pb.PBChangeSource
The last kind of ChangeSource actually listens on a TCP port for
clients to connect and push change notices @emph{into} the
Buildmaster. This is used by the built-in @code{buildbot sendchange}
notification tool, as well as the VC-specific
@file{contrib/}, @file{contrib/},
@file{contrib/} tools, and the
@code{buildbot.changes.hgbuildbot} hook. These tools are run by the
repository (in a commit hook script), and connect to the buildmaster
directly each time a file is comitted. This is also useful for
creating new kinds of change sources that work on a @code{push} model
instead of some kind of subscription scheme, for example a script
which is run out of an email .forward file.
This ChangeSource can be configured to listen on its own TCP port, or
it can share the port that the buildmaster is already using for the
buildslaves to connect. (This is possible because the
@code{PBChangeSource} uses the same protocol as the buildslaves, and
they can be distinguished by the @code{username} attribute used when
the initial connection is established). It might be useful to have it
listen on a different port if, for example, you wanted to establish
different firewall rules for that port. You could allow only the SVN
repository machine access to the @code{PBChangeSource} port, while
allowing only the buildslave machines access to the slave port. Or you
could just expose one port and run everything over it. @emph{Note:
this feature is not yet implemented, the PBChangeSource will always
share the slave port and will always have a @code{user} name of
@code{change}, and a passwd of @code{changepw}. These limitations will
be removed in the future.}.
The @code{PBChangeSource} is created with the following arguments. All
are optional.
@table @samp
@item @code{port}
which port to listen on. If @code{None} (which is the default), it
shares the port used for buildslave connections. @emph{Not
Implemented, always set to @code{None}}.
@item @code{user} and @code{passwd}
The user/passwd account information that the client program must use
to connect. Defaults to @code{change} and @code{changepw}. @emph{Not
Implemented, @code{user} is currently always set to @code{change},
@code{passwd} is always set to @code{changepw}}.
@item @code{prefix}
The prefix to be found and stripped from filenames delivered over the
connection. Any filenames which do not start with this prefix will be
removed. If all the filenames in a given Change are removed, the that
whole Change will be dropped. This string should probably end with a
directory separator.
This is useful for changes coming from version control systems that
represent branches as parent directories within the repository (like
SVN and Perforce). Use a prefix of 'trunk/' or
'project/branches/foobranch/' to only follow one branch and to get
correct tree-relative filenames. Without a prefix, the PBChangeSource
will probably deliver Changes with filenames like @file{trunk/foo.c}
instead of just @file{foo.c}. Of course this also depends upon the
tool sending the Changes in (like @command{buildbot sendchange}) and
what filenames it is delivering: that tool may be filtering and
stripping prefixes at the sending end.
@end table
The following hooks are useful for sending changes to a PBChangeSource:
* Mercurial Hook::
* Bzr Hook::
@end menu
@node Mercurial Hook
@subsubsection Mercurial Hook
Since Mercurial is written in python, the hook script can invoke
Buildbot's @code{sendchange} function directly, rather than having to
spawn an external process. This function delivers the same sort of
changes as @code{buildbot sendchange} and the various hook scripts in
contrib/, so you'll need to add a @code{pb.PBChangeSource} to your
buildmaster to receive these changes.
To set this up, first choose a Mercurial repository that represents
your central ``official'' source tree. This will be the same
repository that your buildslaves will eventually pull from. Install
Buildbot on the machine that hosts this repository, using the same
version of python as Mercurial is using (so that the Mercurial hook
can import code from buildbot). Then add the following to the
@code{.hg/hgrc} file in that repository, replacing the buildmaster
hostname/portnumber as appropriate for your buildbot:
changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook
master =
@end example
(Note that Mercurial lets you define multiple @code{changegroup} hooks
by giving them distinct names, like @code{} and
@code{}, which is why we use
@code{changegroup.buildbot} in this example. There is nothing magical
about the ``buildbot'' suffix in the hook name. The
@code{[hgbuildbot]} section @emph{is} special, however, as it is the
only section that the buildbot hook pays attention to.)
Also note that this runs as a @code{changegroup} hook, rather than as
an @code{incoming} hook. The @code{changegroup} hook is run with
multiple revisions at a time (say, if multiple revisions are being
pushed to this repository in a single @command{hg push} command),
whereas the @code{incoming} hook is run with just one revision at a
time. The @code{hgbuildbot.hook} function will only work with the
@code{changegroup} hook.
The @code{[hgbuildbot]} section has two other parameters that you
might specify, both of which control the name of the branch that is
attached to the changes coming from this hook.
One common branch naming policy for Mercurial repositories is to use
it just like Darcs: each branch goes into a separate repository, and
all the branches for a single project share a common parent directory.
For example, you might have @file{/var/repos/PROJECT/trunk/} and
@file{/var/repos/PROJECT/release}. To use this style, use the
@code{branchtype = dirname} setting, which simply uses the last
component of the repository's enclosing directory as the branch name:
master =
branchtype = dirname
@end example
Another approach is to use Mercurial's built-in branches (the kind
created with @command{hg branch} and listed with @command{hg
branches}). This feature associates persistent names with particular
lines of descent within a single repository. (note that the buildbot
@code{source.Mercurial} checkout step does not yet support this kind
of branch). To have the commit hook deliver this sort of branch name
with the Change object, use @code{branchtype = inrepo}:
master =
branchtype = inrepo
@end example
Finally, if you want to simply specify the branchname directly, for
all changes, use @code{branch = BRANCHNAME}. This overrides
master =
branch = trunk
@end example
If you use @code{branch=} like this, you'll need to put a separate
.hgrc in each repository. If you use @code{branchtype=}, you may be
able to use the same .hgrc for all your repositories, stored in
@file{~/.hgrc} or @file{/etc/mercurial/hgrc}.
As twisted needs to hook some Signals, and that some web server are
strictly forbiding that, the parameter @code{fork} in the
@code{[hgbuildbot]} section will instruct mercurial to fork before
sending the change request. Then as the created process will be of short
life, it is considered as safe to disable the signal restriction in
the Apache setting like that @code{WSGIRestrictSignal Off}. Refer to the
documentation of your web server for other way to do the same.
The @code{category} parameter sets the category for any changes generated from
the hook. Likewise, the @code{project} parameter sets the project. Changes'
@code{repository} attributes are formed from the Mercurial repo path by
stripping @code{strip} slashes.
@node Bzr Hook
@subsubsection Bzr Hook
Bzr is also written in Python, and the Bzr hook depends on Twisted to send the
To install, put @code{contrib/} in one of your plugins
locations a bzr plugins directory (e.g.,
@code{~/.bazaar/plugins}). Then, in one of your bazaar conf files (e.g.,
@code{~/.bazaar/locations.conf}), set the location you want to connect with buildbot
with these keys:
@table @code
@item buildbot_on
one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via
commit, changes via push, or any changes to the trunk. 'change' is
@item buildbot_server
(required to send to a buildbot master) the URL of the buildbot master to
which you will connect (as of this writing, the same server and port to which
slaves connect).
@item buildbot_port
(optional, defaults to 9989) the port of the buildbot master to which you will
connect (as of this writing, the same server and port to which slaves connect)
@item buildbot_pqm
(optional, defaults to not pqm) Normally, the user that commits the revision
is the user that is responsible for the change. When run in a pqm (Patch Queue
Manager, see environment, the user that commits is
the Patch Queue Manager, and the user that committed the *parent* revision is
responsible for the change. To turn on the pqm mode, set this value to any of
(case-insensitive) "Yes", "Y", "True", or "T".
@item buildbot_dry_run
(optional, defaults to not a dry run) Normally, the post-commit hook will
attempt to communicate with the configured buildbot server and port. If this
parameter is included and any of (case-insensitive) "Yes", "Y", "True", or
"T", then the hook will simply print what it would have sent, but not attempt
to contact the buildbot master.
@item buildbot_send_branch_name
(optional, defaults to not sending the branch name) If your buildbot's bzr
source build step uses a repourl, do *not* turn this on. If your buildbot's
bzr build step uses a baseURL, then you may set this value to any of
(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master
append the branch name to the baseURL.
@end table
When buildbot no longer has a hardcoded password, it will be a configuration
option here as well.
Here's a simple example that you might have in your
buildbot_on = change
buildbot_server = localhost
@end example
@node P4Source
@subsection P4Source
@csindex buildbot.changes.p4poller.P4Source
The @code{P4Source} periodically polls a @uref{,
Perforce} depot for changes. It accepts the following arguments:
@table @samp
@item @code{p4base}
The base depot path to watch, without the trailing '/...'.
@item @code{p4port}
The Perforce server to connect to (as host:port).
@item @code{p4user}
The Perforce user.
@item @code{p4passwd}
The Perforce password.
@item @code{p4bin}
An optional string parameter. Specify the location of the perforce command
line binary (p4). You only need to do this if the perforce binary is not
in the path of the buildbot user. Defaults to ``p4''.
@item @code{split_file}
A function that maps a pathname, without the leading @code{p4base}, to a
(branch, filename) tuple. The default just returns (None, branchfile),
which effectively disables branch support. You should supply a function
which understands your repository structure.
@item @code{pollinterval}
How often to poll, in seconds. Defaults to 600 (10 minutes).
@item @code{histmax}
The maximum number of changes to inspect at a time. If more than this
number occur since the last poll, older changes will be silently
@end table
@heading Example
This configuration uses the @code{P4PORT}, @code{P4USER}, and @code{P4PASSWD}
specified in the buildmaster's environment. It watches a project in which the
branch name is simply the next path component, and the file is all path
components after.
from buildbot.changes import p4poller
s = p4poller.P4Source(p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1),
c['change_source'] = s
@end example
@node BonsaiPoller
@subsection BonsaiPoller
@csindex buildbot.changes.bonsaipoller.BonsaiPoller
The @code{BonsaiPoller} periodically polls a Bonsai server. This is a
CGI script accessed through a web server that provides information
about a CVS tree, for example the Mozilla bonsai server at
@uref{}. Bonsai servers are usable by both
humans and machines. In this case, the buildbot's change source forms
a query which asks about any files in the specified branch which have
changed since the last query.
Please take a look at the BonsaiPoller docstring for details about the
arguments it accepts.
@node SVNPoller
@subsection SVNPoller
@csindex buildbot.changes.svnpoller.SVNPoller
The @code{buildbot.changes.svnpoller.SVNPoller} is a ChangeSource
which periodically polls a @uref{,
Subversion} repository for new revisions, by running the @code{svn
log} command in a subshell. It can watch a single branch or multiple
@code{SVNPoller} accepts the following arguments:
@table @code
@item svnurl
The base URL path to watch, like
@code{svn://}, or
@code{}, or even
@code{file:///home/svn/Repository/ProjectA/branches/1.5/}. This must
include the access scheme, the location of the repository (both the
hostname for remote ones, and any additional directory names necessary
to get to the repository), and the sub-path within the repository's
virtual filesystem for the project and branch of interest.
The @code{SVNPoller} will only pay attention to files inside the
subdirectory specified by the complete svnurl.
@item split_file
A function to convert pathnames into (branch, relative_pathname)
tuples. Use this to explain your repository's branch-naming policy to
@code{SVNPoller}. This function must accept a single string and return
a two-entry tuple. There are a few utility functions in
@code{buildbot.changes.svnpoller} that can be used as a
@code{split_file} function, see below for details.
The default value always returns (None, path), which indicates that
all files are on the trunk.
Subclasses of @code{SVNPoller} can override the @code{split_file}
method instead of using the @code{split_file=} argument.
@item svnuser
An optional string parameter. If set, the @code{--user} argument will
be added to all @code{svn} commands. Use this if you have to
authenticate to the svn server before you can do @code{svn info} or
@code{svn log} commands.
@item svnpasswd
Like @code{svnuser}, this will cause a @code{--password} argument to
be passed to all svn commands.
@item pollinterval
How often to poll, in seconds. Defaults to 600 (checking once every 10
minutes). Lower this if you want the buildbot to notice changes
faster, raise it if you want to reduce the network and CPU load on
your svn server. Please be considerate of public SVN repositories by
using a large interval when polling them.
@item histmax
The maximum number of changes to inspect at a time. Every POLLINTERVAL
seconds, the @code{SVNPoller} asks for the last HISTMAX changes and
looks through them for any ones it does not already know about. If
more than HISTMAX revisions have been committed since the last poll,
older changes will be silently ignored. Larger values of histmax will
cause more time and memory to be consumed on each poll attempt.
@code{histmax} defaults to 100.
@item svnbin
This controls the @code{svn} executable to use. If subversion is
installed in a weird place on your system (outside of the
buildmaster's @code{$PATH}), use this to tell @code{SVNPoller} where
to find it. The default value of ``svn'' will almost always be
@item revlinktmpl
This parameter allows a link to be provided for each revision (for example,
to websvn or viewvc). These links appear anywhere changes are shown, such
as on build or change pages. The proper form for this parameter is an URL
with the portion that will substitute for a revision number replaced by
''%s''. For example, @code{'http://myserver/websvn/revision.php?rev=%s'}
could be used to cause revision links to be created to a websvn repository
@end table
@heading Branches
Each source file that is tracked by a Subversion repository has a
fully-qualified SVN URL in the following form:
(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the
@code{SVNPoller}, you give it a @code{svnurl} value that includes all
of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH
string. The @code{SVNPoller} is responsible for producing Changes that
contain a branch name and a FILEPATH (which is relative to the top of
a checked-out tree). The details of how these strings are split up
depend upon how your repository names its branches.
@subheading PROJECT/BRANCHNAME/FILEPATH repositories
One common layout is to have all the various projects that share a
repository get a single top-level directory each. Then under a given
project's directory, you get two subdirectories, one named ``trunk''
and another named ``branches''. Under ``branches'' you have a bunch of
other directories, one per branch, with names like ``1.5.x'' and
``testing''. It is also common to see directories like ``tags'' and
``releases'' next to ``branches'' and ``trunk''.
For example, the Twisted project has a subversion server on
``'' that hosts several sub-projects. The
repository is available through a SCHEME of ``svn:''. The primary
sub-project is Twisted, of course, with a repository root of
``svn://''. Another sub-project is
Informant, with a root of
``svn://'', etc. Inside any
checked-out Twisted tree, there is a file named bin/trial (which is
used to run unit test suites).
The trunk for Twisted is in
``svn://'', and the
fully-qualified SVN URL for the trunk version of @code{trial} would be
``svn://''. The same
SVNURL for that file on a branch named ``1.5.x'' would be
To set up a @code{SVNPoller} that watches the Twisted trunk (and
nothing else), we would use the following:
from buildbot.changes.svnpoller import SVNPoller
c['change_source'] = SVNPoller("svn://")
@end example
In this case, every Change that our @code{SVNPoller} produces will
have @code{.branch=None}, to indicate that the Change is on the trunk.
No other sub-projects or branches will be tracked.
If we want our ChangeSource to follow multiple branches, we have to do
two things. First we have to change our @code{svnurl=} argument to
watch more than just ``.../Twisted/trunk''. We will set it to
``.../Twisted'' so that we'll see both the trunk and all the branches.
Second, we have to tell @code{SVNPoller} how to split the
(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository
out into (BRANCH) and (FILEPATH) pairs.
We do the latter by providing a ``split_file'' function. This function
is responsible for splitting something like
``branches/1.5.x/bin/trial'' into @code{branch}=''branches/1.5.x'' and
@code{filepath}=''bin/trial''. This function is always given a string
that names a file relative to the subdirectory pointed to by the
@code{SVNPoller}'s @code{svnurl=} argument. It is expected to return a
(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the
branch indicated), or None to indicate that the file is outside any
project of interest.
(note that we want to see ``branches/1.5.x'' rather than just
``1.5.x'' because when we perform the SVN checkout, we will probably
append the branch name to the baseURL, which requires that we keep the
``branches'' component in there. Other VC schemes use a different
approach towards branches and may not require this artifact.)
If your repository uses this same PROJECT/BRANCH/FILEPATH naming
scheme, the following function will work:
def split_file_branches(path):
pieces = path.split('/')
if pieces[0] == 'trunk':
return (None, '/'.join(pieces[1:]))
elif pieces[0] == 'branches':
return ('/'.join(pieces[0:2]),
return None
@end example
This function is provided as
@code{buildbot.changes.svnpoller.split_file_branches} for your
convenience. So to have our Twisted-watching @code{SVNPoller} follow
multiple branches, we would use this:
from buildbot.changes.svnpoller import SVNPoller, split_file_branches
c['change_source'] = SVNPoller("svn://",
@end example
Changes for all sorts of branches (with names like ``branches/1.5.x'',
and None to indicate the trunk) will be delivered to the Schedulers.
Each Scheduler is then free to use or ignore each branch as it sees
@subheading BRANCHNAME/PROJECT/FILEPATH repositories
Another common way to organize a Subversion repository is to put the
branch name at the top, and the projects underneath. This is
especially frequent when there are a number of related sub-projects
that all get released in a group.
For example, hosts a project named ``Nevow'' as well as one
named ``Quotient''. In a checked-out Nevow tree there is a directory
named ``formless'' that contains a python source file named
``''. This repository is accessible via webdav (and thus
uses an ``http:'' scheme) through the hostname. There are
many branches in this repository, and they use a
(BRANCHNAME)/(PROJECT) naming policy.
The fully-qualified SVN URL for the trunk version of is
You can do an @code{svn co} with that URL and get a copy of the latest
version. The 1.5.x branch version of this file would have a URL of
The whole Nevow trunk would be checked out with
@code{}, while the Quotient
trunk would be checked out using
Now suppose we want to have an @code{SVNPoller} that only cares about
the Nevow trunk. This case looks just like the PROJECT/BRANCH layout
described earlier:
from buildbot.changes.svnpoller import SVNPoller
c['change_source'] = SVNPoller("")
@end example
But what happens when we want to track multiple Nevow branches? We
have to point our @code{svnurl=} high enough to see all those
branches, but we also don't want to include Quotient changes (since
we're only building Nevow). To accomplish this, we must rely upon the
@code{split_file} function to help us tell the difference between
files that belong to Nevow and those that belong to Quotient, as well
as figuring out which branch each one is on.
from buildbot.changes.svnpoller import SVNPoller
c['change_source'] = SVNPoller("",
@end example
The @code{my_file_splitter} function will be called with
repository-relative pathnames like:
@table @code
@item trunk/Nevow/formless/
This is a Nevow file, on the trunk. We want the Change that includes this
to see a filename of @code{formless/"}, and a branch of None
@item branches/1.5.x/Nevow/formless/
This is a Nevow file, on a branch. We want to get
branch=''branches/1.5.x'' and filename=''formless/''.
@item trunk/Quotient/
This is a Quotient file, so we want to ignore it by having
@code{my_file_splitter} return None.
@item branches/1.5.x/Quotient/
This is also a Quotient file, which should be ignored.
@end table
The following definition for @code{my_file_splitter} will do the job:
def my_file_splitter(path):
pieces = path.split('/')
if pieces[0] == 'trunk':
branch = None
pieces.pop(0) # remove 'trunk'
elif pieces[0] == 'branches':
pieces.pop(0) # remove 'branches'
# grab branch name
branch = 'branches/' + pieces.pop(0)
return None # something weird
projectname = pieces.pop(0)
if projectname != 'Nevow':
return None # wrong project
return (branch, '/'.join(pieces))
@end example
@node Bzr Poller
@subsection Bzr Poller
If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To
use, put @code{contrib/} somewhere that your buildbot
configuration can import it. Even putting it in the same directory as the master.cfg
should work. Install the poller in the buildbot configuration as with any
other change source. Minimally, provide a URL that you want to poll (bzr://,
bzr+ssh://, or lp:), though make sure the buildbot user has necessary
privileges. You may also want to specify these optional values.
@table @code
@item poll_interval
The number of seconds to wait between polls. Defaults to 10 minutes.
@item branch_name
Any value to be used as the branch name. Defaults to None, or specify a
string, or specify the constants from @code{} SHORT or FULL to
get the short branch name or full branch address.
@item blame_merge_author
normally, the user that commits the revision is the user that is responsible
for the change. When run in a pqm (Patch Queue Manager, see environment, the user that commits is the Patch
Queue Manager, and the user that committed the merged, *parent* revision is
responsible for the change. set this value to True if this is pointed against
a PQM-managed branch.
@end table
@node Change Source Index
@subsection Change Source Index
@printindex cs
Jump to Line
Something went wrong with that request. Please try again.