A Version Control System maintains a source tree, and tells the buildmaster when it changes. The first step of each Build
is typically to acquire a copy of some version of this tree.
This chapter describes how the Buildbot learns about what Change
s have occurred. For more information on VC systems and Change
s, see Version-Control-Systems
.
Change
s can be provided by a variety of ChangeSource
types, although any given project will typically have only a single ChangeSource
active. This section provides a description of all available ChangeSource
types and explains how to set up each of them.
In general, each Buildmaster watches a single source tree. It is possible to work around this, but true support for multi-tree builds remains elusive.
There are a variety of ChangeSource
classes available, some of which are meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.
As a quick guide, here is a list of VC systems and the ChangeSource
s that might be useful with them. Note that some of these modules are in Buildbot's "contrib" directory, meaning that they have been offered by other users in hopes they may be useful, and might require some additional work to make them functional.
- CVS
- :bb
CVSMaildirSource
(watching mail sent bycontrib/buildbot_cvs_mail.py
script) - :bb
PBChangeSource
(listening for connections frombuildbot sendchange
run in a loginfo script) - :bb
PBChangeSource
(listening for connections from a long-runningcontrib/viewcvspoll.py
polling process which examines the ViewCVS database directly - :bb
Change Hooks
in WebStatus
- :bb
- SVN
- :bb
PBChangeSource
(listening for connections fromcontrib/svn_buildbot.py
run in a postcommit script) - :bb
PBChangeSource
(listening for connections from a long-runningcontrib/svn_watcher.py
orcontrib/svnpoller.py
polling process - :bb
SVNCommitEmailMaildirSource
(watching for email sent bycommit-email.pl
) - :bb
SVNPoller
(polling the SVN repository) - :bb
Change Hooks
in WebStatus - :bb
GoogleCodeAtomPoller
(polling the commit feed for a GoogleCode Git repository)
- :bb
- Darcs
- :bb
PBChangeSource
(listening for connections fromcontrib/darcs_buildbot.py
in a commit script) - :bb
Change Hooks
in WebStatus
- :bb
- Mercurial
- :bb
PBChangeSource
(listening for connections fromcontrib/hg_buildbot.py
run in an 'changegroup' hook) - :bb
Change Hooks
in WebStatus - :bb
PBChangeSource
(listening for connections frombuildbot/changes/hgbuildbot.py
run as an in-process 'changegroup' hook) - :bb
HgPoller
(polling a remote Mercurial repository) - :bb
GoogleCodeAtomPoller
(polling the commit feed for a GoogleCode Git repository)
- :bb
- Bzr (the newer Bazaar)
- :bb
PBChangeSource
(listening for connections fromcontrib/bzr_buildbot.py
run in a post-change-branch-tip or commit hook) - :bb
BzrPoller
(polling the Bzr repository) - :bb
Change Hooks
in WebStatus
- :bb
- Git
- :bb
PBChangeSource
(listening for connections fromcontrib/git_buildbot.py
run in the post-receive hook) - :bb
PBChangeSource
(listening for connections fromcontrib/github_buildbot.py
, which listens for notifications from GitHub) - :bb
Change Hooks
in WebStatus - GitHub change hook (specifically designed for GitHub notifications, but requiring a publicly-accessible WebStatus)
- :bb
GitPoller
(polling a remote Git repository) - :bb
GoogleCodeAtomPoller
(polling the commit feed for a GoogleCode Git repository)
- :bb
- Repo/Git
- :bb
GerritChangeSource
connects to Gerrit via SSH to get a live stream of changes
- :bb
- Monotone
- :bb
PBChangeSource
(listening for connections frommonotone-buildbot.lua
, which is available with Monotone)
- :bb
All VC systems can be driven by a :bbPBChangeSource
and the buildbot sendchange
tool run from some form of commit script. If you write an email parsing function, they can also all be driven by a suitable mail-parsing
source <Mail-parsing-ChangeSources>
. Additionally, handlers for web-based notification (i.e. from GitHub) can be used with WebStatus' change_hook module. The interface is simple, so adding your own handlers (and sharing!) should be a breeze.
See :bbchsrc
for a full list of change sources.
Change Sources
The :bbchange_source
configuration key holds all active change sources for the configuration.
Most configurations have a single ChangeSource
, watching only a single tree, e.g., :
c['change_source'] = PBChangeSource()
For more advanced configurations, the parameter can be a list of change sources:
source1 = ...
source2 = ...
c['change_source'] = [ source1, source1 ]
ChangeSource
s will, in general, automatically provide the proper repository
attribute for any changes they produce. For systems which operate on URL-like specifiers, this is a repository URL. Other ChangeSource
s adapt the concept as necessary.
Many ChangeSource
s allow you to specify a project, as well. This attribute is useful when building from several distinct codebases in the same buildmaster: the project string can serve to differentiate the different codebases. Scheduler
s can filter on project, so you can configure different builders to run for each project.
Many projects publish information about changes to their source tree by sending an email message out to a mailing list, frequently named {PROJECT}-commits
or {PROJECT}-changes
. Each message usually contains a description of the change (who made the change, which files were affected) and sometimes a copy of the diff. Humans can subscribe to this list to stay informed about what's happening to the source tree.
The Buildbot can also be subscribed to a -commits mailing list, and can trigger builds in response to Changes that it hears about. The buildmaster admin needs to arrange for these email messages to arrive in a place where the buildmaster can find them, and configure the buildmaster to parse the messages correctly. Once that is in place, the email parser will create Change objects and deliver them to the Schedulers (see Schedulers
) just like any other ChangeSource.
There are two components to setting up an email-based ChangeSource. The first is to route the email messages to the buildmaster, which is done by dropping them into a maildir. The second is to actually parse the messages, which is highly dependent upon the tool that was used to create them. Each VC system has a collection of favorite change-emailing tools, and each has a slightly different format, so each has a different parsing function. There is a separate ChangeSource variant for each parsing function.
Once you've chosen a maildir location and a parsing function, create the change source and put it in change_source
:
from buildbot.changes.mail import CVSMaildirSource
c['change_source'] = CVSMaildirSource("~/maildir-buildbot",
prefix="/trunk/")
The recommended way to install the buildbot is to create a dedicated account for the buildmaster. If you do this, the account will probably have a distinct email address (perhaps buildmaster@example.org). Then just arrange for this account's email to be delivered to a suitable maildir (described in the next section).
If the buildbot does not have its own account, extension addresses can be used to distinguish between email intended for the buildmaster and email intended for the rest of the account. In most modern MTAs, the e.g. foo@example.org account has control over every email address at example.org which begins with "foo", such that email addressed to account-foo@example.org can be delivered to a different destination than account-bar@example.org. qmail does this by using separate .qmail
files for the two destinations (.qmail-foo
and .qmail-bar
, with .qmail
controlling the base address and .qmail-default
controlling all other extensions). Other MTAs have similar mechanisms.
Thus you can assign an extension address like foo-buildmaster@example.org to the buildmaster, and retain foo@example.org for your own use.
A maildir is a simple directory structure originally developed for qmail that allows safe atomic update without locking. Create a base directory with three subdirectories: new
, tmp
, and cur
. When messages arrive, they are put into a uniquely-named file (using pids, timestamps, and random numbers) in tmp
. When the file is complete, it is atomically renamed into new
. Eventually the buildmaster notices the file in new
, reads and parses the contents, then moves it into cur
. A cronjob can be used to delete files in cur
at leisure.
Maildirs are frequently created with the maildirmake
tool, but a simple mkdir -p ~/MAILDIR/\{cur,new,tmp\}
is pretty much equivalent.
Many modern MTAs can deliver directly to maildirs. The usual .forward
or .procmailrc
syntax is to name the base directory with a trailing slash, so something like ~/MAILDIR/
. qmail and postfix are maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail Delivery Agent).
Here is an example procmail config, located in ~/.procmailrc
:
# .procmailrc
# routes incoming mail to appropriate mailboxes
PATH=/usr/bin:/usr/local/bin
MAILDIR=$HOME/Mail
LOGFILE=.procmail_log
SHELL=/bin/sh
:0
*
new
If procmail is not setup on a system wide basis, then the following one-line .forward
file will invoke it. :
!/usr/bin/procmail
For MTAs which cannot put files into maildirs directly, the safecat tool can be executed from a .forward
file to accomplish the same thing.
The Buildmaster uses the linux DNotify facility to receive immediate notification when the maildir's new
directory has changed. When this facility is not available, it polls the directory for new messages, every 10 seconds by default.
The second component to setting up an email-based ChangeSource
is to parse the actual notices. This is highly dependent upon the VC system and commit script in use.
A couple of common tools used to create these change emails, along with the buildbot tools to parse them, are:
- CVS
- Buildbot CVS MailNotifier
:bb
CVSMaildirSource
- SVN
- svnmailer
commit-email.pl
:bb
SVNCommitEmailMaildirSource
- Bzr
- Launchpad
:bb
BzrLaunchpadEmailMaildirSource
- Mercurial
- NotifyExtension
http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension
- Git
The following sections describe the parsers available for each of these tools.
Most of these parsers accept a prefix=
argument, which is used to limit the set of files that the buildmaster pays attention to. This is most useful for systems like CVS and SVN which put multiple projects in a single repository (or use repository names to indicate branches). Each filename that appears in the email is tested against the prefix: if the filename does not start with the prefix, the file is ignored. If the filename does start with the prefix, that prefix is stripped from the filename before any further processing is done. Thus the prefix usually ends with a slash.
This parser works with the buildbot_cvs_maildir.py
script in the contrib directory.
The script sends an email containing all the files submitted in one directory. It is invoked by using the CVSROOT/loginfo
facility.
The Buildbot's :bbCVSMaildirSource
knows how to parse these messages and turn them into Change objects. It takes the directory name of the maildir root. For example:
from buildbot.changes.mail import CVSMaildirSource
c['change_source'] = CVSMaildirSource("/home/buildbot/Mail")
CVS must be configured to invoke the buildbot_cvs_mail.py script when files are checked in. This is done via the CVS loginfo configuration file.
To update this, first do:
cvs checkout CVSROOT
cd to the CVSROOT directory and edit the file loginfo, adding a line like:
SomeModule /cvsroot/CVSROOT/buildbot_cvs_mail.py --cvsroot :ext:example.com:/cvsroot -e buildbot -P SomeModule %@{sVv@}
Note
For cvs version 1.12.x, the --path %p
option is required. Version 1.11.x and 1.12.x report the directory path differently.
The above example you put the buildbot_cvs_mail.py script under /cvsroot/CVSROOT. It can be anywhere. Run the script with --help to see all the options. At the very least, the options -e
(email) and -P
(project) should be specified. The line must end with %{sVv}
This is expanded to the files that were modified.
Additional entries can be added to support more modules.
See buildbot_cvs_mail.py --help
` for more information on the available options.
:bbSVNCommitEmailMaildirSource
parses message sent out by the commit-email.pl
script, which is included in the Subversion distribution.
It does not currently handle branches: all of the Change objects that it creates will be associated with the default (i.e. trunk) branch. :
from buildbot.changes.mail import SVNCommitEmailMaildirSource
c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot")
:bbBzrLaunchpadEmailMaildirSource
parses the mails that are sent to addresses that subscribe to branch revision notifications for a bzr branch hosted on Launchpad.
The branch name defaults to lp:{Launchpad path}
. For example lp:~maria-captains/maria/5.1
.
If only a single branch is used, the default branch name can be changed by setting defaultBranch
.
For multiple branches, pass a dictionary as the value of the branchMap
option to map specific repository paths to specific branch names (see example below). The leading lp:
prefix of the path is optional.
The prefix
option is not supported (it is silently ignored). Use the branchMap
and defaultBranch
instead to assign changes to branches (and just do not subscribe the buildbot to branches that are not of interest).
The revision number is obtained from the email text. The bzr revision id is not available in the mails sent by Launchpad. However, it is possible to set the bzr append_revisions_only option for public shared repositories to avoid new pushes of merges changing the meaning of old revision numbers. :
from buildbot.changes.mail import BzrLaunchpadEmailMaildirSource
bm = { 'lp:~maria-captains/maria/5.1' : '5.1', 'lp:~maria-captains/maria/6.0' : '6.0' }
c['change_source'] = BzrLaunchpadEmailMaildirSource("~/maildir-buildbot", branchMap = bm)
:bbPBChangeSource
actually listens on a TCP port for clients to connect and push change notices into the Buildmaster. This is used by the built-in buildbot sendchange
notification tool, as well as several version-control hook scripts. This change is also useful for creating new kinds of change sources that work on a push model instead of some kind of subscription scheme, for example a script which is run out of an email .forward
file. This ChangeSource always runs on the same TCP port as the slaves. It shares the same protocol, and in fact shares the same space of "usernames", so you cannot configure a :bbPBChangeSource
with the same name as a slave.
If you have a publicly accessible slave port, and are using :bbPBChangeSource
, you must establish a secure username and password for the change source. If your sendchange credentials are known (e.g., the defaults), then your buildmaster is susceptible to injection of arbitrary changes, which (depending on the build factories) could lead to arbitrary code execution on buildslaves.
The :bbPBChangeSource
is created with the following arguments.
port
which port to listen on. If
None
(which is the default), it shares the port used for buildslave connections.user
The user account that the client program must use to connect. Defaults to
change
passwd
The password for the connection - defaults to
changepw
. Do not use this default on a publicly exposed port!prefix
The prefix to be found and stripped from filenames delivered over the connection, defaulting to
None
. Any filenames which do not start with this prefix will be removed. If all the filenames in a given Change are removed, the that whole Change will be dropped. This string should probably end with a directory separator.This is useful for changes coming from version control systems that represent branches as parent directories within the repository (like SVN and Perforce). Use a prefix of
trunk/
orproject/branches/foobranch/
to only follow one branch and to get correct tree-relative filenames. Without a prefix, the :bbPBChangeSource
will probably deliver Changes with filenames liketrunk/foo.c
instead of justfoo.c
. Of course this also depends upon the tool sending the Changes in (like :bbbuildbot sendchange <sendchange>
) and what filenames it is delivering: that tool may be filtering and stripping prefixes at the sending end.
For example:
from buildbot.changes import pb
c['change_source'] = pb.PBChangeSource(port=9999, user='laura', passwd='fpga')
The following hooks are useful for sending changes to a :bbPBChangeSource
:
Since Mercurial is written in Python, the hook script can invoke Buildbot's sendchange
function directly, rather than having to spawn an external process. This function delivers the same sort of changes as buildbot sendchange
and the various hook scripts in contrib/
, so you'll need to add a :bbPBChangeSource
to your buildmaster to receive these changes.
To set this up, first choose a Mercurial repository that represents your central official source tree. This will be the same repository that your buildslaves will eventually pull from. Install Buildbot on the machine that hosts this repository, using the same version of Python as Mercurial is using (so that the Mercurial hook can import code from buildbot). Then add the following to the .hg/hgrc
file in that repository, replacing the buildmaster hostname/portnumber as appropriate for your buildbot:
[hooks]
changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook
[hgbuildbot]
master = buildmaster.example.org:9987
# .. other hgbuildbot parameters ..
The master
configuration key allows to have more than one buildmaster specification. The buildmasters have to be separated by a whitspace or comma (see also 'hg help config'):
master =
buildmaster.example.org:9987
buildmaster2.example.org:9989
Note
Mercurial lets you define multiple changegroup
hooks by giving them distinct names, like changegroup.foo
and changegroup.bar
, which is why we use changegroup.buildbot
in this example. There is nothing magical about the buildbot suffix in the hook name. The [hgbuildbot]
section is special, however, as it is the only section that the buildbot hook pays attention to.)
Also note that this runs as a changegroup
hook, rather than as an incoming
hook. The changegroup
hook is run with multiple revisions at a time (say, if multiple revisions are being pushed to this repository in a single hg push
command), whereas the incoming
hook is run with just one revision at a time. The hgbuildbot.hook
function will only work with the changegroup
hook.
Changes' attribute properties
has an entry is_merge
which is set to true when the change was caused by a merge.
If the buildmaster :bbPBChangeSource
is configured to require sendchange credentials then you can set these with the auth
parameter. When this parameter is not set it defaults to change:changepw
, which are the defaults for the user
and password
values of a PBChangeSource
which doesn't require authentication.
[hgbuildbot]
auth = clientname:supersecret
# ...
You can set this parameter in either the global /etc/mercurial/hgrc
, your personal ~/.hgrc
file or the repository local .hg/hgrc
file. But since this value is stored in plain text, you must make sure that it can only be read by those users that need to know the authentication credentials.
The [hgbuildbot]
section has two other parameters that you might specify, both of which control the name of the branch that is attached to the changes coming from this hook.
One common branch naming policy for Mercurial repositories is to use Mercurial's built-in branches (the kind created with hg
branch
and listed with hg branches
). This feature associates persistent names with particular lines of descent within a single repository. (note that the buildbot source.Mercurial
checkout step does not yet support this kind of branch). To have the commit hook deliver this sort of branch name with the Change object, use branchtype = inrepo
, this is the default behavior:
[hgbuildbot]
branchtype = inrepo
# ...
Another approach is for each branch to go into a separate repository, and all the branches for a single project share a common parent directory. For example, you might have /var/repos/{PROJECT}/trunk/
and /var/repos/{PROJECT}/release
. To use this style, use the branchtype = dirname
setting, which simply uses the last component of the repository's enclosing directory as the branch name:
[hgbuildbot]
branchtype = dirname
# ...
Finally, if you want to simply specify the branchname directly, for all changes, use branch = BRANCHNAME
. This overrides branchtype
:
[hgbuildbot]
branch = trunk
# ...
If you use branch=
like this, you'll need to put a separate .hgrc
in each repository. If you use branchtype=
, you may be able to use the same .hgrc
for all your repositories, stored in ~/.hgrc
or /etc/mercurial/hgrc
.
As twisted needs to hook some signals, and some web servers strictly forbid that, the parameter fork
in the [hgbuildbot]
section will instruct Mercurial to fork before sending the change request. Then as the created process will be of short life, it is considered as safe to disable the signal restriction in the Apache setting like that WSGIRestrictSignal Off
. Refer to the documentation of your web server for other way to do the same.
The category
parameter sets the category for any changes generated from the hook. Likewise, the project
parameter sets the project.
Changes' repository
attributes are formed from the Mercurial repo path by stripping strip
slashes on the left, then prepending the baseurl
. For example, assume the following parameters:
[hgbuildbot]
baseurl = http://hg.myorg.com/repos/
strip = 3
# ...
Then a repopath of /var/repos/myproject/release
would have its left 3 slashes stripped, leaving myproject/release
, after which the base URL would be prepended, to create http://hg.myorg.com/repos/myproject/release
.
The hgbuildbot
baseurl
value defaults to the value of the same parameter in the web
section of the configuration.
Note
older versions of Buildbot created repository strings that did not contain an entire URL. To continue this pattern, set the hgbuildbot
baseurl
parameter to an empty string:
[hgbuildbot]
baseurl = http://hg.myorg.com/repos/
Bzr is also written in Python, and the Bzr hook depends on Twisted to send the changes.
To install, put contrib/bzr_buildbot.py
in one of your plugins locations a bzr plugins directory (e.g., ~/.bazaar/plugins
). Then, in one of your bazaar conf files (e.g., ~/.bazaar/locations.conf
), set the location you want to connect with buildbot with these keys:
buildbot_on
one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via commit, changes via push, or any changes to the trunk. 'change' is recommended.buildbot_server
(required to send to a buildbot master) the URL of the buildbot master to which you will connect (as of this writing, the same server and port to which slaves connect).buildbot_port
(optional, defaults to 9989) the port of the buildbot master to which you will connect (as of this writing, the same server and port to which slaves connect)buildbot_pqm
(optional, defaults to not pqm) Normally, the user that commits the revision is the user that is responsible for the change. When run in a pqm (Patch Queue Manager, see https://launchpad.net/pqm) environment, the user that commits is the Patch Queue Manager, and the user that committed the parent revision is responsible for the change. To turn on the pqm mode, set this value to any of (case-insensitive) "Yes", "Y", "True", or "T".buildbot_dry_run
(optional, defaults to not a dry run) Normally, the post-commit hook will attempt to communicate with the configured buildbot server and port. If this parameter is included and any of (case-insensitive) "Yes", "Y", "True", or "T", then the hook will simply print what it would have sent, but not attempt to contact the buildbot master.buildbot_send_branch_name
(optional, defaults to not sending the branch name) If your buildbot's bzr source build step uses a repourl, do not turn this on. If your buildbot's bzr build step uses a baseURL, then you may set this value to any of (case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master append the branch name to the baseURL.
Note
The bzr smart server (as of version 2.2.2) doesn't know how to resolve bzr://
urls into absolute paths so any paths in locations.conf
won't match, hence no change notifications will be sent to Buildbot. Setting configuration parameters globally or in-branch might still work. When buildbot no longer has a hardcoded password, it will be a configuration option here as well.
Here's a simple example that you might have in your ~/.bazaar/locations.conf
.
[chroot-*:///var/local/myrepo/mybranch]
buildbot_on = change
buildbot_server = localhost
The :bbP4Source
periodically polls a Perforce depot for changes. It accepts the following arguments:
p4base
The base depot path to watch, without the trailing '/...'.
p4port
The Perforce server to connect to (as
{host}:{port}
).p4user
The Perforce user.
p4passwd
The Perforce password.
p4bin
An optional string parameter. Specify the location of the perforce command line binary (p4). You only need to do this if the perforce binary is not in the path of the buildbot user. Defaults to p4.
split_file
A function that maps a pathname, without the leading
p4base
, to a (branch, filename) tuple. The default just returns(None, branchfile)
, which effectively disables branch support. You should supply a function which understands your repository structure.pollInterval
How often to poll, in seconds. Defaults to 600 (10 minutes).
histmax
The maximum number of changes to inspect at a time. If more than this number occur since the last poll, older changes will be silently ignored.
encoding
The character encoding of
p4
's output. This defaults to "utf8", but if your commit messages are in another encoding, specify that here.
This configuration uses the P4PORT
, P4USER
, and P4PASSWD
specified in the buildmaster's environment. It watches a project in which the branch name is simply the next path component, and the file is all path components after. :
from buildbot.changes import p4poller
s = p4poller.P4Source(p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1),
)
c['change_source'] = s
The :bbBonsaiPoller
periodically polls a Bonsai server. This is a CGI script accessed through a web server that provides information about a CVS tree, for example the Mozilla bonsai server at http://bonsai.mozilla.org. Bonsai servers are usable by both humans and machines. In this case, the buildbot's change source forms a query which asks about any files in the specified branch which have changed since the last query.
:bbBonsaiPoller
accepts the following arguments:
bonsaiURL
The base URL of the Bonsai server, e.g.,
http://bonsai.mozilla.org
module
The module to look for changes in. Commonly this is
all
.branch
The branch to look for changes in. This will appear in the
branch
field of the resulting change objects.tree
The tree to look for changes in. Commonly this is
all
.cvsroot
The CVS root of the repository. Usually this is
/cvsroot
.pollInterval
The time (in seconds) between queries for changes.
project
The project name to attach to all change objects produced by this change source.
The :bbSVNPoller
is a ChangeSource which periodically polls a Subversion repository for new revisions, by running the svn log
command in a subshell. It can watch a single branch or multiple branches.
:bbSVNPoller
accepts the following arguments:
svnurl
The base URL path to watch, like
svn://svn.twistedmatrix.com/svn/Twisted/trunk
, orhttp://divmod.org/svn/Divmo/
, or evenfile:///home/svn/Repository/ProjectA/branches/1.5/
. This must include the access scheme, the location of the repository (both the hostname for remote ones, and any additional directory names necessary to get to the repository), and the sub-path within the repository's virtual filesystem for the project and branch of interest.The :bb
SVNPoller
will only pay attention to files inside the subdirectory specified by the complete svnurl.split_file
A function to convert pathnames into
(branch, relative_pathname)
tuples. Use this to explain your repository's branch-naming policy to :bbSVNPoller
. This function must accept a single string (the pathname relative to the repository) and return a two-entry tuple. Directory pathnames always end with a right slash to distinguish them from files, liketrunk/src/
, orsrc/
. There are a few utility functions inbuildbot.changes.svnpoller
that can be used as asplit_file
function; see below for details.For directories, the relative pathname returned by
split_file
should end with a right slash but an empty string is also accepted for the root, like("branches/1.5.x", "")
being converted from"branches/1.5.x/"
.The default value always returns
(None, path)
, which indicates that all files are on the trunk.Subclasses of :bb
SVNPoller
can override thesplit_file
method instead of using thesplit_file=
argument.project
Set the name of the project to be used for the :bb
SVNPoller
. This will then be set in any changes generated by the :bbSVNPoller
, and can be used in aChange Filter <Change-Filters>
for triggering particular builders.svnuser
An optional string parameter. If set, the
--user
argument will be added to allsvn
commands. Use this if you have to authenticate to the svn server before you can dosvn info
orsvn log
commands.svnpasswd
Like
svnuser
, this will cause a--password
argument to be passed to allsvn
commands.pollInterval
How often to poll, in seconds. Defaults to 600 (checking once every 10 minutes). Lower this if you want the buildbot to notice changes faster, raise it if you want to reduce the network and CPU load on your svn server. Please be considerate of public SVN repositories by using a large interval when polling them.
histmax
The maximum number of changes to inspect at a time. Every
pollInterval
seconds, the :bbSVNPoller
asks for the lasthistmax
changes and looks through them for any revisions it does not already know about. If more thanhistmax
revisions have been committed since the last poll, older changes will be silently ignored. Larger values ofhistmax
will cause more time and memory to be consumed on each poll attempt.histmax
defaults to 100.svnbin
This controls the
svn
executable to use. If subversion is installed in a weird place on your system (outside of the buildmaster'sPATH
), use this to tell :bbSVNPoller
where to find it. The default value of svn will almost always be sufficient.revlinktmpl
This parameter is deprecated in favour of specifying a global revlink option. This parameter allows a link to be provided for each revision (for example, to websvn or viewvc). These links appear anywhere changes are shown, such as on build or change pages. The proper form for this parameter is an URL with the portion that will substitute for a revision number replaced by ''%s''. For example,
'http://myserver/websvn/revision.php?rev=%s'
could be used to cause revision links to be created to a websvn repository viewer.cachepath
If specified, this is a pathname of a cache file that :bb
SVNPoller
will use to store its state between restarts of the master.extra_args
If specified, the extra arguments will be added to the svn command args.
Several split file functions are available for common SVN repository layouts. For a poller that is only monitoring trunk, the default split file function is available explicitly as split_file_alwaystrunk
:
from buildbot.changes.svnpoller import SVNPoller
from buildbot.changes.svnpoller import split_file_alwaystrunk
c['change_source'] = SVNPoller(
svnurl="svn://svn.twistedmatrix.com/svn/Twisted/trunk",
split_file=split_file_alwaystrunk)
For repositories with the /trunk
and /branches/{BRANCH}
layout, split_file_branches
will do the job:
from buildbot.changes.svnpoller import SVNPoller
from buildbot.changes.svnpoller import split_file_branches
c['change_source'] = SVNPoller(
svnurl="https://amanda.svn.sourceforge.net/svnroot/amanda/amanda",
split_file=split_file_branches)
When using this splitter the poller will set the project
attribute of any changes to the project
attribute of the poller.
For repositories with the {PROJECT}/trunk
and {PROJECT}/branches/{BRANCH}
layout, split_file_projects_branches
will do the job:
from buildbot.changes.svnpoller import SVNPoller
from buildbot.changes.svnpoller import split_file_projects_branches
c['change_source'] = SVNPoller(
svnurl="https://amanda.svn.sourceforge.net/svnroot/amanda/",
split_file=split_file_projects_branches)
When using this splitter the poller will set the project
attribute of any changes to the project determined by the splitter.
The :bbSVNPoller
is highly adaptable to various Subversion layouts. See Customizing-SVNPoller
for details and some common scenarios.
If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To use, put contrib/bzr_buildbot.py
somewhere that your buildbot configuration can import it. Even putting it in the same directory as the master.cfg
should work. Install the poller in the buildbot configuration as with any other change source. Minimally, provide a URL that you want to poll (bzr://
, bzr+ssh://
, or lp:
), making sure the buildbot user has necessary privileges. :
# bzr_buildbot.py in the same directory as master.cfg
from bzr_buildbot import BzrPoller
c['change_source'] = BzrPoller(
url='bzr://hostname/my_project',
poll_interval=300)
The BzrPoller
parameters are:
url
The URL to poll.
poll_interval
The number of seconds to wait between polls. Defaults to 10 minutes.
branch_name
Any value to be used as the branch name. Defaults to None, or specify a string, or specify the constants from
bzr_buildbot.py
SHORT
orFULL
to get the short branch name or full branch address.blame_merge_author
normally, the user that commits the revision is the user that is responsible for the change. When run in a pqm (Patch Queue Manager, see https://launchpad.net/pqm) environment, the user that commits is the Patch Queue Manager, and the user that committed the merged, parent revision is responsible for the change. set this value to
True
if this is pointed against a PQM-managed branch.
If you cannot take advantage of post-receive hooks as provided by contrib/git_buildbot.py
for example, then you can use the :bbGitPoller
.
The :bbGitPoller
periodically fetches from a remote Git repository and processes any changes. It requires its own working directory for operation. The default should be adequate, but it can be overridden via the workdir
property.
Note
There can only be a single GitPoller pointed at any given repository.
The :bbGitPoller
requires Git-1.7 and later. It accepts the following arguments:
repourl
the git-url that describes the remote repository, e.g.
git@example.com:foobaz/myrepo.git
(see thegit fetch
help for more info on git-url formats)branches
a list of the branches to fetch, will default to
['master']
branch
accepts a single branch name to fetch. Exists for backwards compatibility with old configurations.
pollInterval
interval in seconds between polls, default is 10 minutes
gitbin
path to the Git binary, defaults to just
'git'
category
Set the category to be used for the changes produced by the :bb
GitPoller
. This will then be set in any changes generated by the :bbGitPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used for the :bb
GitPoller
. This will then be set in any changes generated by theGitPoller
, and can be used in a Change Filter for triggering particular builders.usetimestamps
parse each revision's commit timestamp (default is
True
), or ignore it in favor of the current time (so recently processed commits appear together in the waterfall page)encoding
Set encoding will be used to parse author's name and commit message. Default encoding is
'utf-8'
. This will not be applied to file names since Git will translate non-ascii file names to unreadable escape sequences.workdir
the directory where the poller should keep its local repository. The default is
gitpoller_work
. If this is a relative path, it will be interpreted relative to the master's basedir. Multiple Git pollers can share the same directory.
An configuration for the Git poller might look like this:
from buildbot.changes.gitpoller import GitPoller
c['change_source'] = GitPoller('git@example.com:foobaz/myrepo.git',
branches=['master', 'great_new_feature'])
If you cannot take advantage of post-receive hooks as provided by buildbot/changes/hgbuildbot.py
for example, then you can use the :bbHgPoller
.
The :bbHgPoller
periodically pulls a named branch from a remote Mercurial repository and processes any changes. It requires its own working directory for operation, which must be specified via the workdir
property.
The :bbHgPoller
requires a working hg
executable, and at least a read-only access to the repository it polls (possibly through ssh keys or by tweaking the hgrc
of the system user buildbot runs as).
The :bbHgPoller
will not transmit any change if there are several heads on the watched named branch. This is similar (although not identical) to the Mercurial executable behaviour. This exceptional condition is usually the result of a developer mistake, and usually does not last for long. It is reported in logs. If fixed by a later merge, the buildmaster administrator does not have anything to do: that merge will be transmitted, together with the intermediate ones.
The :bbHgPoller
accepts the following arguments:
repourl
the url that describes the remote repository, e.g.
http://hg.example.com/projects/myrepo
. Any url suitable forhg pull
can be specified.branch
the desired branch to pull, will default to
'default'
workdir
the directory where the poller should keep its local repository. It is mandatory for now, although later releases may provide a meaningful default.
It also serves to identify the poller in the buildmaster internal database. Changing it may result in re-processing all changes so far.
Several :bb
HgPoller
instances may share the sameworkdir
for mutualisation of the common history between two different branches, thus easing on local and remote system resources and bandwidth.If relative, the
workdir
will be interpreted from the master directory.pollInterval
interval in seconds between polls, default is 10 minutes
hgbin
path to the Mercurial binary, defaults to just
'hg'
category
Set the category to be used for the changes produced by the :bb
HgPoller
. This will then be set in any changes generated by the :bbHgPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used for the :bb
HgPoller
. This will then be set in any changes generated by theHgPoller
, and can be used in a Change Filter for triggering particular builders.usetimestamps
parse each revision's commit timestamp (default is
True
), or ignore it in favor of the current time (so recently processed commits appear together in the waterfall page)encoding
Set encoding will be used to parse author's name and commit message. Default encoding is
'utf-8'
.
A configuration for the Mercurial poller might look like this:
from buildbot.changes.hgpoller import HgPoller
c['change_source'] = HgPoller('http://hg.example.org/projects/myrepo',
branch='great_new_feature',
workdir='hg-myrepo')
The :bbGerritChangeSource
class connects to a Gerrit server by its SSH interface and uses its event source mechanism, gerrit stream-events.
This class adds a change to the buildbot system for each of the following events:
patchset-created
A change is proposed for review. Automatic checks like
checkpatch.pl
can be automatically triggered. Beware of what kind of automatic task you trigger. At this point, no trusted human has reviewed the code, and a patch could be specially crafted by an attacker to compromise your buildslaves.ref-updated
A change has been merged into the repository. Typically, this kind of event can lead to a complete rebuild of the project, and upload binaries to an incremental build results server.
This class will populate the property list of the triggered build with the info received from Gerrit server in JSON format.
Properties; from GerritChangeSource
In case of patchset-created
event, these properties will be:
event.change.branch
Branch of the Change
event.change.id
Change's ID in the Gerrit system (the ChangeId: in commit comments)
event.change.number
Change's number in Gerrit system
event.change.owner.email
Change's owner email (owner is first uploader)
event.change.owner.name
Change's owner name
event.change.project
Project of the Change
event.change.subject
Change's subject
event.change.url
URL of the Change in the Gerrit's web interface
event.patchSet.number
Patchset's version number
event.patchSet.ref
Patchset's Gerrit "virtual branch"
event.patchSet.revision
Patchset's Git commit ID
event.patchSet.uploader.email
Patchset uploader's email (owner is first uploader)
event.patchSet.uploader.name
Patchset uploader's name (owner is first uploader)
event.type
Event type (
patchset-created
)event.uploader.email
Patchset uploader's email
event.uploader.name
Patchset uploader's name
In case of ref-updated
event, these properties will be:
event.refUpdate.newRev
New Git commit ID (after merger)
event.refUpdate.oldRev
Previous Git commit ID (before merger)
event.refUpdate.project
Project that was updated
event.refUpdate.refName
Branch that was updated
event.submitter.email
Submitter's email (merger responsible)
event.submitter.name
Submitter's name (merger responsible)
event.type
Event type (
ref-updated
)event.submitter.email
Submitter's email (merger responsible)
event.submitter.name
Submitter's name (merger responsible)
A configuration for this source might look like:
from buildbot.changes.gerritchangesource import GerritChangeSource
c['change_source'] = GerritChangeSource(gerrit_server, gerrit_user)
see master/docs/examples/repo_gerrit.cfg
in the Buildbot distribution for a full example setup of :bbGerritChangeSource
.
Buildbot already provides a web frontend, and that frontend can easily be used to receive HTTP push notifications of commits from services like GitHub or GoogleCode. See Change-Hooks
for more information.
The :bbGoogleCodeAtomPoller
periodically polls a Google Code Project's commit feed for changes. Works on SVN, Git, and Mercurial repositories. Branches are not understood (yet). It accepts the following arguments:
feedurl
The commit Atom feed URL of the GoogleCode repository (MANDATORY)
pollinterval
Polling frequency for the feed (in seconds). Default is 1 hour (OPTIONAL)
As an example, to poll the Ostinato project's commit feed every 3 hours, the configuration would look like this:
from googlecode_atom import GoogleCodeAtomPoller
c['change_source'] = GoogleCodeAtomPoller(
feedurl="http://code.google.com/feeds/p/ostinato/hgchanges/basic",
pollinterval=10800)
(note that you will need to download googlecode_atom.py
from the Buildbot source and install it somewhere on your PYTHONPATH first)